Alex Rivera | Logout

Kinect Depth and Image Frames Alignment

Asked 2012-04-30T23:09:43.590
10

I am playing around with new Kinect SDK v1.0.3.190. (other related questions in stackoverflow are on previous sdk of kinect) I get depth and color streams from Kinect. As the depth and RGB streams are captured with different sensors there is a misalignment between two frames as can be seen below.

Only RGB RGB

Only Depth Depth

Depth & RGB RGB & Depth equally blended

I need to align them and there is a function named MapDepthToColorImagePoint exactly for this purpose. However it doesn't seem to work. here is a equally blended (depth and mapped color) result below which is created with the following code

            Parallel.For(0, this.depthFrameData.Length, i =>
        {
            int depthVal = this.depthFrameData[i] >> 3;
            ColorImagePoint point = this.kinectSensor.MapDepthToColorImagePoint(DepthImageFormat.Resolution640x480Fps30, i / 640, i % 640, (short)depthVal, ColorImageFormat.RgbResolution640x480Fps30);
            int baseIndex = Math.Max(0, Math.Min(this.videoBitmapData.Length - 4, (point.Y * 640 + point.X) * 4));

            this.mappedBitmapData[baseIndex] = (byte)((this.videoBitmapData[baseIndex]));
            this.mappedBitmapData[baseIndex + 1] = (byte)((this.videoBitmapData[baseIndex + 1]));
            this.mappedBitmapData[baseIndex + 2] = (byte)((this.videoBitmapData[baseIndex + 2]));
        });

where

depthFrameData -> raw depth data (short array)

videoBitmapData -> raw image data (byte array)

mappedBitmapData -> expected result data (byte array)

order of the parameters, resolution, array sizes are correct (double checked).

The result of the code is: depth and MapDepthToColorImagePoint

The m

Edit
Report

1 Answer

1

This is a very common problem, something inherent in the math involved to sync the two images since the two cameras are sitting in two different places. Kind of like taking the two video feeds produced by a 3D camera and trying to sync them. They will always be slightly off.

Two ideas for correction:

  1. Manually shift the bits from the depth image as you calculate it.
  2. Add a shaker motor to the Kinect to reduce noise: http://www.youtube.com/watch?v=CSBDY0RuhS4 (I have found a simple pager motor to be effective.)

The MapDepthToColorImagePoint is for use in the Skeletal API to map a skeletal point to a depth point and then to an image point so that you can show joints on top of the RGB image.

Hope this helps.

answered 2012-05-07T18:06:17.983

Your Answer