Kinect background removal - kinect

Question

I followed the code provided by Robert Levy at this link: http://channel9.msdn.com/coding4fun/kinect/Display-Kinect-color-image-containing-only-players-aka-background-removal
I tried implementing it into my existing code, and have had inconsistent results. If the user is in the kinect's field of view when the program starts up it will remove the background some of the time. If the user walks into the field of view it will not pick them up.
namespace KinectUserRecognition
{
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
}
//Kinect Runtime
Runtime kinect = Runtime.Kinects[0];
PlanarImage colorImage;
PlanarImage depthImage;
bool isDepthImage;
WriteableBitmap player1;
private void Window_Loaded(object sender, RoutedEventArgs e)
{
isDepthImage = false;
//UseDepthAndPlayerIndex and UseSkeletalTracking
kinect.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseColor);// | RuntimeOptions.UseSkeletalTracking);
//register for event
kinect.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
kinect.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
//Video image type
kinect.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480,
ImageType.Color);
//DepthAndPlayerIndex ImageType
kinect.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240,
ImageType.DepthAndPlayerIndex);
}
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
colorImage = e.ImageFrame.Image;
image1.Source = BitmapSource.Create(colorImage.Width, colorImage.Height, 96, 96,
PixelFormats.Bgr32, null, colorImage.Bits, colorImage.Width * colorImage.BytesPerPixel);
if (isDepthImage)
{
player1 = GeneratePlayerImage(e.ImageFrame, 1);
image3.Source = player1;
}
}
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
//Convert depth information for a pixel into color information
byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
depthImage = e.ImageFrame.Image;
image2.Source = BitmapSource.Create(depthImage.Width, depthImage.Height, 96, 96, PixelFormats.Bgr32, null,
ColoredBytes, depthImage.Width * PixelFormats.Bgr32.BitsPerPixel / 8);
isDepthImage = true;
}
private WriteableBitmap GeneratePlayerImage(ImageFrame imageFrame, int playerIndex)
{
int depthWidth = kinect.DepthStream.Width;
int depthHeight = kinect.DepthStream.Height;
WriteableBitmap target = new WriteableBitmap(depthWidth, depthHeight, 96, 96, PixelFormats.Bgra32, null);
var depthRect = new System.Windows.Int32Rect(0, 0, depthWidth, depthHeight);
byte[] color = imageFrame.Image.Bits;
byte[] output = new byte[depthWidth * depthHeight * 4];
//loop over each pixel in the depth image
int outputIndex = 0;
for (int depthY = 0, depthIndex = 0; depthY < depthHeight; depthY++)
{
for(int depthX = 0; depthX < depthWidth; depthX++, depthIndex +=2)
{
short depthValue = (short)(depthImage.Bits[depthIndex] | (depthImage.Bits[depthIndex + 1] << 8));
int colorX, colorY;
kinect.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(
imageFrame.Resolution,
imageFrame.ViewArea,
depthX, depthY, //depth coordinate
depthValue, //depth value
out colorX, out colorY); //color coordinate
//ensure that the calculate color location is within the bounds of the image
colorX = Math.Max(0, Math.Min(colorX, imageFrame.Image.Width - 1));
colorY = Math.Max(0, Math.Min(colorY, imageFrame.Image.Height - 1));
output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 0];
output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 1];
output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 2];
output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) == playerIndex ? (byte)255 : (byte)0;
}
}
target.WritePixels(depthRect, output, depthWidth * PixelFormats.Bgra32.BitsPerPixel / 8, 0);
return target;
//return output;
}
private static int GetPlayerIndex(byte firstFrame)
{
//returns 0 = no player, 1 = 1st player, 2 = 2nd player...
//bitwise & on firstFrame
return (int)firstFrame & 7;
}
}
}
-Edit 1-
I think I've narrowed the problem down, but I'm not sure of a way to resolve it. I assumed that having only one person in the kinect's field of view would return a value of one from my "GetPlayerIndex" method. This is not the case. I was hoping to produce a separate image for each person with the background removed. What type of values should I assume to receive from:
-Edit 2-
From my tests I've noticed that I can a max value of 6 for the player index, but the index that I get isn't consistent. If there a way to know what player index will be assigned to a skeleton? For example, if I were the only person in the fov would there be a way to know that my player index would always be 1?

Answer 1

The player index is not guaranteed to be anything. Once it catches a skeleton, the index will stay the same for that skeleton until it loses sight of it, but you can't assume that the first player will be 1, the second 2, etc.
What you'll need to do is determine a valid skeleton index prior to the player1 = GeneratePlayerImage(e.ImageFrame, 1); call, or alter the GeneratePlayerImage function to find an index. If you're only interested in removing the background and leaving the pixels for all the people in the frame untouched, just change this:
output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) == playerIndex ? (byte)255 : (byte)0;
to this, which will just check for ANY player, instead of a specific player:
output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) != 0 ? (byte)255 : (byte)0;
The other two ways I can think of to do this for a specific player instead of all players:
Open the Kinect's Skeleton feed, and loop through the array of skeletons it gives you to find a valid index. Create a global integer to hold this index, then call the GeneratePlayerImage method with this global integer.
Change the GeneratePlayerImage method to check for a player index in each pixel, and if one is found use that index to remove the background for the entire image (ignore any other index it finds).