I'm using OpenNI SDK v1 and attempting to store the alignment between rgb and depth data.
In NiViewer, I enable the overlay mode with registration turned on like so:
// sets the the depth image output from the vantage point of the rgb image
g_Depth.GetAlternativeViewPointCap().SetViewPoint(g_Image);
I understand that this would give me a 1:1 pixel mapping between rgb and depth if both were recorded at the same resolution.
However, for my application, I need rgb to be at 1280x1024 (high res) and depth to be at 640x480.
I'm not sure how the mapping between the depth pixels to rgb would work in this mode.
I had the same problem. By following the advice here, I was able to get it working as desired. It's a bit hacky, but basically you:
Get the 1280x1024 image from OpenNI.
Cut off the bottom to make it 1280x960.
Scale the depth image to 1280x960.
Then they should line up. It's working for me.
Related
When taking a depth pixel buffer from ARKit session like:
CVPixelBufferRef depthDataMap = frame.capturedDepthData.depthDataMap;
many pixels are in black. What do they represent? Why are they black? This doesn't reflect the real depth of the real world.
How to interpret this?
It's some sort of "trail" artifact.
Those black pixels in a Depth channel are digital artifacts. The appearance of this black outline became possible due to the fact that the depth data is captured at a frequency of 15 fps (at every fourth frame), in contrast to the RGB image which is captured at a frequency of 60 fps. Also, a Depth channel image is of a low quality (low-res). Thus, if the subject quickly or abruptly moves its head at the time of AR session running, when every three out of four depth frames contain a completely black background – this leads to a situation when RGBA and Depth data don't match each other, so we get these nasty digital artifacts.
Look at a picture in a post How to improve People Occlusion in ARKit 3.0 (click on a link below).
If you need an additional info about a depth map and its trailing effect, please read this post.
This means no depth data for this pixel (depth = 0)
I'm using Kinect v2 and Kinect SDK v2.
I have couple of questions about coordinate mapping:
How to transfer a camera space point (point in 3d coordinate system) to depth space with depth value?
Current MapCameraPointToDepthSpace method can only return the depth space coordinate.
But without depth value, this method is useless.
Did anyone know how to get the depth value?
How to get the color camera intrinsic?
There is only a GetDepthCameraIntrinsics methos to get depth camera intrinsic.
But how about color camera?
How to use the depth camera intrinsic?
Seems that the Kinect 2 consider the radial distortion.
But how to use these intrinsic to do the transformation between depth pixel and 3d point?
Is there any example code can do this?
Regarding 1: The depth value of your remapped world coordinate is the same as in the original world coordinate's Z value. Read the description of the depth buffer and world coordinate space: this value in both is simply the distance from the point to Kinect's plane, in meters. Now, if you want the depth value of the object being seen on the depth frame directly behind your remapped coordinate, you have to read the depth image buffer in that position.
Regarding 3: You use the camera's intrinsic when you have to manually construct a CoordinateMapper object (i.e. when you don't have a Kinect available). When you get the CoordinateMapper associated to a Kinect (using Kinect object's CoordinateMapper property), it already contains that Kinect's intrinsics... that's why you have a GetDepthCameraIntrinsics method which returns that specific Kinect's intrinsics (they can vary from device to device).
Regarding 2: There is now way to get the color camera intrinsic. You have to evaluate them by camera calibration.
I'm using 2 of the sensors on the asus xtion pro (kinect knockoff), the RGB cam and the user data as a mask. Not sure if it's called user data, the other one that's not depth or color.
It works except for 2 issues:
When you combine the 2 textures together, the mask is actually slightly offset from the color texture, leaving an outline around the character of the background. I believe it's because the two sensors are simply shooting straight out and aren't calibrated, so one is 2" in real world space off from the other.
The second problem is a question about optimizing the mask edges. Is there any way to feather the edges around the character, or smooth based on the difference of neighboring pixels? I find that the edges really jump around on the edges of objects.
So, the shader I'm using to combine the base texture with the mask requires both textures to be the same size, so I can't simply resize it to be slightly smaller to get rid of the gap around the character.
I'm curious of how you would shrink the mask texture by a couple % and add more black around the edges, like if you resized the texture smaller than the rect it occupies, how could you fill in the perimeter with black?
I'm using Unity + OpenNI + Asus xtion pro sensor. The mis-alignment ins't noticeable in most uses, but when doing something really precise it's not that accurate...
Any ideas or pointers? Looking for direction.
Are you using two Asus Xtion Pro sensors at the same time ? The 3 images on the left look like the scene, depth and rgb streams.
Which version of OpenNI are you using ? I haven't used OpenNI with Unity, but I assume you there are equivalent calls to the original API.
For OpenNI 1.5.x look into the Alternative View Point Capability:
yourDepthGenerator.GetAlternativeViewPointCap().SetViewPoint(image);
For OpenNI 2.x it should be something like:
device.setDepthColorSyncEnabled(true);
If you want to calibrate between two sensors, it's a bit more complicated.
What would be the most efficient way to remap the colors of an image to a gradient for iOS? This is defined as "apply a color lookup table to the image" in the Image Magic docs, and generally I think. Is there something built in core image for instance to do this? I know it can be done with ImageMagick code using convert -clut, but not certain that is the most efficient way to do it.
the result of remapping the image to a gradient is as pictured here:
http://owolf.net/uploads/ny.jpg
The basic formula, copied from fraxel's comment is:
1.Open your image as grayscale, and RGB
2.Convert the RGB image to HSV (Hue, Saturation, Value/Brightness) color space. This is a cylindrical space, with hue represented by a single value on the polar axis.
3.Set the hue channel to the grayscale image we already opened, this is the crucial step.
4.Set value, and saturation channels both to maximal values.
5.Convert back to RGB space (otherwise display will be incorrect).
I am writing a Cocoa application for mac osx. I'm trying to figure out how to determine the size of an image that will be captured by a camera? I would like to know the size of the image that will be captured so I can setup a view with an aspect ratio that won't distort the image. For example, if my view is defined to be 640x360 and my camera captures images that are 640x480, the displayed image looks short and fat. I'm also displaying some other layers over the image and I need the image size to be able to scale and position the layers properly.
I won't know the type of camera that is attached until run-time so I'd like to be able to interrogate the device and get attributes like image size. Thanks for the help...
You are altering the aspect ratio of the image when you capture in 640x360 instead of 640x480 or 320x240. You are doing something similar as a resize, using the whole image and making it a different size.
If you don't want to distort the image, but use only a portion of it you need to do a crop. Some hardware support cropping, others don't and you have to do it in software. Cropping is using only portions of the original image. In your case, you would discard the bottom 120 lines.
Example (from here):
The blue rectangle is the natural, or original image and the red is a crop of it.