The short question: I am wondering if the kinect SDK / Nite can be exploited to get a depth image IN, skeleton OUT software.
The long question: I am trying to dump depth,rgb,skeleton data streams captured from a v2 Kinect into rosbags. However, to the best of my knowledge, capturing the skeleton stream on Linux with ros, kinect v2 isn't possible yet. Therefore, I was wondering if I could dump rosbags containing rgb,depth streams, and then post-process these to get the skeleton stream.
I can capture all three streams on windows using the Microsoft kinect v2 SDK, but then dumping them to rosbags, with all the metadata (camera_info, sync info etc) would be painful (correct me if I am wrong).
It's quite some time ago that I worked with NITE (and I only used Kinect v1) so maybe someone else can give a more up-to-date answer, but from what I remember, this should easily be possible.
As long as all relevant data is published via ROS topics, it is quite easy to record them with rosbag and play them back afterwards. Every node that can handle live data from the sensor will also be able to do the same work on recorded data coming from a bag file.
One issue you may encounter is that if you record kinect-data, the bag files are quickly becoming very large (several gigabytes). This can be problematic if you want to edit the file afterwards on a machine with very little RAM. If you only want to play the file or if you have enough RAM, this should not really be a problem, though.
Indeed it is possible to perform a NiTE2 skeleton tracking on any depth-image-stream.
Refer to:
https://github.com/VIML/VirtualDeviceForOpenNI2/wiki/How-to-use
and
https://github.com/VIML/VirtualDeviceForOpenNI2/wiki/About-PrimeSense-NiTE
With this extension one can add a virtual device which allows to manipulate each pixel of the depth stream. This device can then be used for creation of a userTracker object. As long as the right device name is provided skeleton tracking can be done
\OpenNI2\VirtualDevice\Kinect
but consider usage limits:
NiTE only allow to been used with "Authorized Hardware"
Related
Required Info
Camera Model
D435
Firmware Version
05.12.13.50
Operating System & Version
Linux (Ubuntu18.04.5)
Kernel Version (Linux Only)
4.9.201
Platform
NVIDIA JetsonNano B01
SDK Version
2.41.0
Language
ROS packages }
Segment
Robot
Hello, I need to use the obstacle avoidance function in the process of using D435. At present, there are two ways I have inquired:
1、Use depthimage_to_laserscan to convert the depth information into a lidar signal, but the current problem is that there is also a lidar on my robot. Now these two topics are scan, so there is a conflict. I don't know how to solve it.
2、I want to know whether the two lidar signals can be fused, and what configuration is needed to be fused. Is there any relevant information or code?
3、Using plotcloud2 point cloud information, I don't understand how to do this at present. Although the point cloud image can be seen on the map now, it does not have the effect of avoiding obstacles. And does this point cloud information need to be passed to AMCL? If so, how does it need to be delivered? So I hope someone can help me.
I am very new to Kinect programming and am tasked to understand several methods for 3D point cloud stitching using Kinect and OpenCV. While waiting for the Kinect sensor to be shipped over, I am trying to run the SDK samples on some data sets.
I am really clueless as to where to start now, so I downloaded some datasets here, and do not understand how I am supposed to view/parse these datasets. I tried running the Kinect SDK Samples (DepthBasic-D2D) in Visual Studio but the only thing that appears is a white screen with a screenshot button.
There seems to be very little documentation with regards to how all these things work, so I would appreciate if anyone can point me to the right resources on how to obtain and parse depth maps, or how to get the SDK Samples work.
The Point Cloud Library (or PCL) it is a good starting point to handle point cloud data obtained using Kinect and OpenNI driver.
OpenNI is, among other things, an open-source software that provides an API to communicate with vision and audio sensor devices (such as the Kinect). Using OpenNI you can access to the raw data acquired with your Kinect and use it as a input for your PCL software that can process the data. In other words, OpenNI is an alternative to the official KinectSDK, compatible with many more devices, and with great support and tutorials!
There are plenty of tutorials out there like this, this and these.
Also, this question is highly related.
I'm working with a new Kinect v2 sensor, and using Kinect Studio to record the Kinect stream data during some experiments. The problem is our experiments are expected to last ~10 minutes, which including the uncompressed video would be equivalent to ~80gb. In addition, the buffer fills up quite fast and around 2 minutes in and the remainder of the data ends up stuttering at around 2fps instead of the smooth 25fps.
Is there any way I can record all the data I need in compressed form? Would it be easy to create an app similar to kinect studio that just prints out a video file and a .xed file containing all the other sensor data?
Kinect Studio does have APIs that can be used to programmatically record particular data streams into an XEF file. Additionally, it's possible to have multiple applications using the sensor simultaneously, so in theory you should be able to have three applications collecting data from the sensor (you could combine these into one application as well):
Your application;
An application using the Kinect Studio APIs, or Kinect Studio itself, to record the non-RGB streams;
Another application that collects the RGB data stream and performs compression and then saves the data.
However, the latency and buffer issue is likely to be a problem here. Kinect Studio data collection is extremely resource-intensive and it may not be possible to do real-time video compression while maintaining 25fps. Depending on the network infrastructure available you might be able to offload the RGB data to another machine to compress and store, but this would need to be well tested. This is likely to be a lot of work.
I'd suggest that you first see whether switching to another high-spec machine, with a fast SSD drive and good CPU and GPU, makes the buffering issue go away. If this is the case you could then record using Kinect Studio and then post-process the XEF files after the sessions to compress the videos (using the Kinect Studio APIs to open the XEF files).
Working with multiple Kinect v1 sensors is very difficult because of the IR interference between the sensors.
Based on what I read on this gamastura article, Microsoft got rid of the interference problem with the time-of-flight mechanism that Kinect v2 sensor uses to gauge depth.
Does that mean I could use multiple Kinect v2 sensors at the same time, or did I misunderstand the article?
Thanks for the help!
I asked this question, in person, of the dev team at the meetup in San Francisco in April. The answer I got was:
"This feature is 3+ months away. We want to prioritize single-Kinect features before working on multiple Kinects."
I'm a researcher, and my goal is to have a bunch of odd setups, so this is a frustrating answer, but I understand that they need to prioritize usage that will be immediately useful to a larger market.
Could you connect them to multiple computers and stream data back and forth?
As #escapecharacter mentioned not likely to have support for multiple kinect v2 sensors in the very near future.
I can also confirm, one of the Kinect V2 SDK samples has this comment:
// for Alpha, one sensor is supported
this.kinectSensor = KinectSensor.Default;
I think the hardware itself is capable of avoiding the interference problem. Hopefully the slightly larger amount of data (higher res RGB stream) won't be a problem with multiple sensors(and available USB bandwidth) and it would be a matter of enabling the SDK to safely handle multiple sensor instances in the future.
I wouldn't expect a fast/quick update to the SDK enabling though, so in the meantime, although not ideal you could try either:
Using multiple V2 sensors on multiple machines communicating over a
local network, passing only processed/minimal data (to keep the delay
as small as possible)
Using multiple V1 sensors using Shake'n'Sense (pdf link to paper) to reduce interference
At least you would to a certain extent make some progress testing some of your assumptions for your project with multiple sensors, and update the project when the updated SDK is out.
I realize I misread your question, and interpreted it as "how can I connect to Kinect 2's to a computer" when you were actually asking about how to avoid interference, and Kinect 2 was your hoped-for solution.
You can hack avoiding Kinect 1 interference by lighting shaking one of them independent of the other. See here:
http://channel9.msdn.com/coding4fun/kinect/Shaking-some-sense-into-using-multiple-Kinects-with-Shake-n-Sense
One of the craziest things I've ever seen that actually worked. I was at Microsoft Research when they figured this out, and it works quite well.
You can have a Kinect v1 viewing the same scene as a Kinect v2 without interference. I know this isn't exactly what you're looking for, but it could be useful.
2 Years later, and this still cannot be done.
See:
https://social.msdn.microsoft.com/Forums/en-US/8e2233b6-3c4f-485b-a683-6bacd6a74d53/how-to-prevent-interference-between-multiple-kinect-v2-sensors?forum=kinectv2sdk
https://github.com/OpenKinect/libfreenect2/issues/424
As stated in the second link,
What happens is this: Each Kinect v2 continuously switches between different modulation frequencies. When two Kinects switch to the same frequency range, the interference occurs. They typically gradually drift into the same range and after a while drift out of that range again. So, theoretically, you just have to wait a bit until the interference is gone. The only way I found to stop the interference immediately was to disconnect (and reconnect) the concerned Kinect from its power supply
...
Quite unfortunate that these modulation frequencies aren't controllable at this time. Let's hope MS surprises us with that custom firmware
IIRC, I came across a group at MIT that got custom firmware from MS which solved the problem, but I can't seem to find the reference. Unfortunately, it is not available to the public.
I think we cant use multiple Kinect v2 in same environment because they will interfere lot comparatively kinect v1. As Kinect v2 depth sensing based on time of flight principle, multiple kinect v2 will interfere lot. For kinect v1 interference is not that much severe.
I'm new to the GPS world. I need to know how to perform post-processing with DGPS. I found definitions on the net of what DGPS, post-processing, etc. are, but, couldn't find clear steps on how to actually do DGPS post-processing. While searching for providers, I found that CORS data Rinex files are available to the public for free download from NOAA.
I don't get what I should do to use the downloaded data to post-process my GPS data. Is any free post-processing software available? Also, I use an Android phone (HTC Magic) as my GPS receiver, and I need to figure out how to enable GPS logging into a file in my Android application so that the GPS data can be used for post-processing.
RTKLIB has many nice features and produces reliable results.