I'm trying to develop what seems to be a simple program that uses the Kinect for Xbox360 to calculate the distance traveled by a person. The room that the Kinect will be pointed at will be 10 x 10. After the user presses the button, the subject will move about this space. Once the subject reaches their final destination in the area, the user will press the button again. The Kinect will then output how far the subject traveled in between both button presses. Having never developed for the Kinect before, it's been pretty daunting to get started. My issue is that I'm not entirely sure what I should be using to measure the distance. In my research, I've found ways to calculate the distance an object is FROM the Kinect but that's about it.
What you have hear is a simple question of dealing with a Cartesian plane. The Kinect has 20 joints that exist in XYZ space, and distance is measured in meters. In order to access these joints, you have these statements inside a "Tracker" class (this is C#... not sure if you're using C# or C++ in the SDK):
public Tracker(KinectSensor sn, MainWindow win, string fileName)
{
window = win;
sensor = sn;
try
{
sensor.Start();
}
catch (IOException)
{
sensor = null;
MessageBox.Show("No Kinect sensor found. Please connect one and restart the application", "*****ERROR*****");
return;
}
sensor.SkeletonFrameReady += SensorSkeletonFrameReady; //Frame handlers
sensor.ColorFrameReady += SensorColorFrameReady;
sensor.SkeletonStream.Enable();
sensor.ColorStream.Enable();
}
These access the color and skeleton streams from the Kinect. The skeleton stream contains the joints, so you focus on that with these statements:
//Start sending skeleton stream
private void SensorSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
//Access the skeleton frame
using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
{
if (skeletonFrame != null)
{
//Check to see if there is any data in the skeleton
if (this.skeletons == null)
//Allocate array of skeletons
this.skeletons = new Skeleton[skeletonFrame.SkeletonArrayLength];
//Copy skeletons from this frame
skeletonFrame.CopySkeletonDataTo(this.skeletons);
//Find first tracked skeleton, if any
Skeleton skeleton = this.skeletons.Where(s => s.TrackingState == SkeletonTrackingState.Tracked).FirstOrDefault();
if (skeleton != null)
{
//Initialize joints
///<summary>
///Joints to be displayed, projected, recorded, etc.
///</summary>
Joint leftFoot = skeleton.Joints[JointType.FootLeft];
}
}
So, at the beginning of your program, you want to pick a joint (there are 20... choose one that will ALWAYS be facing towards the Kinect when you are executing the program) and get its location with something like the following statements:
if(skeleton.Joints[JointType.FootLeft].TrackingState == JointTrackingState.Tracked)
{
double xPosition = skeleton.Joints[JointType.FootLeft].Position.X;
double yPosition = skeleton.Joints[JointType.FootLeft].Position.Y;
double zPosition = skeleton.Joints[JointType.FootLeft].Position.Z;
}
At the end, you'll want to have a slight delay before you stop the stream... some time between the click and when you shut off the stream from the Kinect. You will then do the math you need to do to get the distance between the two points. If you don't have the delay, you won't be able to get your Cartesian point.
Related
I acknowledge that I have used the sample codes that #Benjamin used in different examples.
I want to have both object detection numbers and accident numbers in my model. I need a code to detect object. But object detection does not necessarily lead to an accident. When an object is detected the agent(transporter) should either stop or change its route. The following code is about this functionality. field of view is a polygonal in front of the transporter.
for (Worker thisPed: main.worker) {
//for each pedestrain in model
double pedX = thisPed.getX() -getX();
double pedY = thisPed.getY() -getY();
if (fieldOfView.contains(pedX, pedY)) {
v_pedInDanger = true;
setSpeed(0);
break;
}
}
How to tell transporter to change route instead of stop? I could not find a code in this regard.
However, I should use another code to calculate distance between transporter and the detected object and if the distance <= 1 METER then we count it as an accident. like the following:
for (Worker ped: main.worker){
double dist = 0;
dist = distanceTo(ped);
if (dist <= 1){
v_pedCollisionNumber += 1;
ped.v_isWorkerCollide = true;
send ("accident", this);
}
}
the second one does not work.
Any Advise please? Any better approach?
I'm trying to implement a feature similar to HTC Vive's controller with the leap motion on my Unity project. I wanted to generate a laser pointer from the index finger and teleport the Vive's room on the position of the laser (as it's done with the controller). The problem is the latest leap motion (orion) documentation, it's unclear. Any ideas how to do that? More in general, we thought about using HandController but we don't understand where to add the script component.
Thanks!
It's unclear to me whether the problem you're having is getting hand data in your scene at all, or using that hand data.
If you're just trying to get hand data in your scene, you can copy a prefab from one of the Unity SDK's example scenes. If you're trying to integrate Leap into an existing scene that already has a VR rig set up, check out the documentation on the core Leap components to understand what pieces need to be in place for you to start getting Hand data. LeapServiceProvider has to be somewhere in your scene to receive hand data.
As long as you have a LeapServiceProvider somewhere, you can access hands from the Leap Motion from any script, anywhere. So for getting a ray from the index fingertip, just pop this script any old place:
using Leap;
using Leap.Unity;
using UnityEngine;
public class IndexRay : MonoBehaviour {
void Update() {
Hand rightHand = Hands.Right;
Vector3 indexTipPosition = rightHand.Fingers[1].TipPosition.ToVector3();
Vector3 indexTipDirection = rightHand.Fingers[1].bones[3].Direction.ToVector3();
// You can try using other bones in the index finger for direction as well;
// bones[3] is the last bone; bones[1] is the bone extending from the knuckle;
// bones[0] is the index metacarpal bone.
Debug.DrawRay(indexTipPosition, indexTipDirection, Color.cyan);
}
}
For what it's worth, the index fingertip direction is probably not going to be stable enough to do what you want. A more reliable strategy is to cast a line from the camera (or a theoretical "shoulder position" at a constant offset from the camera) through the index knuckle bone of the hand:
using Leap;
using Leap.Unity;
using UnityEngine;
public class ProjectiveRay : MonoBehaviour {
// To find an approximate shoulder, let's try 12 cm right, 15 cm down, and 4 cm back relative to the camera.
[Tooltip("An approximation for the shoulder position relative to the VR camera in the camera's (non-scaled) local space.")]
public Vector3 cameraShoulderOffset = new Vector3(0.12F, -0.15F, -0.04F);
public Transform shoulderTransform;
void Update() {
Hand rightHand = Hands.Right;
Vector3 cameraPosition = Camera.main.transform.position;
Vector3 shoulderPosition = cameraPosition + Camera.main.transform.rotation * cameraShoulderOffset;
Vector3 indexKnucklePosition = rightHand.Fingers[1].bones[1].PrevJoint.ToVector3();
Vector3 dirFromShoulder = (indexKnucklePosition - shoulderPosition).normalized;
Debug.DrawRay(indexKnucklePosition, dirFromShoulder, Color.white);
Debug.DrawLine(shoulderPosition, indexKnucklePosition, Color.red);
}
}
I am currently programming with the Microsoft Kinect for Windows SDK 2 on Windows 8.1. Things are going well, and in a home dev environment obviously there is not much noise in the background compared to the 'real world'.
I would like to seek some advice from those with experience in 'real world' applications with the Kinect. How does Kinect (especially v2) fare in a live environment with passers-by, onlookers and unexpected objects in the background? I do expect, in the space from the Kinect sensor to the user there will usually not be interference however - what I am very mindful of right now is the background noise as such.
While I am aware that the Kinect does not track well under direct sunlight (either on the sensor or the user) - are there certain lighting conditions or other external factors I need to factor into the code?
The answer I am looking for is:
What kind of issues can arise in a live environment?
How did you code or work your way around it?
Outlaw Lemur has descibed in detail most of the issues you may encounter in real-world scenarios.
Using Kinect for Windows version 2, you do not need to adjust the motor, since there is no motor and the sensor has a larger field of view. This will make your life much easier.
I would like to add the following tips and advice:
1) Avoid direct light (physical or internal lighting)
Kinect has an infrared sensor that might be confused. This sensor should not have direct contact with any light sources. You can emulate such an environment at your home/office by playing with an ordinary laser pointer and torches.
2) If you are tracking only one person, select the closest tracked user
If your app only needs one player, that player needs to be a) fully tracked and b) closer to the sensor than the others. It's an easy way to make participants understand who is tracked without making your UI more complex.
public static Body Default(this IEnumerable<Body> bodies)
{
Body result = null;
double closestBodyDistance = double.MaxValue;
foreach (var body in bodies)
{
if (body.IsTracked)
{
var position = body.Joints[JointType.SpineBase].Position;
var distance = position.Length();
if (result == null || distance < closestBodyDistance)
{
result = body;
closestBodyDistance = distance;
}
}
}
return result;
}
3) Use the tracking IDs to distinguish different players
Each player has a TrackingID property. Use that property when players interfere or move at random positions. Do not use that property as an alternative to face recognition though.
ulong _trackinfID1 = 0;
ulong _trackingID2 = 0;
void BodyReader_FrameArrived(object sender, BodyFrameArrivedEventArgs e)
{
using (var frame = e.FrameReference.AcquireFrame())
{
if (frame != null)
{
frame.GetAndRefreshBodyData(_bodies);
var bodies = _bodies.Where(b => b.IsTracked).ToList();
if (bodies != null && bodies.Count >= 2 && _trackinfID1 == 0 && _trackingID2 == 0)
{
_trackinfID1 = bodies[0].TrackingId;
_trackingID2 = bodies[1].TrackingId;
// Alternatively, specidy body1 and body2 according to their distance from the sensor.
}
Body first = bodies.Where(b => b.TrackingId == _trackinfID1).FirstOrDefault();
Body second = bodies.Where(b => b.TrackingId == _trackingID2).FirstOrDefault();
if (first != null)
{
// Do something...
}
if (second != null)
{
// Do something...
}
}
}
}
4) Display warnings when a player is too far or too close to the sensor.
To achieve higher accuracy, players need to stand at a specific distance: not too far or too close to the sensor. Here's how to check this:
const double MIN_DISTANCE = 1.0; // in meters
const double MAX_DISTANCE = 4.0; // in meters
double distance = body.Joints[JointType.SpineBase].Position.Z; // in meters, too
if (distance > MAX_DISTANCE)
{
// Prompt the player to move closer.
}
else if (distance < MIN_DISTANCE)
{
// Prompt the player to move farther.
}
else
{
// Player is in the right distance.
}
5) Always know when a player entered or left the scene.
Vitruvius provides an easy way to understand when someone entered or left the scene.
Here is the source code and here is how to use it in your app:
UsersController userReporter = new UsersController();
userReporter.BodyEntered += UserReporter_BodyEntered;
userReporter.BodyLeft += UserReporter_BodyLeft;
userReporter.Start();
void UserReporter_BodyEntered(object sender, UsersControllerEventArgs e)
{
// A new user has entered the scene. Get the ID from e param.
}
void UserReporter_BodyLeft(object sender, UsersControllerEventArgs e)
{
// A user has left the scene. Get the ID from e param.
}
6) Have a visual clue of which player is tracked
If there are a lot of people surrounding the player, you may need to show on-screen who is tracked. You can highlight the depth frame bitmap or use Microsoft's Kinect Interactions.
This is an example of removing the background and keeping the player pixels only.
7) Avoid glossy floors
Some floors (bright, glossy) may mirror people and Kinect may confuse some of their joints (for example, Kinect may extend your legs to the reflected body). If you can't avoid glossy floors, use the FloorClipPlane property of your BodyFrame. However, the best solution would be to have a simple carpet where you expect people to stand. A carpet would also act as an indication of the proper distance, so you would provide a better user experience.
I created an application for home use like you have before, and then presented that same application in a public setting. The result was embarrassing for me, because there were many errors that I would never have anticipated within a controlled environment. However that did help me because it led me to add some interesting adjustments to my code, which is centered around human detection only.
Have conditions for checking the validity of a "human".
When I showed my application in the middle of a presentation floor with many other objects and props, I found that even chairs could be mistaken for people for brief moments, which led to my application switching between the user and an inanimate object, causing it to lose track of the user and lost their progress. To counter this or other false-positive human detections, I added my own additional checks for a human. My most successful method was comparing the proportions of a humans body. I implemented this measured in head units. (head units picture) Below is code of how I did this (SDK version 1.8, C#)
bool PersonDetected = false;
double[] humanRatios = { 1.0f, 4.0, 2.33, 3.0 };
/*Array indexes
* 0 - Head (shoulder to head)
* 1 - Leg length (foot to knee to hip)
* 2 - Width (shoulder to shoulder center to shoulder)
* 3 - Torso (hips to shoulder)
*/
....
double[] currentRatios = new double[4];
double headSize = Distance(skeletons[0].Joints[JointType.ShoulderCenter], skeletons[0].Joints[JointType.Head]);
currentRatios[0] = 1.0f;
currentRatios[1] = (Distance(skeletons[0].Joints[JointType.FootLeft], skeletons[0].Joints[JointType.KneeLeft]) + Distance(skeletons[0].Joints[JointType.KneeLeft], skeletons[0].Joints[JointType.HipLeft])) / headSize;
currentRatios[2] = (Distance(skeletons[0].Joints[JointType.ShoulderLeft], skeletons[0].Joints[JointType.ShoulderCenter]) + Distance(skeletons[0].Joints[JointType.ShoulderCenter], skeletons[0].Joints[JointType.ShoulderRight])) / headSize;
currentRatios[3] = Distance(skeletons[0].Joints[JointType.HipCenter], skeletons[0].Joints[JointType.ShoulderCenter]) / headSize;
int correctProportions = 0;
for (int i = 1; i < currentRatios.Length; i++)
{
diff = currentRatios[i] - humanRatios[i];
if (abs(diff) <= MaximumDiff)//I used .2 for my MaximumDiff
correctProportions++;
}
if (correctProportions >= 2)
PersonDetected = true;
Another method I had success with was finding the average of the sum of the joints distance squared from one another. I found that non-human detections had more variable summed distances, whereas humans are more consistent. The average I learned using a single dimensional support vector machine (I found user's summed distances were generally less than 9)
//in AllFramesReady or SkeletalFrameReady
Skeleton data;
...
float lastPosX = 0; // trying to detect false-positives
float lastPosY = 0;
float lastPosZ = 0;
float diff = 0;
foreach (Joint joint in data.Joints)
{
//add the distance squared
diff += (joint.Position.X - lastPosX) * (joint.Position.X - lastPosX);
diff += (joint.Position.Y - lastPosY) * (joint.Position.Y - lastPosY);
diff += (joint.Position.Z - lastPosZ) * (joint.Position.Z - lastPosZ);
lastPosX = joint.Position.X;
lastPosY = joint.Position.Y;
lastPosZ = joint.Position.Z;
}
if (diff < 9)//this is what my svm learned
PersonDetected = true;
Use player IDs and indexes to remember who is who
This ties in with the previous issue, where if Kinect switched the two users that it was tracking to others, then my application would crash because of the sudden changes in data. To counter this, I would keep track of both each player's skeletal index and their player ID. To learn more about how I did this, see Kinect user Detection.
Add adjustable parameters to adopt to varying situations
Where I was presenting, the same tilt angle and other basic kinect parameters (like near-mode) did not work in the new environment. Let the user be able to adjust some of these parameters so they can get the best setup for the job.
Expect people to do stupid things
The next time I presented, I had adjustable tilt, and you can guess whether someone burned out the Kinect's motor. Anything that can be broken on Kinect, someone will break. Leaving a warning in your documentation will not be sufficient. You should add in cautionary checks on Kinect's hardware to make sure people don't get upset when they break something inadvertently. Here is some code checking whether the user has used the motor more than 20 times in two minutes.
int motorAdjustments = 0;
DateTime firstAdjustment;
...
//in motor adjustment code
if (motorAdjustments == 0)
firstAdjustment = DateTime.Now;
++motorAdjustments;
if (motorAdjustments < 20)
{
//adjust the tilt
}
else
{
DateTime timeCheck = firstAdjustment;
if (DateTime.Now > timeCheck.AddMinutes(2))
{
//reset all variables
motorAdjustments = 1;
firstAdjustment = DateTime.Now;
//adjust the tilt
}
}
I would note that all of these were issues for me with the first version of Kinect, and I don't know how many of them have been solved in the second version as I sadly haven't gotten my hands on one yet. However I would still implement some of these techniques if not back-up techniques because there will be exceptions, especially in computer vision.
In my project i have recorded sound using mediaplayer and save as .3gp file but when i want to play it using some audio effect or fast forwarding or change pitch of audio while playing. i have used mediaplayer but not working.then i used audiotrack but audiotrack takes only bytestream as input to play. i just want to play .3gp file and change pitch while playing.. i use this one below.
Help me...thanks in advance...
public void play() {
File path = new File(
Environment.getExternalStorageDirectory().getAbsolutePath()
+ "/sdcard/meditest/");
File[] f=path.listFiles();
isPlaying=true;
int bufferSize = AudioTrack.getMinBufferSize(outfrequency,
channelConfigurationout, audioEncoding);
short[] audiodata = new short[bufferSize];
try {
DataInputStream dis = new DataInputStream(
new BufferedInputStream(new FileInputStream(
f[0])));
audioTrack = new AudioTrack(
AudioManager.STREAM_MUSIC, outfrequency,
channelConfigurationout, audioEncoding, bufferSize,
AudioTrack.MODE_STREAM);
audioTrack.setPlaybackRate((int) (frequency*1.5));
AudioManager audioManager = (AudioManager)this.getSystemService(Context.AUDIO_SERVICE);
// Set the volume of played media to maximum.
audioTrack.setStereoVolume(1.0f,1.0f);
Log.d("Clapper","player start");
audioTrack.play();
while (isPlaying && dis.available() > 0) {
int i = 0;
while (dis.available() > 0 && i < audiodata.length) {
audiodata[i] = dis.readShort();
i++;
if(i/50==0)
Log.d("Clapper", "playing now"+i);
}
audioTrack.write(audiodata, 0, audiodata.length);
}
Log.d("Clapper","AUDIO LENGTH: "+String.valueOf(audiodata));
dis.close();
audioTrack.stop();
} catch (Throwable t) {
Log.e("AudioTrack", "Playback Failed");
}
Log.d("Clapper","AUDIO state: "+String.valueOf(audioTrack.getPlayState()));
talkAnimation.stop();
if(audioTrack.getPlayState()!=AudioTrack.PLAYSTATE_PLAYING)
{
runOnUiThread(new Runnable() {
public void run() {
// TODO Auto-generated method stub
imgtalk.setBackgroundResource(R.drawable.talk1);
}
});
}
}
I tried library called Sonic. Its basically for Speech as it use PSOLA algo to change pitch and tempo.
Sonic Library
i got your problem .Media player does not support cha
Consider using a SoundPool
http://developer.android.com/reference/android/media/SoundPool.html
It supports changing the pitch in realtime while playing
The playback rate can also be changed. A playback rate of 1.0 causes the sound to play at its original frequency (resampled, if necessary, to the hardware output frequency). A playback rate of 2.0 causes the sound to play at twice its original frequency, and a playback rate of 0.5 causes it to play at half its original frequency. The playback rate range is 0.5 to 2.0.
Once the sounds are loaded and play has started, the application can trigger sounds by calling SoundPool.play(). Playing streams can be paused or resumed, and the application can also alter the pitch by adjusting the playback rate in real-time for doppler or synthesis effects.
http://developer.android.com/reference/android/media/SoundPool.html#setRate(int, float)
IF you want to change pitch while playing sound you have to use sound pool .this is the best way to do this.you can fast forward your playing by some amount and see you feel that pitch has been changed.
I want to do a simple kinect aplication in processing, I just want to when kinect detect a skeleton, show a simple jpeg image, just that. I wrote some code, all works but when someone appears in front of kinect, nothing happens, can anyone help me?
This is my code:
import SimpleOpenNI.*;
SimpleOpenNI kinect;
void setup()
{
// Começar o evento
kinect = new SimpleOpenNI(this);
// Ativar o RGB
kinect.enableRGB();
background(200,0,0);
// Criar a janela do tamanho do dephMap
size(kinect.rgbWidth(), kinect.rgbHeight());
}
void draw()
{
// update da camera
kinect.update();
// mostrar o depthMap
image(kinect.rgbImage(),0,0);
// Definir quantidade de pessoas
int i;
for (i=1; i<=10; i++)
{
// Verificar presença da pessoa
if(kinect.isTrackingSkeleton(i))
{
mostrarImagem(); // draw the skeleton
}
}
}
// Mostrar a imagem
void mostrarImagem()
{
PImage img;
img = loadImage("proverbio1.jpg");
image(img, 0, 0);
}
You haven't setup the callbacks for OpenNI user events.
Also if you simply want to display an image when someone is detected, you don't actually need to track the skeleton: simply use the scene image. You can get some information about the user's position without tracking the skeleton, like the user's centre of mass.
This way you'd have a simpler and faster application if you don't actually need skeleton data.
Here a basic example:
import SimpleOpenNI.*;
SimpleOpenNI context;//OpenNI context
PVector pos = new PVector();//this will store the position of the user
int user;//this will keep track of the most recent user added
PImage sample;
void setup(){
size(640,480);
context = new SimpleOpenNI(this);//initialize
context.enableScene();//enable features we want to use
context.enableUser(SimpleOpenNI.SKEL_PROFILE_NONE);//enable user events, but no skeleton tracking, needed for the CoM functionality
sample = loadImage("proverbio1.jpg");
}
void draw(){
context.update();//update openni
image(context.sceneImage(),0,0);
if(user > 0){//if we have a user
context.getCoM(user,pos);//store that user's position
println("user " + user + " is at: " + pos);//print it in the console
image(sample,0,0);
}
}
//OpenNI basic user events
void onNewUser(int userId){
println("detected" + userId);
user = userId;
}
void onLostUser(int userId){
println("lost: " + userId);
user = 0;
}
You can see some handy SimpleOpenNI samples in this Kinect article which is part of a workshop I held last year.