Htc Vive + leap motion as controller - leap-motion

I'm trying to implement a feature similar to HTC Vive's controller with the leap motion on my Unity project. I wanted to generate a laser pointer from the index finger and teleport the Vive's room on the position of the laser (as it's done with the controller). The problem is the latest leap motion (orion) documentation, it's unclear. Any ideas how to do that? More in general, we thought about using HandController but we don't understand where to add the script component.
Thanks!

It's unclear to me whether the problem you're having is getting hand data in your scene at all, or using that hand data.
If you're just trying to get hand data in your scene, you can copy a prefab from one of the Unity SDK's example scenes. If you're trying to integrate Leap into an existing scene that already has a VR rig set up, check out the documentation on the core Leap components to understand what pieces need to be in place for you to start getting Hand data. LeapServiceProvider has to be somewhere in your scene to receive hand data.
As long as you have a LeapServiceProvider somewhere, you can access hands from the Leap Motion from any script, anywhere. So for getting a ray from the index fingertip, just pop this script any old place:
using Leap;
using Leap.Unity;
using UnityEngine;
public class IndexRay : MonoBehaviour {
void Update() {
Hand rightHand = Hands.Right;
Vector3 indexTipPosition = rightHand.Fingers[1].TipPosition.ToVector3();
Vector3 indexTipDirection = rightHand.Fingers[1].bones[3].Direction.ToVector3();
// You can try using other bones in the index finger for direction as well;
// bones[3] is the last bone; bones[1] is the bone extending from the knuckle;
// bones[0] is the index metacarpal bone.
Debug.DrawRay(indexTipPosition, indexTipDirection, Color.cyan);
}
}
For what it's worth, the index fingertip direction is probably not going to be stable enough to do what you want. A more reliable strategy is to cast a line from the camera (or a theoretical "shoulder position" at a constant offset from the camera) through the index knuckle bone of the hand:
using Leap;
using Leap.Unity;
using UnityEngine;
public class ProjectiveRay : MonoBehaviour {
// To find an approximate shoulder, let's try 12 cm right, 15 cm down, and 4 cm back relative to the camera.
[Tooltip("An approximation for the shoulder position relative to the VR camera in the camera's (non-scaled) local space.")]
public Vector3 cameraShoulderOffset = new Vector3(0.12F, -0.15F, -0.04F);
public Transform shoulderTransform;
void Update() {
Hand rightHand = Hands.Right;
Vector3 cameraPosition = Camera.main.transform.position;
Vector3 shoulderPosition = cameraPosition + Camera.main.transform.rotation * cameraShoulderOffset;
Vector3 indexKnucklePosition = rightHand.Fingers[1].bones[1].PrevJoint.ToVector3();
Vector3 dirFromShoulder = (indexKnucklePosition - shoulderPosition).normalized;
Debug.DrawRay(indexKnucklePosition, dirFromShoulder, Color.white);
Debug.DrawLine(shoulderPosition, indexKnucklePosition, Color.red);
}
}

Related

How do I gradually apply velocity in Spigot?

I am using player.setVelocity(player.getLocation().getDirection().multiply(Main.instance.getConfig().getDouble("velocity_multiplier")).setY(Main.instance.getConfig().getInt("Y_axis"))); to set velocity to a player. It allows high configuration of movement via config, but the problem is that when you set it too high, Spigot blocks it. I do not want to enable:
server.properties: allow_flight.
So how can I avoid this? I bumped up the multiplier to 30 just for a test, and it would start to move you, glitch, and pull you back down. It also says that the player moved too quickly in console even from smaller amounts of velocity. I was thinking of making it gradually apply the velocity. When you jump, it applies the starting velocity and as you go it goes higher(Y_axis) and farther(velocity_multiplier), but I do not know how to do that.
You can enable just for the player before applying the velocity and in a delayed task disabled it
public void blabla(Player player){
player.setAllowFlight(true);
player.setVelocity(player.getLocation().getDirection().multiply(Main.instance.getConfig().getDouble("velocity_multiplier")).setY(Main.instance.getConfig().getInt("Y_axis")));
new BukkitRunnable() {
#Override
public void run() {
player.setAllowFlight(false);
}
}.runTaskLater(this, 20 * 5);
}
In the code I used 20 * 5 to disable the flight after 5 seconds, you can change it to what you want.
Beyond code, you likely would be best situated to address your issue by allowing flight in the Spigot file and installing or developing an anti-cheat in the game. Spigot's flight protection works poorly with many plugins and does not successfully block many players who attempt to fly.
Best advice would be to look beyond a makeshift code solution and rather create your own anti-fly.
The maximum velocity in bukkit (and spigot) is 10 blocks per tick. This is all directions combined.
If your initial velocity is to high, you can use the scheduler to repeatedly calculate the next velocity.
To calculate this, we need some magic values: The following values come from The Minecraft Wiki.
private final static double DECELERATION_RATE = 0.98D;
private final static double GRAVITY_CONSTANT = 0.08D;
private final static double VANILA_ANTICHEAT_THRESHOLD = 9.5D; // actual 10D
We first need to calculate the spot the player would reach using those speeds, and then teleport him while applying the velocity for the first part.
We are going to use a BukkitRunnable to run a task that calculates the above:
Vector speed = ...;
Player player = ...;
new BukkitRunnable() {
double velY = speed.getY();
Location locCached = new Location(null,0,0,0);
#Override
public void run() {
if (velY > VANILA_ANTICHEAT_THRESHOLD) {
player.getLocation(locCached).setY(locCached.getY() + velY);
player.teleport(locCached);
player.setVelocity(new Vector(0,ANILA_ANTICHEAT_THRESHOLD,0));
} else {
player.setVelocity(new Vector(0,velY,0));
this.cancel();
}
velY -= GRAVITY_CONSTANT;
velY *= DECELERATION_RATE;
}
}.runTaskTimer(plugin,0,1);
The above code will then handle the velocity problems for us and can be used in place of setVelocity.

Unity Input Touch issue

Could you please advise how i would go about using the input touch function in Unity to make an object changes its x direction every time the user tap on the screen. For example, for 2d setting game, an object is moving forward (to the right) in the x position, if the user tap then the object would move backward in the x position (to the left). Sorry no code is produced.
It's simple as your name "tony" :)
What you can do is to make a simple script which'd move your object to left and right. And on screen touch you can easily change its direction by just -1 multiplication.
Simple script that you can attach to your object.
using UnityEngine;
using System.Collections;
public class MoveObject : MonoBehaviour
{
float _limit = 5;
// 1 for right and -1 for left.
float _direction = 1;
// You can call it as speed
float _speed = 0.01f;
void Start ()
{
}
void Update ()
{
transform.position = Vector3.MoveTowards (transform.position, new Vector3 (transform.position.x + _direction, transform.position.y, transform.position.z), _speed);
if (Input.GetMouseButtonDown (0))
_direction *= -1;
}
}
Hope this helps :)

Microsoft Kinect and background/environmental noise

I am currently programming with the Microsoft Kinect for Windows SDK 2 on Windows 8.1. Things are going well, and in a home dev environment obviously there is not much noise in the background compared to the 'real world'.
I would like to seek some advice from those with experience in 'real world' applications with the Kinect. How does Kinect (especially v2) fare in a live environment with passers-by, onlookers and unexpected objects in the background? I do expect, in the space from the Kinect sensor to the user there will usually not be interference however - what I am very mindful of right now is the background noise as such.
While I am aware that the Kinect does not track well under direct sunlight (either on the sensor or the user) - are there certain lighting conditions or other external factors I need to factor into the code?
The answer I am looking for is:
What kind of issues can arise in a live environment?
How did you code or work your way around it?
Outlaw Lemur has descibed in detail most of the issues you may encounter in real-world scenarios.
Using Kinect for Windows version 2, you do not need to adjust the motor, since there is no motor and the sensor has a larger field of view. This will make your life much easier.
I would like to add the following tips and advice:
1) Avoid direct light (physical or internal lighting)
Kinect has an infrared sensor that might be confused. This sensor should not have direct contact with any light sources. You can emulate such an environment at your home/office by playing with an ordinary laser pointer and torches.
2) If you are tracking only one person, select the closest tracked user
If your app only needs one player, that player needs to be a) fully tracked and b) closer to the sensor than the others. It's an easy way to make participants understand who is tracked without making your UI more complex.
public static Body Default(this IEnumerable<Body> bodies)
{
Body result = null;
double closestBodyDistance = double.MaxValue;
foreach (var body in bodies)
{
if (body.IsTracked)
{
var position = body.Joints[JointType.SpineBase].Position;
var distance = position.Length();
if (result == null || distance < closestBodyDistance)
{
result = body;
closestBodyDistance = distance;
}
}
}
return result;
}
3) Use the tracking IDs to distinguish different players
Each player has a TrackingID property. Use that property when players interfere or move at random positions. Do not use that property as an alternative to face recognition though.
ulong _trackinfID1 = 0;
ulong _trackingID2 = 0;
void BodyReader_FrameArrived(object sender, BodyFrameArrivedEventArgs e)
{
using (var frame = e.FrameReference.AcquireFrame())
{
if (frame != null)
{
frame.GetAndRefreshBodyData(_bodies);
var bodies = _bodies.Where(b => b.IsTracked).ToList();
if (bodies != null && bodies.Count >= 2 && _trackinfID1 == 0 && _trackingID2 == 0)
{
_trackinfID1 = bodies[0].TrackingId;
_trackingID2 = bodies[1].TrackingId;
// Alternatively, specidy body1 and body2 according to their distance from the sensor.
}
Body first = bodies.Where(b => b.TrackingId == _trackinfID1).FirstOrDefault();
Body second = bodies.Where(b => b.TrackingId == _trackingID2).FirstOrDefault();
if (first != null)
{
// Do something...
}
if (second != null)
{
// Do something...
}
}
}
}
4) Display warnings when a player is too far or too close to the sensor.
To achieve higher accuracy, players need to stand at a specific distance: not too far or too close to the sensor. Here's how to check this:
const double MIN_DISTANCE = 1.0; // in meters
const double MAX_DISTANCE = 4.0; // in meters
double distance = body.Joints[JointType.SpineBase].Position.Z; // in meters, too
if (distance > MAX_DISTANCE)
{
// Prompt the player to move closer.
}
else if (distance < MIN_DISTANCE)
{
// Prompt the player to move farther.
}
else
{
// Player is in the right distance.
}
5) Always know when a player entered or left the scene.
Vitruvius provides an easy way to understand when someone entered or left the scene.
Here is the source code and here is how to use it in your app:
UsersController userReporter = new UsersController();
userReporter.BodyEntered += UserReporter_BodyEntered;
userReporter.BodyLeft += UserReporter_BodyLeft;
userReporter.Start();
void UserReporter_BodyEntered(object sender, UsersControllerEventArgs e)
{
// A new user has entered the scene. Get the ID from e param.
}
void UserReporter_BodyLeft(object sender, UsersControllerEventArgs e)
{
// A user has left the scene. Get the ID from e param.
}
6) Have a visual clue of which player is tracked
If there are a lot of people surrounding the player, you may need to show on-screen who is tracked. You can highlight the depth frame bitmap or use Microsoft's Kinect Interactions.
This is an example of removing the background and keeping the player pixels only.
7) Avoid glossy floors
Some floors (bright, glossy) may mirror people and Kinect may confuse some of their joints (for example, Kinect may extend your legs to the reflected body). If you can't avoid glossy floors, use the FloorClipPlane property of your BodyFrame. However, the best solution would be to have a simple carpet where you expect people to stand. A carpet would also act as an indication of the proper distance, so you would provide a better user experience.
I created an application for home use like you have before, and then presented that same application in a public setting. The result was embarrassing for me, because there were many errors that I would never have anticipated within a controlled environment. However that did help me because it led me to add some interesting adjustments to my code, which is centered around human detection only.
Have conditions for checking the validity of a "human".
When I showed my application in the middle of a presentation floor with many other objects and props, I found that even chairs could be mistaken for people for brief moments, which led to my application switching between the user and an inanimate object, causing it to lose track of the user and lost their progress. To counter this or other false-positive human detections, I added my own additional checks for a human. My most successful method was comparing the proportions of a humans body. I implemented this measured in head units. (head units picture) Below is code of how I did this (SDK version 1.8, C#)
bool PersonDetected = false;
double[] humanRatios = { 1.0f, 4.0, 2.33, 3.0 };
/*Array indexes
* 0 - Head (shoulder to head)
* 1 - Leg length (foot to knee to hip)
* 2 - Width (shoulder to shoulder center to shoulder)
* 3 - Torso (hips to shoulder)
*/
....
double[] currentRatios = new double[4];
double headSize = Distance(skeletons[0].Joints[JointType.ShoulderCenter], skeletons[0].Joints[JointType.Head]);
currentRatios[0] = 1.0f;
currentRatios[1] = (Distance(skeletons[0].Joints[JointType.FootLeft], skeletons[0].Joints[JointType.KneeLeft]) + Distance(skeletons[0].Joints[JointType.KneeLeft], skeletons[0].Joints[JointType.HipLeft])) / headSize;
currentRatios[2] = (Distance(skeletons[0].Joints[JointType.ShoulderLeft], skeletons[0].Joints[JointType.ShoulderCenter]) + Distance(skeletons[0].Joints[JointType.ShoulderCenter], skeletons[0].Joints[JointType.ShoulderRight])) / headSize;
currentRatios[3] = Distance(skeletons[0].Joints[JointType.HipCenter], skeletons[0].Joints[JointType.ShoulderCenter]) / headSize;
int correctProportions = 0;
for (int i = 1; i < currentRatios.Length; i++)
{
diff = currentRatios[i] - humanRatios[i];
if (abs(diff) <= MaximumDiff)//I used .2 for my MaximumDiff
correctProportions++;
}
if (correctProportions >= 2)
PersonDetected = true;
Another method I had success with was finding the average of the sum of the joints distance squared from one another. I found that non-human detections had more variable summed distances, whereas humans are more consistent. The average I learned using a single dimensional support vector machine (I found user's summed distances were generally less than 9)
//in AllFramesReady or SkeletalFrameReady
Skeleton data;
...
float lastPosX = 0; // trying to detect false-positives
float lastPosY = 0;
float lastPosZ = 0;
float diff = 0;
foreach (Joint joint in data.Joints)
{
//add the distance squared
diff += (joint.Position.X - lastPosX) * (joint.Position.X - lastPosX);
diff += (joint.Position.Y - lastPosY) * (joint.Position.Y - lastPosY);
diff += (joint.Position.Z - lastPosZ) * (joint.Position.Z - lastPosZ);
lastPosX = joint.Position.X;
lastPosY = joint.Position.Y;
lastPosZ = joint.Position.Z;
}
if (diff < 9)//this is what my svm learned
PersonDetected = true;
Use player IDs and indexes to remember who is who
This ties in with the previous issue, where if Kinect switched the two users that it was tracking to others, then my application would crash because of the sudden changes in data. To counter this, I would keep track of both each player's skeletal index and their player ID. To learn more about how I did this, see Kinect user Detection.
Add adjustable parameters to adopt to varying situations
Where I was presenting, the same tilt angle and other basic kinect parameters (like near-mode) did not work in the new environment. Let the user be able to adjust some of these parameters so they can get the best setup for the job.
Expect people to do stupid things
The next time I presented, I had adjustable tilt, and you can guess whether someone burned out the Kinect's motor. Anything that can be broken on Kinect, someone will break. Leaving a warning in your documentation will not be sufficient. You should add in cautionary checks on Kinect's hardware to make sure people don't get upset when they break something inadvertently. Here is some code checking whether the user has used the motor more than 20 times in two minutes.
int motorAdjustments = 0;
DateTime firstAdjustment;
...
//in motor adjustment code
if (motorAdjustments == 0)
firstAdjustment = DateTime.Now;
++motorAdjustments;
if (motorAdjustments < 20)
{
//adjust the tilt
}
else
{
DateTime timeCheck = firstAdjustment;
if (DateTime.Now > timeCheck.AddMinutes(2))
{
//reset all variables
motorAdjustments = 1;
firstAdjustment = DateTime.Now;
//adjust the tilt
}
}
I would note that all of these were issues for me with the first version of Kinect, and I don't know how many of them have been solved in the second version as I sadly haven't gotten my hands on one yet. However I would still implement some of these techniques if not back-up techniques because there will be exceptions, especially in computer vision.

Unity: camera falls through terrain

Problem is:
I have created terrain and I need to fly over terrain with Camera. I added to Camera "Mouse Look" script, RigidBody: usegravity - unchecked and I have added my code in Update method:
float vert = Input.GetAxis("Vertical");
float hor = Input.GetAxis("Horizontal");
if (vert != 0)
{
if (!Physics.Raycast(this.transform.position, this.transform.forward, 5))
{
transform.Translate(Vector3.forward * flySpeed * vert);
}
else
{
transform.Translate(Vector3.up * flySpeed * vert);
}
}
if (hor != 0)
{
if (!Physics.Raycast(this.transform.position, this.transform.forward, 5))
{
transform.Translate(Vector3.right * flySpeed * hor);
}
else
{
transform.Translate(Vector3.up * flySpeed* hor);
}
}
if (Input.GetKey(KeyCode.E))
{
transform.Translate(Vector3.up * flySpeed);
}
else if (Input.GetKey(KeyCode.Q))
{
Vector3 v = Vector3.down * flySpeed;
if (!Physics.Raycast(this.transform.position, this.transform.forward, 5))
{
transform.Translate(v);
}
}
But sometimes then i go down - Q - camera goes through terrain. Why?
Also looks ugly if you are moving with camera forward as low as possible over terrain and camera does not fall through it - it starts to jump. Also why?
Make sure you have a Terrain Collider on your terrain.
In addition to S.Richmonds answer, you can add a character controller or other similar collider-component object to your camera.
See this answer in the unity questions network:
http://answers.unity3d.com/questions/45763/getting-camera-to-not-see-under-the-ground.html
The Update() method in a monobehavior gets called once each fram. Because the rate which update is called is dependent on frame rate, moving an object by a constant value in Update() can result in inconsistant motion. This can be corrected by multiplying a constant speed by Time.deltaTime, which is the time in seconds since the last frame was rendered. This will fix the fallthrough unless flySpeed is set too high (where the change in position each frame is greater than the collider's size). Additionally as suggested above, using a CharacterController without a rigidbody would be better suited to this situation. Rigidbodies are for objects primarily controlled by physics, while the CharacterController is for objects controlled by scripts.

Using Kinect to calculate distance traveled

I'm trying to develop what seems to be a simple program that uses the Kinect for Xbox360 to calculate the distance traveled by a person. The room that the Kinect will be pointed at will be 10 x 10. After the user presses the button, the subject will move about this space. Once the subject reaches their final destination in the area, the user will press the button again. The Kinect will then output how far the subject traveled in between both button presses. Having never developed for the Kinect before, it's been pretty daunting to get started. My issue is that I'm not entirely sure what I should be using to measure the distance. In my research, I've found ways to calculate the distance an object is FROM the Kinect but that's about it.
What you have hear is a simple question of dealing with a Cartesian plane. The Kinect has 20 joints that exist in XYZ space, and distance is measured in meters. In order to access these joints, you have these statements inside a "Tracker" class (this is C#... not sure if you're using C# or C++ in the SDK):
public Tracker(KinectSensor sn, MainWindow win, string fileName)
{
window = win;
sensor = sn;
try
{
sensor.Start();
}
catch (IOException)
{
sensor = null;
MessageBox.Show("No Kinect sensor found. Please connect one and restart the application", "*****ERROR*****");
return;
}
sensor.SkeletonFrameReady += SensorSkeletonFrameReady; //Frame handlers
sensor.ColorFrameReady += SensorColorFrameReady;
sensor.SkeletonStream.Enable();
sensor.ColorStream.Enable();
}
These access the color and skeleton streams from the Kinect. The skeleton stream contains the joints, so you focus on that with these statements:
//Start sending skeleton stream
private void SensorSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
//Access the skeleton frame
using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
{
if (skeletonFrame != null)
{
//Check to see if there is any data in the skeleton
if (this.skeletons == null)
//Allocate array of skeletons
this.skeletons = new Skeleton[skeletonFrame.SkeletonArrayLength];
//Copy skeletons from this frame
skeletonFrame.CopySkeletonDataTo(this.skeletons);
//Find first tracked skeleton, if any
Skeleton skeleton = this.skeletons.Where(s => s.TrackingState == SkeletonTrackingState.Tracked).FirstOrDefault();
if (skeleton != null)
{
//Initialize joints
///<summary>
///Joints to be displayed, projected, recorded, etc.
///</summary>
Joint leftFoot = skeleton.Joints[JointType.FootLeft];
}
}
So, at the beginning of your program, you want to pick a joint (there are 20... choose one that will ALWAYS be facing towards the Kinect when you are executing the program) and get its location with something like the following statements:
if(skeleton.Joints[JointType.FootLeft].TrackingState == JointTrackingState.Tracked)
{
double xPosition = skeleton.Joints[JointType.FootLeft].Position.X;
double yPosition = skeleton.Joints[JointType.FootLeft].Position.Y;
double zPosition = skeleton.Joints[JointType.FootLeft].Position.Z;
}
At the end, you'll want to have a slight delay before you stop the stream... some time between the click and when you shut off the stream from the Kinect. You will then do the math you need to do to get the distance between the two points. If you don't have the delay, you won't be able to get your Cartesian point.