How to map kinect skeleton data to a model? - kinect

I have set up a Kinect device and written a simple program that reads the stream to a QImage using OpenNI 2.0. I have set up skeleton tracking with NiTE 2.0, so I have access to the coordinates of all the 15 joints. I have also set up a simple scene using SceniX. The hand coordinates provided by the skeleton tracking are beeing used to draw 2 boxes to represent the hands.
I would like to bind the whole skeleton to a (rigged)model, and cant seem to find any good tutorials. Anyone have any idea how I should proceed?

depending on your requirements you could look at something like this for Unity Engine https://www.assetstore.unity3d.com/en/#!/content/10693
There is also a Plugin for the Unreal 4 Engine called KINECT 4 UNREAL FROM OPAQUE MULTIMEDIA
But if you have to write it all by hand for yourself, i have done something similar using OpenGL.
I used Assimp http://assimp.sourceforge.net/ to be able to load animated Collada models and OpenNi with NiTE for skeletal tracking. I then used the rotation data from the Nite skeleton and applied it to the corresponding Bones of my rigged mesh, overwriting the rotation values of the animation. Don't use positional Data. It will strech your bones and distort the mesh.
There are many sources for free 3D Models, like TF3DM.com . I for myself used a custom Rig for my models to be suitable for my code. So you might look into using Blender and how to Rig a Model.
Also remember that the Nite Skeleton has no joint for the Pelvis, and that Nite joints don't inherit their parents rotation, contrary to the bones in a rigged model.
I hope this helps to have something to go on.

You can try DigitalRune, they have examples of binding a rigged model to joints. They have mentioned some examples too. try http://www.digitalrune.com/Support/Blog/tabid/719/EntryId/155/Research-Augmented-Reality-with-Microsoft-Kinect.aspx
Also you would need to know to animate model in blender and export it to XNA or to your working graphics framework. Eg:-http://www.codeproject.com/Articles/230540/Animating-single-bones-in-a-Blender-3D-model-with#SkinningSampleProject132

Related

Hide an object for a specific camera

I use godot to create my 3d game. I ran into a problem while creating portals using camera viewport rendering to texture. The problem is that the camera captures unnecessary objects that are behind portal. I partially solved this problem by setting the parameter "near " for the camera at a distance from the camera itself to the portal, but the part behind the portal began to be cut off.
The question is, is it possible to hide objects for a particular camera so that other cameras can see them? Perhaps there is another way to do this, for example by creating a static clipping plane?
Proximity Fade
Probably not what you are looking for, but I'll mention it for completeness sake.
The default material has proximity fade and distance fade, which you can use to make the material disappear if it is too close or to distant from the camera, respectively.
It is important to note that this is not a cull plane, and that the fading is gradual.
Thus, using proximity fade you can make objects near the camera appear semitransparent.
Using Visibility layers and cull mask
is it possible to hide objects for a particular camera so that other cameras can see them?
Every VisualInstance (you know, all things that are visible in 3D) has layers. And every Camera has a cull_mask. If the cull_mask of the Camera does not include any of the layers of a VisualInstance, then the Camera does not see that VisualInstance.
A VisualInstance with no layers will not show on no Camera, even if the Camera has all the layers in its cull_mask (which is the default).
You can either edit the cull_mask of the camera to not include the layers of the VisualInstance, or edit the layers of the VisualInstance, or both.
Using a custom shader cull plane
Perhaps there is another way to do this, for example by creating a static clipping plane?
You can use a custom spatial shader to cut things out based on a plane.
You need to define the plane as a uniforms. For this answer I'll use a point-normal definition of a plane:
n·(r - r_0)
That is:
dot(plane_normal, (world_position - plane_point)
Thus, we define a plane_normal and plane_point uniforms:
uniform vec3 plane_normal;
uniform vec3 plane_point;
The plane_normal gives us the orientation of the plane, while the plane_point is a point on the plane which allows us to position it.
And then use this logic:
vec3 wold_position = (CAMERA_MATRIX * vec4(VERTEX, 1.0)).xyz;
ALPHA = clamp(sign(dot(plane_normal, wold_position - plane_point)), 0.0, 1.0);
Here we are converting the coordinates of the current point to world space, and then using definition of the plane to find the points on one side (using sign), and set ALPHA based on that, such that everything on one side of the plane becomes invisible.
Note: This is not the only way to define the plane. Another popular definition is a 4D vector, where the xyz are the normal, and the w is the distance from plane to the origin.
Sadly, I don't think there is a way to make this work with multiple material passes, because ALPHA controls the blending of the passes, and will not result in transparency. And no, using discard; does not solve it either, because the other passes can write the fragment regardless. Thus, you are going to need to modify your materials to include that.
Further Sadly Godot 3.x does not support global uniforms (see Godot 4.0 gets global and per-instance shader uniforms). Which means you will have to set these parameter everywhere you need them.
Using Constructive Solid Geometry (CSG)
Add a CSGCombiner make the geometry that needs to disappear with other CSG nodes as children.
Then you can, for example, add a CSGSphere with operation set to "Subtraction", and move it with the Camera (for this purpose, I suggest to add a RemoteTransform node as child to the Camera and set its remote path to the CSGSphere).
Of course, it does not have to be a CSGSphere, you can use any CSG nodes for this purpose. For the portal, I imagine you could use a CSGBox and align it to the portal plane.
Note: Currently on Godot 3.3 CSG nodes do not support baking lights. This is a regression. See: Unable to bake lightmap with CSG due to the lack of ability to generate UV2 for CSG nodes.
Portals, actually
Bartleby Lawnjelly has a portal (godot-lportal) module for Godot 3.x.
Being a module, they require to build Godot from source. See Compiling on the official Godot documentation. It is not that bad, I promise. Or use build from godot-titan.
I have to explain that these portals are not portals in the Valve Portal video game series sense… The module lets you define areas as "rooms", and planes as "portals" that connect those rooms, in such way that you can look from one to the other. The purpose of this is to cull entire rooms unless you are looking through one of the portals.
Hopefully that makes more sense with a video. This is a somewhat old one, but good to get the idea across: Portal rendering module in Godot 3.2 - Improved performance. Seeing shadow pooping in the video? Bartleby Lawnjelly also has a custom lightmapper.

Godot doesn't import glTF material shaders correctly

I have a model that I am using as my gameworld in Godot and I am putting some textures on it in Blender. When exporting the model to .glb it seems to not be exporting some of the materials correctly.
The ones that are an image texture that I scaled using mapping.
It just skips the first two nodes. When I disconnect them the material looks like it does in Godot.
What it should look like (what it looks like in Blender):
What it looks like in Godot:
There also are some materials that use mix shaders to mix images together. These are also skipped and it just uses one of the two instead of mixing them.
What it should look like:
What it looks like in Godot:
The F is from an image with text that is overlayed ontop of the other image.
I am not really sure if these issues lie with the glTF format and certain shaders just don't work with it or that Godot doesn't want to import them.
I hope someone can help me with this.
Thanks in advance and have a nice day,
Rover
In general, arbitrary node graphs cannot be exported from Blender — see https://blender.stackexchange.com/a/57541/43930 for the long explanation of that. To ensure that the material will export correctly, you'll need to refer to the documentation for the Blender glTF exporter and configure your material accordingly. For more complicated effects, you can also bake Blender nodes down to simpler textures.

SharpDX How To Render a 3D Environment

I just started coding some basics in SharpDX (VB.net) and I already got it to Render a 2D triangle. And I know how to render other 2D stuff, but I want to create something in 3D where I'm able to rotate the camera around some cubes. I tried it, but failed at converting the 3D Space to screen coordinates. Now Here are my Questions:
How can I calculate a Matrix for Perspective projection?
How can I pass that Matrix to my Vertex Shader
How can I make the Camera rotate around the Objects when I drag the mouse over the screen?
Please explain these things to me and give some code examples. I'm just a Beginner in SharpDX and everything I found was just not understandable for me.
A few things you can do when you first start.
Firstly, there are some great examples you can leverage (Even in c# but you need VB) that you can use to learn from.
I suggest you look at this within the Sharpdx repository. Sharpdx direct 3d 11 samples
Within these examples (especially triangle example), it goes through the basics including setting up the device, the creation of simple resources to bind to your GPU and compiling the bytecode.
The samples though use the effects methodology, which is deprecated and as such once you become familiar with compiling code, I would advise moving away from this paradigm.
The more advanced examples will show you how to set up your matrices.
The last item you wanted to know about is mouse movement. I would advise just having a look at MSDN around mousemove events. You will need to bind one to your window/control and then read the deltas. Use those deltas to create your rotation/movement based upon this. Look into Vector3 (sharpdx), basically, you need to do this all in vector space and then create the various translation/rotation matrices from this.
Hope this is start.

Implementing a 3D Minimap for VR in Unreal Engine

I am working on a VR tower defense game. I want to place my towers on a small map of the field and at the same time show the field/units/towers on the small map, in 3D. Like this:
http://halo.bungie.net/images/games/Halo3ODST/imagery/screenshots/H3ODST_PreparetoDropCinematic.jpg
The map would be something like a small clone of the field. Is there a way to do so with camera etc. So that my minimap is just a re-render/clone of the field.
Sorry if this is the wrong place, but the Unreal Engine Forum is not working at the moment.
I dont know of a simple camera projection to 3D (since the camera map would be about scene capture onto 2D textures)
You can do a fake/tricky camera implementation tho. You just show a normal camera projection and put it into the world. Now you let the actual camera position depend on where the users location is in relation to the map.
You can cover this up with particle effects so it doesnt look like some TV that is following you....
depending on the complexity of your game and how u like the solution obove it might be better to implement a calculated model.
Like you have miniature models of everything u want to show on the map and then let a map class project every "object that is shown on map" the miniature version onto the map object.
Ofc this requires tons of work compared to just setting up a camera but you have alot of better control of how the map looks and what it can do (u could add functionallity, control options and special views etc....)

How to scale a Kinect Skeleton to match to other Kinect Skeleton

I have some saved skeleton data, the data corresponds to my own skeleton proportion, lets call this skeleton Sk1.
And i have other skeleton data (that corresponds to a ten old year child, by example). lets call this Sk2.
I want to scale one skeleton size, to another skeleton size, mantaining the proportions.
PS: I know the ScaleTo() extension method os Coding4Fun library but i dont understand this method in deeply.
How can i achieve it?
EDIT 1
Tha ScaleTo() extension method from the Coding4Fun did not work for me.
So i am trying to implement my ow scale method to scale the entire skeleton, please refer to this question for the actual code