How can I convert "rs2::video frame" to "CvCapture* "? - realsense

I'm newbie to the Intel Realsense SDK and coding in Visual Studio 2017(C or C++) for Intel Realsense camera D435.
In my example I have the following,
static rs2::frameset current_frameset;
auto color = current_frameset.get_color_frame();
frame = cvQueryFrame(color);
I've got an error on line 3 as "can not convert 'rs2::video_frame' to 'CvCapture'"
I've not being able to find a solution to this issue and it's proving difficult and resulted in more errors.
Does anyone know how I can overcome this problem?
Thanks for the help!

The cvQueryFrame accepts cvCapture instance, and it is used to retrieve the frame from camera. In LibRS, the frame you retrieved back can be used already, you don't have to get back it again. attached the snippet of CV example in LibRS, you can refer to the complete code here
rs2::pipeline pipe;
// Start streaming with default recommended configuration
pipe.start();
using namespace cv;
const auto window_name = "Display Image";
namedWindow(window_name, WINDOW_AUTOSIZE);
while (waitKey(1) < 0 && cvGetWindowHandle(window_name))
{
rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera
rs2::frame depth = color_map(data.get_depth_frame());
// Query frame size (width and height)
const int w = depth.as<rs2::video_frame>().get_width();
const int h = depth.as<rs2::video_frame>().get_height();
// Create OpenCV matrix of size (w,h) from the colorized depth data
Mat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);
// Update the window with new data
imshow(window_name, image);
}

Related

ARCore: accessing camera image data from frame.acquireCameraImage

I have an app in Java which needs to get the camera image from ARCore for custom rendering. Recently, two new APIs have been added to ARCore (v1.1.0) which should support this. Well, everything else works for me except these two calls. Both calls are failing with same error (SIGINIT):
SIGINIT
and I have this message in the Logcat:
04-18 14:52:32.007 32623-32623/ A/Ion: [java/com/google/vr/dynamite/client/native/dynamite_client.cc:74] CHECK failed: expression='"env"'
04-18 14:52:32.007 32623-32623/ A/Ion: Dumping stack:
relevant code is:
try
{
// Obtain the current frame from session.
mSession.setCameraTextureName( mTextureId );
Frame frame = mSession.update();
TrackingState trackingState = frame.getCamera().getTrackingState();
if ( trackingState == TrackingState.TRACKING )
{
// Compute lighting, this is also crashing
//final float[] colorCorrectionRgba = new float[4];
//frame.getLightEstimate().getColorCorrection( colorCorrectionRgba, 0 );
//getPixelIntensity works fine
float lightIntensity = frame.getLightEstimate().getPixelIntensity();
Log.d( TAG, " light intensity is " + lightIntensity );
//This is also crashing
android.media.Image image = frame.acquireCameraImage();
//do something useful with the image
image.close();
app level build.gradle has the following entry under dependencies:
implementation 'com.google.ar:core:1.1.0'
and I have tried this with both proguard enabled and disabled.
Can anyone spot what I am doing wrong?
PS: In case it's not clear, the failing calls are frame.acquireCameraImage() and frame.getLightEstimate().getColorCorrection().

Kinect V2 IR Video Stream

I'm working on a new application which can access the video stream from the Kinect V2 sensor. I've got the application working with the standard RGB and Depth video streams... but I am running into an issue with the IR video stream. I've modified the example found here to fit my application... but the pixel values that I am returning as part of my bitmap are always black (ie. value = 0). Here's the part of the code that I have running in my MultiSourceFrameArrived event handler:
using (InfraredFrame IRFrame = framew.InfraredFrameReference.AcquireFrame())
{
if (IRFrame != null)
{
FrameDescription FrameDesc = IRFrame.FrameDescription;
ushort[] IRData = new ushort[FrameDesc.Width * FrameDesc.Height];
IRImgBuffer = new byte[4 * FrameDesc.Width * FrameDesc.Height];
IRFrame.CopyFrameDataToArray(IRData);
int colorIndex = 0;
for (int IRIndex = 0; IRIndex < IRData.Length; ++IRIndex)
{
ushort depth = IRData[IRIndex];
ushort ir = IRData[IRIndex];
byte intensity = (byte)(ir >> 8);
IRImgBuffer[colorIndex++] = (byte)ir; // Blue
IRImgBuffer[colorIndex++] = (byte)ir; // Green
IRImgBuffer[colorIndex++] = (byte)ir; // Red
++colorIndex;
}
gotframe = true;
}
}
It's even more frustrating as I can't seem to launch this in the debugger. If I put a breakpoint in the code to see what the Blue pixel value (for example) is - the debugger never seems to catch it (not entirely sure why). Can anyone help me understand why the intensity value is always 0?
It appears that the issue was happening because of an outdated graphics card. See this thread: Kinect Infrared Camera Not working. I have a relatively new laptop, but once I updated my Nvidia drivers it now seems to be working.

Integrate FBX SDK to DirectX

I am working with FBX SDK 2013.3 and project on DirectX .
I need some basic knowledge,
Think that i have a cube, exported from Maya with .fbx extension, it has animation but problem isn't that now.
Now i need Load this .fbx file to DirectX, so indices,vertex positions must be handle.
I looked FBX SDK Documentations, Samples ( mostly ViewScene sample ) and i obtain some information.
const char* lFilename = ".\..\..\..\DuzZemin.fbx";
/* Memory Management */
FbxManager* myManager = FbxManager::Create();
FbxIOSettings* ioSettings = FbxIOSettings::Create(myManager,IOSROOT);
myManager->SetIOSettings(ioSettings);
// Importer
FbxImporter* myImporter = FbxImporter::Create(myManager,"");
if( !myImporter->Initialize(lFilename,-1,ioSettings))
{
/* Error Handling */
}
/* Scene for Imported File */
FbxScene* myScene = FbxScene::Create(myManager,"My Scene");
myImporter->Import(myScene);
// Memory DeAllocation
myImporter->Destroy();
FbxNode* rootNode = myScene->GetRootNode();
FbxMesh* myMesh = FbxMesh::Create(myScene,"");
int indexCount = myMesh->GetControlPointsCount();
and when build this code snippet, i am not getting any error. But in runtime ,
indexCount return with 0 value.
Do you see any wrong or missed requirement ?
Thanks for interest.
Well you create a mesh, but you don't point it to a mesh in your scene. When i have loaded an .fbx file i create a mesh pointer and then go into my scene and grab the mesh. Consider this code.
FbxMesh* mesh;
FbxNode* node = myScene->GetRootNode()->GetChild(0);
//I want my mesh to be formed by triangles so i convert the mesh into triangles
FbxGeometryConverter lConverter(node->GetFbxManager());
lConverter.TriangulateInPlace(node);
mesh = node->GetMesh();
now you should be able to get a correct vertex count from your mesh, and be able to extract the vertices. NOTE: If your .fbx file consist of more than one mesh then you would need to iterate thru the scene untill you find the desired mesh.
Hope this helps
FbxMesh* mesh;
FbxNode* node = myScene->GetRootNode()->GetChild(0);
//I want my mesh to be formed by triangles so i convert the mesh into triangles
FbxGeometryConverter lConverter(node->GetFbxManager());
lConverter.TriangulateInPlace(node);
mesh = node->GetMesh();

How to detect an image border programmatically?

I'm searching for a program which detects the border of a image,
for example I have a square and the program detects the X/Y-Coords
Example:
alt text http://img709.imageshack.us/img709/1341/22444641.png
This is a very simple edge detector. It is suitable for binary images. It just calculates the differences between horizontal and vertical pixels like image.pos[1,1] = image.pos[1,1] - image.pos[1,2] and the same for vertical differences. Bear in mind that you also need to normalize it in the range of values 0..255.
But! if you just need a program, use Adobe Photoshop.
Code written in C#.
public void SimpleEdgeDetection()
{
BitmapData data = Util.SetImageToProcess(image);
if (image.PixelFormat != PixelFormat.Format8bppIndexed)
return;
unsafe
{
byte* ptr1 = (byte *)data.Scan0;
byte* ptr2;
int offset = data.Stride - data.Width;
int height = data.Height - 1;
int px;
for (int y = 0; y < height; y++)
{
ptr2 = (byte*)ptr1 + data.Stride;
for (int x = 0; x < data.Width; x++, ptr1++, ptr2++)
{
px = Math.Abs(ptr1[0] - ptr1[1]) + Math.Abs(ptr1[0] - ptr2[0]);
if (px > Util.MaxGrayLevel) px = Util.MaxGrayLevel;
ptr1[0] = (byte)px;
}
ptr1 += offset;
}
}
image.UnlockBits(data);
}
Method from Util Class
static public BitmapData SetImageToProcess(Bitmap image)
{
if (image != null)
return image.LockBits(
new Rectangle(0, 0, image.Width, image.Height),
ImageLockMode.ReadWrite,
image.PixelFormat);
return null;
}
If you need more explanation or algorithm just ask with more information without being so general.
It depends what you want to do with the border, if you are looking at getting just the values of the edges of the region, use an algorithm called the Connected Components Region. You must know the value of the region prior to using the algorithm. This will navigate around the border and collect the outside region. If you are trying to detect just the outside lines get the gradient of the image and it will reveal where the lines are. To do this convolve the image with an edge detection filter such as Prewitt, Sobel, etc.
You can use any image processing library such as Opencv. which is in c++ or python.
You should look for edge detection functions such as Canny edge detection.
Of course this would require some diving into image processing.
The example image you gave should be straight forward to detect, how noisy/varied are the images going to be?
A shape recognition algorithm might help you out, providing it has a solid border of some kind, and the background colour is a solid one.
From the sounds of it, you just want a blob extraction algorithm. After that, the lowest/highest values for x/y will give you the coordinates of the corners.

Glut glLoadMatrixf camera equivalent

In my glut application I'm simulating a plane with the camera. When the planes speed is low I intend to have the nose start to point towards the ground as the camera falls. My first instinct was to just change the pitch until it was pointed downwards at -90degrees. However I can't just change the pitch because if the plane is tilted on its side or upside down then it would note be changing direction towards the ground.
Now i'm trying to do a rough simulation of this by shifting the 'lookAt.y' downwards. To do this I am trying to get all the current camera coordinates that I use to set the camera
(eye.x, eye.y, eye.z, look.x, look.y, look.z, up.x, up.y, up.z). Then recall the set with the new modified values.
I've been working with the Camera.cpp and Camera.h to control my camera functions. They can be found here
after adding methods to get all the values, only the eye values are actually updated when various camera motions are made. I guess my question is how do I retrieve these values.
The glLoadMaxtrix call is in this function
void Camera :: setModelViewMatrix(void)
{ // load model view matrix with existing camera values
float m[16];
Vector3 eVec(eye.x, eye.y, eye.z);
m[0] = u.x; m[4] = u.y; m[8] = u.z; m[12] = -eVec.dot(u);
m[1] = v.x; m[5] = v.y; m[9] = v.z; m[13] = -eVec.dot(v);
m[2] = n.x; m[6] = n.y; m[10] = n.z; m[14] = -eVec.dot(n);
m[3] = 0; m[7] = 0; m[11] = 0; m[15] = 1.0;
look.x = u.y; look.y = v.y; look.z = n.y;
glMatrixMode(GL_MODELVIEW);
glLoadMatrixf(m);
}
Is there a way to get 'eye', 'lookAt', and 'up' values from the matrix here? Or should I do something else to get these values?
-Thanks in advance for your help
The camera class you link to is not an actual OpenGL class, but it should be simple enough to work with.
The function quoted just takes the current values of the camera object and sends them to OpenGL. If you look at the camera's set function, you can see how the program calculates the values it actually stores.
The eye value is stored directly. The lookAt value is just the value of (eye - n), by vector math. The up value is the hardest, but if I remember my vector math correctly, I believe that up = (n cross u).