I want to implement capture over time by sending RDP screen data to Java using freerdp library written in C.
I found the unsigned char* type data value in the C module structure and extracted the image using the stb_image library. and success.
stbi_write_bmp(locationPath, int width, int height, 4,unsign char* data); // simply succeeded.
Using JNI - JByteArray, I received the corresponding value as a byte[] from JAVA, and I want to extract this byte[] value as an image.
public static void JNIcallback2(int width, int height,byte[] data) throws IOException {
ByteArrayInputStream inputStream = new ByteArrayInputStream(data);
BufferedImage bufferedImage = ImageIO.read(inputStream);
ImageIO.write(bufferedImage, "png", new File("/Users/kimfk567/Documents/javaJNI_Image/data6.png"));
}
but error
Exception in thread "Thread-4" java.lang.IllegalArgumentException: image == null!
at java.desktop/javax.imageio.ImageTypeSpecifier.createFromRenderedImage(ImageTypeSpecifier.java:925)
at java.desktop/javax.imageio.ImageIO.getWriter(ImageIO.java:1610)
at java.desktop/javax.imageio.ImageIO.write(ImageIO.java:1542)
The Android implementation of Freerdp uses the Bitmap class to process data, but I can't use Android Studio.
https://developer.android.com/reference/android/graphics/BitmapFactory
byte[] to hex string
This value is estimated as 4-byte RGBA and is not extracted from a specific image.
AND
It seems to be pixel data of a bitmap.
To output this value as an image, I converted unsigned char* to int* in C, received it as int[] in JAVA, and also tried WritableRaster.setPixels
BufferedImage a = new BufferedImage(1024,768,2);
WritableRaster raster = (WritableRaster) a.getData();
raster.setPixels(0,0,1024,768, convertData);
a.setData(raster);
ImageIO.write(bufferedImage, "jpg", new File("/Users/Documents/javaJNI_Image/data.jpg"))
but failed..
I invested a lot of time but couldn't solve it.
Related
I'm in a C++/CLI project, and I have a byte* variable that I want to fully convert into a managed array<byte> as efficiently as possible.
Currently, the only way that I've seen is to manually create the managed array<byte> object, and then copy individual bytes from the byte* variable, as shown below:
void Foo(byte* source, int bytesCount)
{
auto buffer = gcnew array<byte>(bytesCount);
for (int i = 0; i < bytesCount; ++i)
{
buffer[i] = source[i];
}
}
Is there any other way to do this more efficiently? Ideally, to not have to copy the memory at all.
If not, is there any way to do this more cleanly?
You can't create a managed array from an unmanaged buffer without copying.
However, you don't need to copy the individual bytes in a loop, though. You can pin the managed array (see pin_ptr) and then copy the bytes directly from the unmanaged buffer into the memory of the managed array, such as with memcpy() or equivalent.
void Foo(byte* source, int bytesCount)
{
auto buffer = gcnew array<byte>(bytesCount);
{
pin_ptr<byte> p = &buffer[0];
byte *cp = p;
memcpy(cp, source, bytesCount);
}
// use buffer as needed...
}
I'm implementing a camera application on Android devices. Currently, I use Camera2 API and ImageReader to get image data in YUV_420_888 format, but I don't know how to exactly write these data to MediaCodec.
Here are my questions:
What is YUV_420_888?
The format YUV_420_888 is ambiguous because it can be any format which belongs to the YUV420 family, such as YUV420P, YUV420PP, YUV420SP and YUV420PSP, right?
By accessing the image's three planes(#0, #1, #2), I can get the Y(#0), U(#1), V(#2) values of this image. But the arrangement of these values may not be the same on different devices. For example, if YUV_420_888 truly means YUV420P, the size of both Plane#1 and Plane#2 is a quarter of the size of Plane#0. If YUV_420_888 truly means YUV420SP, the size of both Plane#1 and Plane#2 is half of the size of Plane#0(Each of Plane#1 and Plane#2 contains U, V values).
If I want to write these data from image's three planes to MediaCodec, what kind of format I need to convert to? YUV420, NV21, NV12, ...?
What is COLOR_FormatYUV420Flexible?
The format COLOR_FormatYUV420Flexible is also ambiguous because it can be any format which belongs to the YUV420 family, right? If I set KEY_COLOR_FORMAT option of a MediaCodec object to COLOR_FormatYUV420Flexible, what format(YUV420P, YUV420SP...?) of data should I input to the MediaCodec object?
How about using COLOR_FormatSurface?
I know MediaCodec has its own surface, which can be used if I set KEY_COLOR_FORMAT option of a MediaCodec object to COLOR_FormatSurface. And with Camera2 API, I don't need to write any data by myself to the MediaCodec object. I can just drain the output buffer.
However, I need to change the image from the camera. For example, draw other pictures, write some text on it, or insert another video as POP(Picture of Picture).
Can I use ImageReader to read the image from Camera, and after re-drawing that, write the new data to MediaCodec's surface, and then drain it out? How to do that?
EDIT1
I implemented the function by using COLOR_FormatSurface and RenderScript. Here is my code:
onImageAvailable method:
public void onImageAvailable(ImageReader imageReader) {
try {
try (Image image = imageReader.acquireLatestImage()) {
if (image == null) {
return;
}
Image.Plane[] planes = image.getPlanes();
if (planes.length >= 3) {
ByteBuffer bufferY = planes[0].getBuffer();
ByteBuffer bufferU = planes[1].getBuffer();
ByteBuffer bufferV = planes[2].getBuffer();
int lengthY = bufferY.remaining();
int lengthU = bufferU.remaining();
int lengthV = bufferV.remaining();
byte[] dataYUV = new byte[lengthY + lengthU + lengthV];
bufferY.get(dataYUV, 0, lengthY);
bufferU.get(dataYUV, lengthY, lengthU);
bufferV.get(dataYUV, lengthY + lengthU, lengthV);
imageYUV = dataYUV;
}
}
} catch (final Exception ex) {
}
}
Convert YUV_420_888 to RGB:
public static Bitmap YUV_420_888_toRGBIntrinsics(Context context, int width, int height, byte[] yuv) {
RenderScript rs = RenderScript.create(context);
ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));
Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs)).setX(yuv.length);
Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);
Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(width).setY(height);
Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);
Bitmap bmpOut = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
in.copyFromUnchecked(yuv);
yuvToRgbIntrinsic.setInput(in);
yuvToRgbIntrinsic.forEach(out);
out.copyTo(bmpOut);
return bmpOut;
}
MediaCodec:
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
...
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
...
surface = mediaCodec.createInputSurface(); // This surface is not used in Camera APIv2. Camera APIv2 uses ImageReader's surface.
And in athother thread:
while (!stop) {
final byte[] image = imageYUV;
// Do some yuv computation
Bitmap bitmap = YUV_420_888_toRGBIntrinsics(getApplicationContext(), width, height, image);
Canvas canvas = surface.lockHardwareCanvas();
canvas.drawBitmap(bitmap, matrix, paint);
surface.unlockCanvasAndPost(canvas);
}
This way works, but the performance is not good. It can't output 30fps video files(only ~12fps). Perhaps I should not use COLOR_FormatSurface and the surface's canvas for encoding. The computed YUV data should be written to the mediaCodec directly without any surface doing any conversion. But I still don't know how to do that.
You are right, YUV_420_888 is a format that can wrap different YUV 420 formats. The spec carefully explains that the arrangement of U and V planes is not prescribed, but there are certain restrictions; e.g. if the U plane has pixel stride 2, same applies to V (and then the underlying byte buffer can be NV21).
COLOR_FormatYUV420Flexible is a synonym of YUV_420_888, but they belong to different classes: MediaCodec and ImageFormat, respectively.
The spec explains:
All video codecs support flexible YUV 4:2:0 buffers since LOLLIPOP_MR1.
COLOR_FormatSurface is an opaque format that can deliver best performance for MediaCodec, but this comes at price: you cannot directly read or manipulate its content. If you need to manipulate the data that goes to the MediaCodec, then using ImageReader is an option; whether it will be more efficient than ByteBuffer, depends on what you do and how you do it. Note that for API 24+ you can work with both camera2 and MediaCodec in C++.
The invaluable resource of details for MediaCodec is http://www.bigflake.com/mediacodec. It references a full example of 264 encoding.
Create a textureID -> SurfaceTexture -> Surface -> Camera 2 -> onFrameAvaliable -> updateTexImage -> glBindTexture -> draw something -> swapbuffer to Mediacodec's inputSurface.
I haven't seen C++ code in more than 10 years and now I'm in the need of developing a very small DLL to use the Ping class (System::Net::NetworkInformation) to make a ping to some remoteAddress.
The argument where I'm receiving the remoteAddress is a FREObject which then needs to be transformed into a const uint8_t *. The previous is mandatory and I can't change anything from it. The remoteAddress has to be received as a FREObject and later be transformed in a const uint8_t *.
The problem I'm having is that I have to pass a String^ to the Ping class and not a const uint8_t * and I have no clue of how to convert my const uint8_t * to a String^. Do you have any ideas?
Next is part of my code:
// argv[ARG_IP_ADDRESS_ARGUMENT holds the remoteAddress value.
uint32_t nativeCharArrayLength = 0;
const uint8_t * nativeCharArray = NULL;
FREResult status = FREGetObjectAsUTF8(argv[ARG_IP_ADDRESS_ARGUMENT], &nativeCharArrayLength, &nativeCharArray);
Basically the FREGetObjectAsUTF8 function fills the nativeCharArray array with the value of argv[ARG_IP_ADDRESS_ARGUMENT] and returns the array's length in nativeCharArrayLength. Also, the string uses UTF-8 encoding terminates with the null character.
My next problem would be to convert a String^ back to a const uint8_t *. If you can help with this as well I would really appreciate it.
As I said before, non of this is changeable and I have no idea of how to change nativeCharArray to a String^. Any advice will help.
PS: Also, the purpose of this DLL is to use it as an ANE (Air Native Extension) for my Adobe Air app.
You'll need UTF8Encoding to convert the bytes to characters. It has methods that take pointers, you'll want to take advantage of that. You first need to count the number of characters in the converted string, then allocate an array to store the converted characters, then you can turn it into System::String. Like this:
auto converter = gcnew System::Text::UTF8Encoding;
auto chars = converter->GetCharCount((Byte*)nativeCharArray, nativeCharArrayLength-1);
auto buffer = gcnew array<Char>(chars);
pin_ptr<Char> pbuffer = &buffer[0];
converter->GetChars((Byte*)nativeCharArray, nativeCharArrayLength-1, pbuffer, chars);
String^ result = gcnew String(buffer);
Note that the -1 on nativeCharArrayLength compensates for the zero terminator being included in the value.
I've been working on DirectX lately and I have a Array<byte>^ function to read the shaders in the program. The function works in a program I made like 3 days ago, but then I ported the whole project to work with XAML and everything works basically the same except that this function now shows errors. The function is:
Array<byte>^ LoadShader(std::string File){
Array<byte>^ FileData = nullptr;
std::ifstream VertexFile(File, std::ios::in | std::ios::binary | std::ios::ate);
if(VertexFile.is_open()){
int Length = (int)VertexFile.tellg();
FileData = ref new Array<byte>(Length);
VertexFile.seekg(0, std::ios::beg);
VertexFile.read(reinterpret_cast<char*>(FileData->Data), Length);
VertexFile.close();
};
return FileData;
};
It's placed in a header file and the 3 errors presented are:
error C2143: syntax error : missing ';' before '<
error C2334: unexpected token(s) preceding '{'; skipping apparent function body
error C4430: missing type specifier - int assumed. Note: C++ does not support default-in
And I just don't know what to do... I've checked correct spelling of the header file, the function is of Array<byte>^ type and I'm certain I didn't jump the only body function in the header file.
If I remove the function, the header file works and I'm just baffled and have no idea how to fix this. For reference I'll post the complete header file underneath (it's not that big).
#pragma once
#include "DirectXHelper.h"
#include <fstream>
ref class DirectXBase abstract{
internal:
DirectXBase();
public:
virtual void Initialize(Windows::UI::Core::CoreWindow^ m_window, Windows::UI::Xaml::Controls::SwapChainBackgroundPanel^ m_panel);
virtual void CreateDeviceResources();
void CreateDepthStencil();
void CreatePipeline();
virtual void Render();
protected private:
Array<byte>^ LoadShader(std::string File){
Array<byte>^ FileData = nullptr;
std::ifstream VertexFile(File, std::ios::in | std::ios::binary | std::ios::ate);
if(VertexFile.is_open()){
int Length = (int)VertexFile.tellg();
FileData = ref new Array<byte>(Length);
VertexFile.seekg(0, std::ios::beg);
VertexFile.read(reinterpret_cast<char*>(FileData->Data), Length);
VertexFile.close();
};
return FileData;
};
protected private:
Platform::Agile<Windows::UI::Core::CoreWindow> window;
Windows::UI::Xaml::Controls::SwapChainBackgroundPanel^ panel;
Microsoft::WRL::ComPtr<ID3D11Device1> DXDevice;
Microsoft::WRL::ComPtr<ID3D11DeviceContext1> DXContext;
Microsoft::WRL::ComPtr<IDXGISwapChain1> SwapChain;
Microsoft::WRL::ComPtr<ID3D11RenderTargetView> RTView; //Render Target View
Microsoft::WRL::ComPtr<ID3D11DepthStencilView> DepthView; //3D Depth Stencil View
Microsoft::WRL::ComPtr<ID3D11Texture2D> DepthBuffer;
Microsoft::WRL::ComPtr<ID3D11InputLayout> InLayout;
Microsoft::WRL::ComPtr<ID3D11VertexShader> VShader; //Vertex Shader
Microsoft::WRL::ComPtr<ID3D11PixelShader> PShader; //Pixel Shader
};
Umm. Is it possible that your previous use had a using namespace Platform; somewhere in the file above the use of Array<byte>?
If so that would explain why it's not working.
The documentation for ResizeSettings says:
"Replaced by the Instructions class"
http://documentation.imageresizing.net/docu/ImageResizer/ResizeSettings.htm
The documentation for Instructions says:
"The successor to ResizeSettings."
http://documentation.imageresizing.net/docu/ImageResizer/Instructions.htm
However, I cannot figure out how to use Instructions instead of ResizeSettings. I've tried
Google
Documentation (documentation.imageresizing.net)
Looking through the Object Browser for uses of Instructions
Searching ImageResizer.dll in .net Reflector for uses of Instructions
Decompiling all of ImageResizer.dll and searching for through the resulting code.
If Instructions replaces ResizeSettings, then how do I use it instead of ResizeSettings?
=== Edit - more detail:
This a way to use ResizeSettings:
public static Bitmap Resize(Bitmap bitmap, int maxHeight, int maxWidth)
{
var setting = new ResizeSettings
{
MaxHeight = maxHeight,
MaxWidth = maxWidth,
};
return ImageBuilder.Current.Build(bitmap, setting);
}
Reading that Instructions was a replacement for ResizeSettings, one of the first things I tried was this: (I was hoping ImageBuilder might have an overloaded Build method)
public static Bitmap Resize(Bitmap bitmap, int maxHeight, int maxWidth)
{
var instructions = new Instructions
{
Width = maxWidth,
Height = maxHeight,
Mode = FitMode.Max
};
return ImageBuilder.Current.Build(bitmap, instructions);
}
In an unexpected turn of events, the documentation is ahead of reality.
You can use the Instructions class, but for now you must convert it to a ResizeSettings instance first like so:
.Build(source, dest, new ResizeSettings(new Instructions("width=20")));
In the next major release, this will accept an Instructions class directly.