Hi I am working on a live streaming solution where I need to edit the local video frame and send it to the connected peer, for this task I am editing
https://webrtc.googlesource.com/src/+/master/sdk/android/src/java/org/webrtc/Camera2Session.java#201, the VideoFrame from here is processed to the editing engine, later on it's return a YUV420 ByteBuffer with following values:
frameWidth=1280
frameHeight=720
RowStrideY:1280
RowStrideU:640
RowStrideV:640
PlaneHeightY:720
PlaneHeightU:360
PlaneHeightV:360
PlaneOffsetY:0
PlaneOffsetU:921600
PlaneOffsetV:1152000
PixelStrideY=1
PixelStrideU=1
PixelStrideV=1
Now I want to recreate the VideoFrame object from this details, I tried various approaches to achieve this and almost very close to solution only having trouble to seprate the Y,U and V ByteBuffer from the ByteBuffer, due to which the output is little bit destorted.
Following code is been used by me for the conversions:
public static I420Buffer convert(ImageProcessResult result) {
int frameWidth = result.getWidth();
int frameHeight = result.getHeight();
int rowStrideY = result.getRowStride(0);
int rowStrideU = result.getRowStride(1);
int rowStrideV = result.getRowStride(2);
int offsetY = result.getPlaneOffset(0);
int offsetU = result.getPlaneOffset(1);
int offsetV = result.getPlaneOffset(2);
ByteBuffer i420ByteBuffer = result.getBuffer();
i420ByteBuffer.position(offsetY);
final ByteBuffer dataY = i420ByteBuffer.slice();
i420ByteBuffer.position(offsetU);
final ByteBuffer dataU = i420ByteBuffer.slice();
i420ByteBuffer.position(offsetV);
final ByteBuffer dataV = i420ByteBuffer.slice();
JavaI420Buffer frame = JavaI420Buffer.wrap(frameWidth, frameHeight, dataY, rowStrideY, dataU, rowStrideU, dataV, rowStrideV,
() -> {
JniCommon.nativeFreeByteBuffer(i420ByteBuffer);
});
return frame;
}
Any help to fix this issue will be appreciated, thanks in advance..!
Related
I am getting the error Type 'CUnsignedChar?' has no subscript members which produces a lot of results in stackoverflow however I can't seem to utilise any of the other available answers for my example. Its clearly a casting issue but I don't not see how to overcome it
I am doing a obj-c to swift conversion and I have a variable being set up as follows
var bBuff1 = [CUnsignedChar](repeating: 0, count: Int(256*512))
var backGreyBuffer : CUnsignedChar = bBuff1[0]
//..
//..
var backGreyBufferOffset : Int = localTexOffset * 512
var grey_val = 0
self.backGreyBuffer[Int(backGreyBufferOffset)]! = grey_val; //Subscript error here
This is the obj-c code that uses in-outs.
unsigned char bBuff1[256*512];
unsigned char *backGreyBuffer = &bBuff1[0];
//..
grey_val = 0;
backGreyBuffer[backGreyBufferOffset] = grey_val;
Any suggestions about the right direction would be great.
I noticed that only a small change is needed in your code. You should make backGreyBuffer a pointer:
var bBuff1 = [CUnsignedChar](repeating: 0, count: Int(256*512))
var backGreyBuffer = UnsafeMutablePointer(mutating: bBuff1)
// ....
var backGreyBufferOffset = localTexOffset * 512
backGreyBuffer[backGreyBufferOffset] = grey_val
Hi I am trying to bind depth memory buffer but I get an error saying as below. I have no idea why this error is popping up.
The depth format is VK_FORMAT_D16_UNORM and the usage is VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT. I have read online that the TILING shouldnt be linear but then I get a different error. Thanks!!!
The code for creating and binding the image is as below.
VkImageCreateInfo imageInfo = {};
// If the depth format is undefined, use fallback as 16-byte value
if (Depth.format == VK_FORMAT_UNDEFINED) {
Depth.format = VK_FORMAT_D16_UNORM;
}
const VkFormat depthFormat = Depth.format;
VkFormatProperties props;
vkGetPhysicalDeviceFormatProperties(*deviceObj->gpu, depthFormat, &props);
if (props.linearTilingFeatures & VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT) {
imageInfo.tiling = VK_IMAGE_TILING_LINEAR;
}
else if (props.optimalTilingFeatures & VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT) {
imageInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
}
else {
std::cout << "Unsupported Depth Format, try other Depth formats.\n";
exit(-1);
}
imageInfo.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO;
imageInfo.pNext = NULL;
imageInfo.imageType = VK_IMAGE_TYPE_2D;
imageInfo.format = depthFormat;
imageInfo.extent.width = width;
imageInfo.extent.height = height;
imageInfo.extent.depth = 1;
imageInfo.mipLevels = 1;
imageInfo.arrayLayers = 1;
imageInfo.samples = NUM_SAMPLES;
imageInfo.queueFamilyIndexCount = 0;
imageInfo.pQueueFamilyIndices = NULL;
imageInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
imageInfo.usage = VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT;
imageInfo.flags = 0;
// User create image info and create the image objects
result = vkCreateImage(deviceObj->device, &imageInfo, NULL, &Depth.image);
assert(result == VK_SUCCESS);
// Get the image memory requirements
VkMemoryRequirements memRqrmnt;
vkGetImageMemoryRequirements(deviceObj->device, Depth.image, &memRqrmnt);
VkMemoryAllocateInfo memAlloc = {};
memAlloc.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO;
memAlloc.pNext = NULL;
memAlloc.allocationSize = 0;
memAlloc.memoryTypeIndex = 0;
memAlloc.allocationSize = memRqrmnt.size;
// Determine the type of memory required with the help of memory properties
pass = deviceObj->memoryTypeFromProperties(memRqrmnt.memoryTypeBits, 0, /* No requirements */ &memAlloc.memoryTypeIndex);
assert(pass);
// Allocate the memory for image objects
result = vkAllocateMemory(deviceObj->device, &memAlloc, NULL, &Depth.mem);
assert(result == VK_SUCCESS);
// Bind the allocated memeory
result = vkBindImageMemory(deviceObj->device, Depth.image, Depth.mem, 0);
assert(result == VK_SUCCESS);
Yes, linear tiling may not be supported for depth usage Images.
Consult the specification and Valid Usage section of VkImageCreateInfo. The capability is queried by vkGetPhysicalDeviceFormatProperties and vkGetPhysicalDeviceImageFormatProperties commands. Though depth formats are "opaque", so there is not much reason to use linear tiling.
This you seem to be doing in your code.
But the error informs you that you are trying to use a memory type that is not allowed for the given Image. Use vkGetImageMemoryRequirements command to query which memory types are allowed.
Possibly you have some error there (you are using 0x1 which is obviously not part of 0x84 per the message). You may want to reuse the example code in the Device Memory chapter of the specification. Provide your memoryTypeFromProperties implementation for more specific answer.
I accidentally set the typeIndex to 1 instead of i and it works now. In my defense I have been vulkan coding the whole day and my eyes are bleeding :). Thanks for the help.
bool VulkanDevice::memoryTypeFromProperties(uint32_t typeBits, VkFlags
requirementsMask, uint32_t *typeIndex)
{
// Search memtypes to find first index with those properties
for (uint32_t i = 0; i < 32; i++) {
if ((typeBits & 1) == 1) {
// Type is available, does it match user properties?
if ((memoryProperties.memoryTypes[i].propertyFlags & requirementsMask) == requirementsMask) {
*typeIndex = i;// was set to 1 :(
return true;
}
}
typeBits >>= 1;
}
// No memory types matched, return failure
return false;
}
Sorry for asking a really basic question, but it's probably the first time for a number of years I feel really confused.
Windows provides two set of controls: Windows.UI.Xaml namespace (I thinks this is referred as Metro), used for Windows Store Apps, and System.Windows (WPF) for Desktop.
Since I am going to develop Windows Store Apps mainly for Windows 8.1 and Windows 10 phones, I will have to stick to Windows.UI.Xaml, and this has not only a separate set of UI elements, but also separate set of bitmaps, brushes, etc. (Windows.UI.Xaml.Media vs System.Windows.Media).
I found that Windows.UI.Xaml provides a very limited support for graphics, much less than provided by WPF, Android or (even!) iOS platform. To start with, I got stuck with a simple task: tiling a background!
Since Windows.UI.Xaml.Media.ImageBrush do not support tiling, I wanted to to do that "manually". Some sites suggest making numerous number of children, each holding a tile. Honestly, it looks as a rather awkward approach to me, so I decided to do it in what appears a more natural way: create an off-screen tiled image, assign it to a brush and assign the brush as the background for a panel.
The tiling code is rather straightforward (it probably has mistakes, possibly won't even not run on a phone, because of some unavailable classes used).
int panelWidth = (int) contentPanel.Width;
int panelHeight = (int) contentPanel.Height;
Bitmap bmpOffscreen = new Bitmap(panelWidth, panelHeight);
Graphics gOffscreen = Graphics.FromImage(bmpOffscreen);
string bmpPath = Path.Combine(Windows.ApplicationModel.Package.Current.InstalledLocation.Path, "Assets/just_a_tile.png");
System.Drawing.Image tile = System.Drawing.Image.FromFile(bmpPath, true);
int tileWidth = tile.Width;
int tileHeight = tile.Height;
for (int y = 0; y < panelHeight; y += tileHeight)
for (int x = 0; x < panelWidth; x += tileWidth)
gOffscreen.DrawImage(tile, x, y);
Now I presumably have the tiled image in bmpOffscreen. But how assign it to a brush? To do that I need to convert Bitmap to BitmapSource, while I couldn't find something similar to System.Windows.Imaging.CreateBitmapSourceFromHBitmap available for WPF structure!
Well, first of all System.Drawing namespace is not available in Windows Universal Platform, so you won't be able to use Bitmap class
But, all hope is not lost - you can use
Windows.UI.Xaml.Media.Imaging.WriteableBitmap
If you look at example included on this page, you will see that at one point image data is extracted to a byte array - all you need to do is copy it according to your needs
Please let me know if you want me to include a complete code sample.
Edit:
StorageFile file = await StorageFile.GetFileFromPathAsync(filePath);
Scenario4WriteableBitmap = new WriteableBitmap(2000, 2000);
// Ensure a file was selected
if (file != null)
{
using (IRandomAccessStream fileStream = await file.OpenAsync(Windows.Storage.FileAccessMode.Read))
{
int columns = 4;
int rows = 4;
BitmapDecoder decoder = await BitmapDecoder.CreateAsync(fileStream);
// Scale image to appropriate size
BitmapTransform transform = new BitmapTransform()
{
ScaledHeight = Convert.ToUInt32(Scenario4ImageContainer.Height),
ScaledWidth = Convert.ToUInt32(Scenario4ImageContainer.Width)
};
PixelDataProvider pixelData = await decoder.GetPixelDataAsync(
BitmapPixelFormat.Bgra8, // WriteableBitmap uses BGRA format
BitmapAlphaMode.Straight,
transform,
ExifOrientationMode.IgnoreExifOrientation, // This sample ignores Exif orientation
ColorManagementMode.DoNotColorManage);
// An array containing the decoded image data, which could be modified before being displayed
byte[] sourcePixels = pixelData.DetachPixelData();
// Open a stream to copy the image contents to the WriteableBitmap's pixel buffer
using (Stream stream = Scenario4WriteableBitmap.PixelBuffer.AsStream())
{
for (int i = 0; i < columns * rows; i++)
{
await stream.WriteAsync(sourcePixels, 0, sourcePixels.Length);
}
}
}
// Redraw the WriteableBitmap
Scenario4WriteableBitmap.Invalidate();
Scenario4Image.Source = Scenario4WriteableBitmap;
Scenario4Image.Stretch = Stretch.None;
}
Thank you, Arkadiusz. Since Australian time goes slightly ahead of Europe,
I had an advantage and seen the code before you posted it. I downloaded
MSDN XAML images sample and it helped me a lot. I gave a +1 to you but someone apparently put -1, so it compensated each other. Don't be upset I get -1 so often, that I stopped paying attention on that :)
So I've managed to do tiling with Windows Universal Platform! On my Lumia 532 phone it works magnifique. I felt like re-inventing a wheel, because all this stuff must be handled by SDK, not by a third-party developer.
public static async Task<bool> setupTiledBackground(Panel panel, string tilePath)
{
Brush backgroundBrush = await createTiledBackground((int)panel.Width, (int)panel.Height, TilePath);
if (backgroundBrush == null) return false;
panel.Background = backgroundBrush;
return true;
}
private static async Task<Brush> createTiledBackground(int width, int height, string tilePath)
{
StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///" + tilePath));
byte[] sourcePixels;
int tileWidth, tileHeight;
using (IRandomAccessStream inputStream = await file.OpenAsync(Windows.Storage.FileAccessMode.Read))
{
if (inputStream == null) return null;
BitmapDecoder tileDecoder = await BitmapDecoder.CreateAsync(inputStream);
if (tileDecoder == null) return null;
tileWidth = (int)tileDecoder.PixelWidth;
tileHeight = (int) tileDecoder.PixelHeight;
PixelDataProvider pixelData = await tileDecoder.GetPixelDataAsync(
BitmapPixelFormat.Bgra8, // WriteableBitmap uses BGRA format
BitmapAlphaMode.Straight,
new BitmapTransform(),
ExifOrientationMode.IgnoreExifOrientation,
ColorManagementMode.DoNotColorManage);
sourcePixels = pixelData.DetachPixelData();
// fileStream.Dispose();
}
WriteableBitmap backgroundBitmap = new WriteableBitmap(width, height);
int tileBmpWidth = tileWidth << 2;
int screenBmpWidth = width << 2;
int tileSize = tileBmpWidth * tileHeight;
int sourceOffset = 0;
using (Stream outputStream = backgroundBitmap.PixelBuffer.AsStream())
{
for (int bmpY=0; bmpY < height; bmpY++) {
for (int bmpX = 0; bmpX < screenBmpWidth; bmpX += tileBmpWidth)
await outputStream.WriteAsync(sourcePixels, sourceOffset, Math.Min(screenBmpWidth - bmpX, tileBmpWidth));
if ((sourceOffset += tileBmpWidth) >= tileSize)
sourceOffset -= tileSize;
}
}
ImageBrush backgroundBrush = new ImageBrush();
backgroundBrush.ImageSource = backgroundBitmap; // It's very easy now!
return backgroundBrush; // Finita la comédia!
}
Just one remark: if you do it on form start, you should not wait for it.
This doesn't work:
public MainPage()
{
this.InitializeComponent();
bool result = setupTiledBackground(contextPanel, TilePath).Result;
}
This works:
private Task<bool> backgroundImageTask;
public MainPage()
{
this.InitializeComponent();
backgroundImageTask = setupTiledBackground(contextPanel, TilePath);
}
Can anyone share a sample code on how to implement CBCBlockCipherMac in objective C. here is how far I got and its giving a different result from the java implementation.
const unsigned char key[16] = "\x1\x2\x3\x4\x5\x6\x7\x8\x9\x0\x1\x2\x3\x4\x5\x6";
const unsigned char data[14] = "\x54\x68\x69\x73\x69\x73\x6d\x79\x73\x74\x72\x69\x6e\x67";
CMAC_CTX *ctx = CMAC_CTX_new();
ret = CMAC_Init(ctx, key, sizeof(key), EVP_des_ede3(), 0);
printf("CMAC_Init = %d\n", ret);
ret = CMAC_Update(ctx, data, sizeof(data));
printf("CMAC_Update = %d\n", ret);
size_t size;
//unsigned int size;
unsigned char tag[4];
ret = CMAC_Final(ctx, tag, &size);
printf("CMAC_Final = %d, size = %u\n", ret, size);
CMAC_CTX_free(ctx);
printf("expected: 391d1520\n"
"got: ");
size_t index;
for (index = 0; index < sizeof(tag) - 1; ++index) {
printf("%02x", tag[index]);
if ((index + 1) % 4 == 0) {
printf(" ");
}
}
printf("%02x\n", tag[sizeof(tag) - 1]);
And my java code looks like this
String *data = "Thisismystring";
String *keyString = "1234567890123456";
bytes[]mac = new byte[4];
CBCBlockCipherMac macCipher = new CBCBlockCipherMac(DESedeEngine);
DESedeParameters keyParameter = new DESedeParameters(keyString.getBytes());
DESedeEngine engine = new DESedeEngine();
engine,init(true, keyParameter);
byte[] dataBytes = data.getBytes();
macCipher.update(dataBytes,0,data.length());
macCipher.doFinal(mac,0);
byte[] macBytesEncoded = Hex.encode(mac);
String macString = new String(macBytesEncoded);
This gives me "391d1520". But the objective c gives me "01000000"
CMAC is not the same as CBC MAC. CMAC has an an additional step at the beginning and the end of the calculation. If possible I would suggest you upgrade your Java code to use CMAC, as CBC is not as secure, e.g. using org.bouncycastle.crypto.macs.CMac.
OpenSSL does not seem to implement CBC MAC directly (at least, I cannot find any reference to it). So if you need it, you need to implement it yourself.
You can use CBC mode encryption with a zero IV and take the last 16 bytes of the encryption. Of course, this means you need to store the rest of the ciphertext in a buffer somewhere, or you need to use the update functions smartly (reusing the same buffer over and over again for the ciphertext).
I followed the code provided by Robert Levy at this link: http://channel9.msdn.com/coding4fun/kinect/Display-Kinect-color-image-containing-only-players-aka-background-removal
I tried implementing it into my existing code, and have had inconsistent results. If the user is in the kinect's field of view when the program starts up it will remove the background some of the time. If the user walks into the field of view it will not pick them up.
namespace KinectUserRecognition
{
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
}
//Kinect Runtime
Runtime kinect = Runtime.Kinects[0];
PlanarImage colorImage;
PlanarImage depthImage;
bool isDepthImage;
WriteableBitmap player1;
private void Window_Loaded(object sender, RoutedEventArgs e)
{
isDepthImage = false;
//UseDepthAndPlayerIndex and UseSkeletalTracking
kinect.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseColor);// | RuntimeOptions.UseSkeletalTracking);
//register for event
kinect.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
kinect.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
//Video image type
kinect.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480,
ImageType.Color);
//DepthAndPlayerIndex ImageType
kinect.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240,
ImageType.DepthAndPlayerIndex);
}
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
colorImage = e.ImageFrame.Image;
image1.Source = BitmapSource.Create(colorImage.Width, colorImage.Height, 96, 96,
PixelFormats.Bgr32, null, colorImage.Bits, colorImage.Width * colorImage.BytesPerPixel);
if (isDepthImage)
{
player1 = GeneratePlayerImage(e.ImageFrame, 1);
image3.Source = player1;
}
}
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
//Convert depth information for a pixel into color information
byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
depthImage = e.ImageFrame.Image;
image2.Source = BitmapSource.Create(depthImage.Width, depthImage.Height, 96, 96, PixelFormats.Bgr32, null,
ColoredBytes, depthImage.Width * PixelFormats.Bgr32.BitsPerPixel / 8);
isDepthImage = true;
}
private WriteableBitmap GeneratePlayerImage(ImageFrame imageFrame, int playerIndex)
{
int depthWidth = kinect.DepthStream.Width;
int depthHeight = kinect.DepthStream.Height;
WriteableBitmap target = new WriteableBitmap(depthWidth, depthHeight, 96, 96, PixelFormats.Bgra32, null);
var depthRect = new System.Windows.Int32Rect(0, 0, depthWidth, depthHeight);
byte[] color = imageFrame.Image.Bits;
byte[] output = new byte[depthWidth * depthHeight * 4];
//loop over each pixel in the depth image
int outputIndex = 0;
for (int depthY = 0, depthIndex = 0; depthY < depthHeight; depthY++)
{
for(int depthX = 0; depthX < depthWidth; depthX++, depthIndex +=2)
{
short depthValue = (short)(depthImage.Bits[depthIndex] | (depthImage.Bits[depthIndex + 1] << 8));
int colorX, colorY;
kinect.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(
imageFrame.Resolution,
imageFrame.ViewArea,
depthX, depthY, //depth coordinate
depthValue, //depth value
out colorX, out colorY); //color coordinate
//ensure that the calculate color location is within the bounds of the image
colorX = Math.Max(0, Math.Min(colorX, imageFrame.Image.Width - 1));
colorY = Math.Max(0, Math.Min(colorY, imageFrame.Image.Height - 1));
output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 0];
output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 1];
output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 2];
output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) == playerIndex ? (byte)255 : (byte)0;
}
}
target.WritePixels(depthRect, output, depthWidth * PixelFormats.Bgra32.BitsPerPixel / 8, 0);
return target;
//return output;
}
private static int GetPlayerIndex(byte firstFrame)
{
//returns 0 = no player, 1 = 1st player, 2 = 2nd player...
//bitwise & on firstFrame
return (int)firstFrame & 7;
}
}
}
-Edit 1-
I think I've narrowed the problem down, but I'm not sure of a way to resolve it. I assumed that having only one person in the kinect's field of view would return a value of one from my "GetPlayerIndex" method. This is not the case. I was hoping to produce a separate image for each person with the background removed. What type of values should I assume to receive from:
-Edit 2-
From my tests I've noticed that I can a max value of 6 for the player index, but the index that I get isn't consistent. If there a way to know what player index will be assigned to a skeleton? For example, if I were the only person in the fov would there be a way to know that my player index would always be 1?
The player index is not guaranteed to be anything. Once it catches a skeleton, the index will stay the same for that skeleton until it loses sight of it, but you can't assume that the first player will be 1, the second 2, etc.
What you'll need to do is determine a valid skeleton index prior to the player1 = GeneratePlayerImage(e.ImageFrame, 1); call, or alter the GeneratePlayerImage function to find an index. If you're only interested in removing the background and leaving the pixels for all the people in the frame untouched, just change this:
output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) == playerIndex ? (byte)255 : (byte)0;
to this, which will just check for ANY player, instead of a specific player:
output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) != 0 ? (byte)255 : (byte)0;
The other two ways I can think of to do this for a specific player instead of all players:
Open the Kinect's Skeleton feed, and loop through the array of skeletons it gives you to find a valid index. Create a global integer to hold this index, then call the GeneratePlayerImage method with this global integer.
Change the GeneratePlayerImage method to check for a player index in each pixel, and if one is found use that index to remove the background for the entire image (ignore any other index it finds).