I am trying to scale an image using vImage_Buffer and the below code works for me. My trouble is I want to maintain the aspect ratio of the source image, so I might need to add a xOffset or yOffset. Below code only works for yOffset. How can I scale the image with xOffset as well. I can not do scaling with CGContext since that affect the performance.
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
size_t finalWidth = 1080;
size_t finalHeight = 720;
size_t sourceWidth = CVPixelBufferGetWidth(imageBuffer);
size_t sourceHeight = CVPixelBufferGetHeight(imageBuffer);
CGRect aspectRect = AVMakeRectWithAspectRatioInsideRect(CGSizeMake(sourceWidth, sourceHeight), CGRectMake(0, 0, finalWidth, finalHeight));
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t startY = aspectRect.origin.y;
size_t yOffSet = (finalWidth*startY*4);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
void* destData = malloc(finalHeight * finalWidth * 4);
vImage_Buffer srcBuffer = { (void *)baseAddress, sourceHeight, sourceWidth, bytesPerRow};
vImage_Buffer destBuffer = { (void *)destData+yOffSet, aspectRect.size.height, aspectRect.size.width, aspectRect.size.width * 4};
vImage_Error err = vImageScale_ARGB8888(&srcBuffer, &destBuffer, NULL, 0);
No pun intended, but you should really read Accelerate.framework documentation.
Replace malloc with calloc ...
void *destData = calloc(finalHeight * finalWidth * 4);
... to zero all the bytes (or use any other way).
What does vImage_Buffer.rowBytes documentation say?
The distance, in bytes, between the start of one pixel row and the next in an image, including any unused space between them.
The rowBytes value must be at least the width multiplied by the pixel size, where the pixel size depends on the image format. You can provide a larger value, in which case the extra bytes will extend beyond the end of each row of pixels. You may want to do so either to improve performance, or to describe an image within a larger image without copying the data. The extra bytes aren't considered part of the image represented by the vImage buffer.
When allocating floating-point data for images, keep the data 4-byte aligned by allocating bytes as integer multiples of 4. For best performance, allocate bytes as integer multiples of 16.
Look at the following image:
Red circle (top/left corner) is offset from the buffer start, let's calculate it (assuming 4 bytes per pixel):
size_t startY = aspectRect.origin.y;
size_t startX = aspectRect.origin.x;
size_t offset = 4 * (finalWidth * startY + startX);
The distance, in bytes, between the start of one pixel row and the next in an image, including any unused space between them is finalWidth * 4 (red line between two other circles).
Let's fix the destBuffer:
vImage_Buffer destBuffer = {
(void *)destData+offset,
aspectRect.size.height,
aspectRect.size.width,
finalWidth * 4
};
Related
I am capturing my system's screen with AVCaptureSession and then create a video file out of the image buffers captured. It works fine.
Now I want to scale the image buffers by maintaining the aspect ratio for the video file's dimension. I have used the following code to scale the images.
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
if (pixelBuffer == NULL) { return; }
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
size_t finalWidth = 1080;
size_t finalHeight = 720;
size_t sourceWidth = CVPixelBufferGetWidth(imageBuffer);
size_t sourceHeight = CVPixelBufferGetHeight(imageBuffer);
CGRect aspectRect = AVMakeRectWithAspectRatioInsideRect(CGSizeMake(sourceWidth, sourceHeight), CGRectMake(0, 0, finalWidth, finalHeight));
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t startY = aspectRect.origin.y;
size_t yOffSet = (finalWidth*startY*4);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
void* destData = malloc(finalHeight * finalWidth * 4);
vImage_Buffer srcBuffer = { (void *)baseAddress, sourceHeight, sourceWidth, bytesPerRow};
vImage_Buffer destBuffer = { (void *)destData+yOffSet, aspectRect.size.height, aspectRect.size.width, aspectRect.size.width * 4};
vImage_Error err = vImageScale_ARGB8888(&srcBuffer, &destBuffer, NULL, 0);
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
OSType pixelFormat = CVPixelBufferGetPixelFormatType(imageBuffer);
CVImageBufferRef pixelBuffer1 = NULL;
CVReturn result = CVPixelBufferCreateWithBytes(NULL, finalWidth, finalHeight, pixelFormat, destData, finalWidth * 4, NULL, NULL, NULL, &pixelBuffer1);
}
I am able scale the image with the above code but the final image seems to be blurry compare to resizing the image with Preview application. Because of this the video is not clear.
This works fine if I change the output pixel format to RGB with below code.
output.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey, nil];
But I want the image buffers in YUV format (which is the default format for AVCaptureVideoDataOutput) since this will reduce the size of the buffer when transferring it over network.
Image after scaling:
Image resized with Preview application:
I have tried using vImageScale_CbCr8 instead of vImageScale_ARGB8888 but the resulting image didn't contain correct RGB values.
I have also noticed there is function to convert image format: vImageConvert_422YpCbYpCr8ToARGB8888(const vImage_Buffer *src, const vImage_Buffer *dest, const vImage_YpCbCrToARGB *info, const uint8_t permuteMap[4], const uint8_t alpha, vImage_Flags flags);
But I don't know what should be the values for vImage_YpCbCrToARGB and permuteMap as I don't know anything about image processing.
Expected Solution:
How to convert YUV pixel buffers to RGB buffers and back to YUV (or) How to scale YUV pixel buffers without affecting the RGB values.
After a lot search and going through different questions related to image rendering, found the below code to convert the pixel format of the image buffers. Thanks to the answer in this link.
CVPixelBufferRef imageBuffer;
CVPixelBufferCreate(kCFAllocatorDefault, sourceWidth, sourceHeight, kCVPixelFormatType_32ARGB, 0, &imageBuffer);
VTPixelTransferSessionTransferImage(pixelTransferSession, pixelBuffer, imageBuffer);
I am converting an Objective-C class that uses calloc() to create a buffer to Swift 3. Here is the pertinent part of the code I'm having issue with. In particular, the rawData assignment and usage.
CGImageRef imageRef = [capturedImage CGImage];
NSUInteger width = CGImageGetWidth(imageRef);
NSUInteger height = CGImageGetHeight(imageRef);
unsigned char *rawData = (unsigned char *)calloc(height * width * 4, sizeof(unsigned char));
NSUInteger bytesPerPixel = 4;
NSUInteger bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(rawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
NSUInteger byteIndex = (bytesPerRow * y) + x * bytesPerPixel;
CGFloat red = (CGFloat)rawData[byteIndex];
CGFloat green = (CGFloat)rawData[byteIndex + 1];
CGFloat blue = (CGFloat)rawData[byteIndex + 2];
From the CGBitmapContextCreate documentation:
data : UnsafeMutableRawPointer?
A pointer to the destination in memory where the drawing is to be rendered. The size of this memory block should be at least (bytesPerRow*height) bytes.
Pass NULL if you want this function to allocate memory for the bitmap. This frees you from managing your own memory, which reduces memory leak issues.
While you could figure out how to allocate the required memory block and obtain an UnsafeMutableRawPointer to it unless you have a good reason to allocate your own buffer just follow the documentation and pass NULL - no need to use calloc() at all.
HTH
I tried to extract all 3 channels from an image with vImageConvert_RGB888toPlanar8 and then put them back together with vImageConvert_Planar8toRGB888 but the image gets totally messed up. Why is that?
vImage_Buffer blueBuffer;
blueBuffer.data = (void*)blueImageData.bytes;
blueBuffer.width = size.width;
blueBuffer.height = size.height;
blueBuffer.rowBytes = [blueImageData length]/size.height;
vImage_Buffer rBuffer;
rBuffer.width = size.width;
rBuffer.height = size.height;
rBuffer.rowBytes = size.width;
void *rPixelBuffer = malloc(size.width * size.height);
if(rPixelBuffer == NULL)
{
NSLog(#"No pixelbuffer");
}
rBuffer.data = rPixelBuffer;
vImage_Buffer gBuffer;
gBuffer.width = size.width;
gBuffer.height = size.height;
gBuffer.rowBytes = size.width;
void *gPixelBuffer = malloc(size.width * size.height);
if(gPixelBuffer == NULL)
{
NSLog(#"No pixelbuffer");
}
gBuffer.data = gPixelBuffer;
vImage_Buffer bBuffer;
bBuffer.width = size.width;
bBuffer.height = size.height;
bBuffer.rowBytes = size.width;
void *bPixelBuffer = malloc(size.width * size.height);
if(bPixelBuffer == NULL)
{
NSLog(#"No pixelbuffer");
}
bBuffer.data = bPixelBuffer;
vImageConvert_RGB888toPlanar8(&blueBuffer, &rBuffer, &gBuffer, &bBuffer, kvImageNoFlags);
size_t destinationImageBytesLength = size.width*size.height*3;
const void* destinationImageBytes = valloc(destinationImageBytesLength);
NSData* destinationImageData = [[NSData alloc] initWithBytes:destinationImageBytes length:destinationImageBytesLength];
vImage_Buffer destinationBuffer;
destinationBuffer.data = (void*)destinationImageData.bytes;
destinationBuffer.width = size.width;
destinationBuffer.height = size.height;
destinationBuffer.rowBytes = [destinationImageData length]/size.height;
vImage_Error result = vImageConvert_Planar8toRGB888(&rBuffer, &gBuffer, &bBuffer, &destinationBuffer, 0);
NSImage* image = nil;
if(result == kvImageNoError)
{
//TODO: If you need color matching, use an appropriate colorspace here
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGDataProviderRef dataProvider = CGDataProviderCreateWithCFData((__bridge CFDataRef)(destinationImageData));
CGImageRef finalImageRef = CGImageCreate(size.width, size.height, 8, 24, destinationBuffer.rowBytes, colorSpace, kCGBitmapByteOrder32Big|kCGImageAlphaNone, dataProvider, NULL, NO, kCGRenderingIntentDefault);
CGColorSpaceRelease(colorSpace);
CGDataProviderRelease(dataProvider);
image = [[NSImage alloc] initWithCGImage:finalImageRef size:NSMakeSize(size.width, size.height)];
CGImageRelease(finalImageRef);
}
free((void*)destinationImageBytes);
return image;
Working with vImage means to work with pixels only. So you must never use the size of an image (or imageRep), you only use pixelsWide and pixelsHigh. Replace all size.width with pixelsWide and all size.height with pixelsHigh. Apple has example code for vImage and they use size values! Don't believe them! Not all Apple example codes are correct.
The size of an image or imageRep determines how big an image shall be drawn on the screen (or a printer). Size values have the dimension of a length and the units are meter, cm, inch or (as in Cocoa) 1/72 inch aka point. They are represented as float values.
PixelsWide and pixelsHigh have no dimension and no unit (they are simply numbers) and are represented as int values.
There may be more bugs in your code, but the first step should be to replace all size values.
Strictly speaking, you want kCGBitmapByteOrderDefault instead of kCGBitmapByteOrder32Big. 32Big doesn't make much sense for a 24 bit pixel format.
This seems like a weak link:
destinationBuffer.rowBytes = [destinationImageData length]/size.height;
Check to see it is the right number.
A picture of the output would help diagnose the problem. Also check the console to see if CG is giving you any spew. Did you try vImageCreateCGImageFromBuffer() with kvImagePrintDiagnosticsToConsole to see if it has anything to say?
Working with vImage means to work with pixels only. So you must never use the size of an image (or imageRep), you only use pixelsWide and pixelsHigh. Replace all size.width with pixelsWide and all size.height with pixelsHigh. Apple has example code for vImage and they use size values! Don't believe them! Not all Apple example codes are correct.
The size of an image or imageRep determines how big an image shall be drawn on the screen (or a printer). Size values have the dimension of a length and the units are meter, cm, inch or (as in Cocoa) 1/72 inch aka point. They are represented as float values.
PixelsWide and pixelsHigh have no dimension and no unit (they are simply numbers) and are represented as int values.
There may be more bugs in your code, but the first step should be to replace all size values.
I'm confused about one strange thing....I have an unsigned char array.... I allocate it using calloc and record some bytes data in it... but when I free this unsigned char and allocate it again, I see that it reserves the same address in memory which was allocated previous time. I understand why....But I cannot understand why the data that I'm trying to write there second time is not written...There is written the data that was written first time....Can anybody explain me this???????
unsigned char *rawData = (unsigned char*) calloc(height * width * 4, sizeof(unsigned char));
This is how I allocate it....
Actually my problem is that because of this allocation , which happens once every 2 secs I have memory leak...But when I try to free the allocated memory sector happens thing described above....:(
Please if anybody can help me....I would be so glad...
Here is the code...
- (unsigned char*) createBitmapContext:(UIImage*)anImage{
CGImageRef imageRef = [anImage CGImage];
NSUInteger width = CGImageGetWidth(imageRef);
NSUInteger height = CGImageGetHeight(imageRef);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
unsigned char *rawData = (unsigned char*) calloc(height * width * 4, sizeof(unsigned char));
bytesPerPixel = 4;
bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(rawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
CGContextRelease(context);
imageRef=nil;
return rawData; }
in this code there is no the part where I free(rawData), and because I cannot free it inside this method I tried to define rawData globally and free it after calling this method...but nothing interesting....
Please if anybody can help me....I would be so glad...
Ok, so this method is rendering a UIImage into a freshly allocated byte buffer and returning the buffer to the caller. Since you're allocating it with calloc, it will be initialised to 0, then overwritten with the image contents.
when I free this unsigned char and allocate it again, I see that it reserves the same address in memory which was allocated previous time
Yes, there are no guarantees about the location of the buffer in memory. Assuming you call free() on the returned memory, requesting the exact same size is quite likely to give you the same buffer back. But - how are you verifying the contents are not written over a second time? What is in the buffer?
my problem is that because of this allocation , which happens once every 2 secs I have memory leak...But when I try to free the allocated memory sector happens thing described above....:(
If there is a leak, it is likely in the code that calls this method, since there is no obvious leakage here. The semantics are obviously such that the caller is responsible for freeing the buffer. So how is that done?
Also, are you verifying that the CGBitmapContext is being correctly created? It is possible that some creation flags or parameters may result in an error. So add a check for context being valid (at least not nil). That could explain why the content is not being overwritten.
One easy way to ensure your memory is being freshly updated is to write your own data to it. You could fill the buffer with a counter, and verify this outside the method. For example, just before you return rawData:
static unsigned updateCounter = 0;
memset(rawData, updateCounter & 0xff, width*height*4);
This will cycle through writing 0-255 into the buffer, which you can easily verify.
Another thing - what are you trying to achieve with this code? There might be an easier way to achieve what you're trying to achieve. Returning bare buffers devoid of metadata is not necessarily the best way to manage your images.
So guys I solved this issue...First thing I've changed createBitmapContext method to this
- (void) createBitmapContext:(UIImage*)anImage andRawData:(unsigned char *)theRawData{
CGImageRef imageRef = [anImage CGImage];
NSUInteger width = CGImageGetWidth(imageRef);
NSUInteger height = CGImageGetHeight(imageRef);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// unsigned char *rawData = (unsigned char*) calloc(height * width * 4, sizeof(unsigned char));
bytesPerPixel = 4;
bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(theRawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
CGContextRelease(context);
imageRef=nil;
// return theRawData;}
then...besides this I missed the part where I assign newRawData to oldRawData and by this I was having two pointers to the same memory address...So from here came the issue... I changed this assignment part to this memcpy(rawDataForOldImage, rawDataForNewImage,newCapturedImage.size.width*newCapturedImage.size.height*4); and here the problem is solved....Thanks to all
I want to read a PNG file such that I can:
a) Access the raw bitmap data of the file, with no color space adjustment or alpha premultiply.
b) Based on that bitmap, display bit slices (any single bit of R, G, B, or A, across the whole image) in an image in the window. If I have the bitmap I can find the right bits, but what can I stuff them into to get them onscreen?
c) After some modification of the bitplanes, write a new PNG file, again with no adjustments.
This is only for certain specific images. The PNG is not expected to have any data other than simply RGBA-32.
From reading some similar questions here, I'm suspecting NSBitmapImageRep for the file read/write, and drawing in an NSView for the onscreen part. Does this sound right?
1.) You can use NSBitmapImageRep's -bitmapData to get the raw pixel data. Unfortunately, CG (NSBitmapImageRep's backend) does not support native unpremultiplication so you would have to unpremultiply yourself. The colorspace used in this will be the same as present in the file. Here is how to unpremultiply the image data:
NSBitmapImageRep *imageRep = [NSBitmapImageRep imageRepWithData:data];
NSInteger width = [imageRep pixelsWide];
NSInteger height = [imageRep pixelsHigh];
unsigned char *bytes = [imageRep bitmapData];
for (NSUInteger y = 0; y < width * height * 4; y += 4) { // bgra little endian + alpha first
uint8_t a, r, g, b;
if (imageRep.bitmapFormat & NSAlphaFirstBitmapFormat) {
a = bytes[y];
r = bytes[y+1];
g = bytes[y+2];
b = bytes[y+3];
} else {
r = bytes[y+0];
g = bytes[y+1];
b = bytes[y+2];
a = bytes[y+3];
}
// unpremultiply alpha if there is any
if (a > 0) {
if (!(imageRep.bitmapFormat & NSAlphaNonpremultipliedBitmapFormat)) {
float factor = 255.0f/a;
b *= factor;
g *= factor;
r *= factor;
}
} else {
b = 0;
g = 0;
r = 0;
}
bytes[y]=a; // for argb
bytes[y+1]=r;
bytes[y+2]=g;
bytes[y+3]=b;
}
2.) I couldn't think of a simple way to do this. You could make your own image drawing method that loops through the raw image data and generates a new image based on the values. Refer above to see how to start doing it.
3.) Here is a method to get a CGImage from raw data places (you can write the png to a file using native CG functions or convert it to NSBitmapImageRep if CG makes you uncomfortable)
static CGImageRef cgImageFrom(NSData *data, uint16_t width, uint16_t height) {
CGDataProviderRef provider = CGDataProviderCreateWithCFData((CFDataRef)data);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGImageAlphaFirst;
CGImageRef cgImage = CGImageCreate(width, height, 8, 32, 4 * width, colorSpace, bitmapInfo, provider, NULL, NO, kCGRenderingIntentDefault);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return cgImage;
}
You can create the NSData object out of the raw data object with +dataWithBytes:length:
I haven't ever worked in this area, but you may be able to use Image IO for this.