glGrab for screen capture on Mac OS 10.7.3 with XCode 4.3.2 - objective-c

I am trying to integrate the glGrab code for screen capture on Mac OS under mentioned config and I am currently stuck at an all blue screen being rendered inside my window. I believe that there is some issue with how the image texture has been created, but can't tell what. I am just a couple of weeks old in OpenGL so please go easy on me if I missed something obvious.
I am using the glGrab code as it is except CGLSetFullScreen method (and not even CGLSetFullScreenOnDisplay) because these methods are now deprecated. So this one line of code has been commented out for the time being.
I have been doing some research on this topic since some time now and found another thread on stackoverflow which possibly could have been the complete answer, but it helped much nonetheless. Convert UIImage to CVImageBufferRef
A direct reference to the glGrab code is http://code.google.com/p/captureme/source/browse/trunk/glGrab.c

The answer to my above question is present below. So no more opengl or glGrab. Use what's best optimized for Mac OSX. This doesn't include the code for capturing the mouse pointer also, but I am sure that if you have landed on this page you're smart enough to figure it out by yourself. Or if someone reading this knows the solution then it's your chance to help the fraternity :) Also this code returns a CVPixelBufferRef. You may choose to send back either the CGImageRef or even the bytestream as it is, just tweak it to your liking. :
void swizzleBitmap(void *data, int rowBytes, int height) {
int top, bottom;
void * buffer;
void * topP;
void * bottomP;
void * base;
top = 0;
bottom = height - 1;
base = data;
buffer = malloc(rowBytes);
while (top < bottom) {
topP = (void *)((top * rowBytes) + (intptr_t)base);
bottomP = (void *)((bottom * rowBytes) + (intptr_t)base);
bcopy( topP, buffer, rowBytes );
bcopy( bottomP, topP, rowBytes );
bcopy( buffer, bottomP, rowBytes );
++top;
--bottom;
}
free(buffer);
}
CVImageBufferRef grabViaOpenGL() {
int bytewidth;
CGImageRef image = CGDisplayCreateImage(kCGDirectMainDisplay); // Main screenshot capture call
CGSize frameSize = CGSizeMake(CGImageGetWidth(image), CGImageGetHeight(image)); // Get screenshot bounds
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:NO], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:NO], kCVPixelBufferCGBitmapContextCompatibilityKey,
nil];
CVPixelBufferRef pxbuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, frameSize.width,
frameSize.height, kCVPixelFormatType_32ARGB, (CFDictionaryRef) options,
&pxbuffer);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, frameSize.width,
frameSize.height, 8, 4*frameSize.width, rgbColorSpace,
kCGImageAlphaNoneSkipLast);
CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image),
CGImageGetHeight(image)), image);
bytewidth = frameSize.width * 4; // Assume 4 bytes/pixel for now
bytewidth = (bytewidth + 3) & ~3; // Align to 4 bytes
swizzleBitmap(pxdata, bytewidth, frameSize.height); // Solution for ARGB madness
CGColorSpaceRelease(rgbColorSpace);
CGImageRelease(image);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}

Related

Image Quality getting affected on scaling the image using vImageScale_ARGB8888 - Cocoa Objective C

I am capturing my system's screen with AVCaptureSession and then create a video file out of the image buffers captured. It works fine.
Now I want to scale the image buffers by maintaining the aspect ratio for the video file's dimension. I have used the following code to scale the images.
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
if (pixelBuffer == NULL) { return; }
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
size_t finalWidth = 1080;
size_t finalHeight = 720;
size_t sourceWidth = CVPixelBufferGetWidth(imageBuffer);
size_t sourceHeight = CVPixelBufferGetHeight(imageBuffer);
CGRect aspectRect = AVMakeRectWithAspectRatioInsideRect(CGSizeMake(sourceWidth, sourceHeight), CGRectMake(0, 0, finalWidth, finalHeight));
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t startY = aspectRect.origin.y;
size_t yOffSet = (finalWidth*startY*4);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
void* destData = malloc(finalHeight * finalWidth * 4);
vImage_Buffer srcBuffer = { (void *)baseAddress, sourceHeight, sourceWidth, bytesPerRow};
vImage_Buffer destBuffer = { (void *)destData+yOffSet, aspectRect.size.height, aspectRect.size.width, aspectRect.size.width * 4};
vImage_Error err = vImageScale_ARGB8888(&srcBuffer, &destBuffer, NULL, 0);
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
OSType pixelFormat = CVPixelBufferGetPixelFormatType(imageBuffer);
CVImageBufferRef pixelBuffer1 = NULL;
CVReturn result = CVPixelBufferCreateWithBytes(NULL, finalWidth, finalHeight, pixelFormat, destData, finalWidth * 4, NULL, NULL, NULL, &pixelBuffer1);
}
I am able scale the image with the above code but the final image seems to be blurry compare to resizing the image with Preview application. Because of this the video is not clear.
This works fine if I change the output pixel format to RGB with below code.
output.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey, nil];
But I want the image buffers in YUV format (which is the default format for AVCaptureVideoDataOutput) since this will reduce the size of the buffer when transferring it over network.
Image after scaling:
Image resized with Preview application:
I have tried using vImageScale_CbCr8 instead of vImageScale_ARGB8888 but the resulting image didn't contain correct RGB values.
I have also noticed there is function to convert image format: vImageConvert_422YpCbYpCr8ToARGB8888(const vImage_Buffer *src, const vImage_Buffer *dest, const vImage_YpCbCrToARGB *info, const uint8_t permuteMap[4], const uint8_t alpha, vImage_Flags flags);
But I don't know what should be the values for vImage_YpCbCrToARGB and permuteMap as I don't know anything about image processing.
Expected Solution:
How to convert YUV pixel buffers to RGB buffers and back to YUV (or) How to scale YUV pixel buffers without affecting the RGB values.
After a lot search and going through different questions related to image rendering, found the below code to convert the pixel format of the image buffers. Thanks to the answer in this link.
CVPixelBufferRef imageBuffer;
CVPixelBufferCreate(kCFAllocatorDefault, sourceWidth, sourceHeight, kCVPixelFormatType_32ARGB, 0, &imageBuffer);
VTPixelTransferSessionTransferImage(pixelTransferSession, pixelBuffer, imageBuffer);

Open CV memory stacked (not released properly)

I am using 3rd party library for image processing, this method seems to be the cause of large memory usage (+30MB) everytime it executed, and it won't release properly. Repeated use of it ends up crashing the app (memory overload). The image used is directly from camera of my iP6.
+ (UIImage *)UIImageFromCVMat:(cv::Mat)cvMat
{
NSData *data = [NSData dataWithBytes:cvMat.data length:cvMat.elemSize() * cvMat.total()];
CGColorSpaceRef colorSpace;
if (cvMat.elemSize() == 1) {
colorSpace = CGColorSpaceCreateDeviceGray();
} else {
colorSpace = CGColorSpaceCreateDeviceRGB();
}
CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data);
CGImageRef imageRef = CGImageCreate(cvMat.cols, // Width
cvMat.rows, // Height
8, // Bits per component
8 * cvMat.elemSize(), // Bits per pixel
cvMat.step[0], // Bytes per row
colorSpace, // Colorspace
kCGImageAlphaNone | kCGBitmapByteOrderDefault, // Bitmap info flags
provider, // CGDataProviderRef
NULL, // Decode
false, // Should interpolate
kCGRenderingIntentDefault); // Intent
// UIImage *image = [[UIImage alloc] initWithCGImage:imageRef];
UIImage *image = [UIImage imageWithCGImage:imageRef];
CGImageRelease(imageRef);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return image;
}
I suspect the problem is here: (__bridge CFDataRef)data. I cant use CFRelease on it cause it make app crash. Project is running with ARC.
EDIT:
It seems the same code is also in openCV official website:
http://docs.opencv.org/2.4/doc/tutorials/ios/image_manipulation/image_manipulation.html
Gah!
EDIT 2
Here is the code how I use it (actually below code is also a part of the 3rd party lib, but i added some lines).
cv::Mat undistorted = cv::Mat( cvSize(maxWidth,maxHeight), CV_8UC4); // here nothing
cv::Mat original = [MMOpenCVHelper cvMatFromUIImage:_adjustedImage]; // here +30MB
//NSLog(#"%f %f %f %f",ptBottomLeft.x,ptBottomRight.x,ptTopRight.x,ptTopLeft.x);
cv::warpPerspective(original, undistorted,
cv::getPerspectiveTransform(src, dst), cvSize(maxWidth, maxHeight)); // here +16MB
_cropRect.hidden=YES;
#autoreleasepool {
_sourceImageView.image=[MMOpenCVHelper UIImageFromCVMat:undistorted]; // here +15MB (PROBLEM)
}
original.release(); // here -30MB (THIS IS OK)
undistorted.release(); // here -16MB (ok)
Guess it is a hard subject since not many people knows OpenCV that well. What I found is that most answer for the similar problem involves putting #autoreleasepool where this method is used. But seems to be not releasing memory either.
As temporary solution I resize the image fed to this method by half. At least app will last longer before it crash finally. It just works.

Memory leak with core graphics

I've been tasked to solve a memory leak with a custom objective class for a legacy app that uses garbage collection.
The class takes in NSData from a jpeg file and can thumb the image. There is a method to return a new NSData object with the newly resized image.
ImageThumber * imgt = [ImageThumber withNSData:dataObjectFromJpeg];
[imgt thumbImage:1024];
NSdata * smallImage = [imgt imageData];
[imgt thumbImage:256];
NSdata * extraSmallImage = [imgt imageData];
It does what it's supposed to do but it's been discovered that for every ImageThumber that's created it allocates a ImageIO_jpeg_Data object that's never deallocated. This was found in Instruments.
When created using ImageThumber withNSData:(NSData) it creates a CGImage and stores it in a private CGImageRef variable
I found that if thumbImage:(int) isn't called the ImageIO_jpeg_Data will deallocate when ImageThumber is deallocated which leads me to believe the problem lies somewhere within the thumbImage method. If thumbImage method is called multiple times it doesn't create extra ImageIO_jpeg_Data.
I have little experience with core graphics and garbage collection.
+(id)SAMImageDataWithNSData:(NSData *)data
{
SAMImageData * new = [[[self alloc] init] autorelease];
new.imageData = [NSMutableData dataWithData:data];
CFDataRef imgData = (CFDataRef)data;
CGDataProviderRef imgDataProvider;
imgDataProvider = CGDataProviderCreateWithCFData(imgData);
new->_cgImageRef = CGImageCreateWithJPEGDataProvider(imgDataProvider, NULL, true, kCGRenderingIntentDefault);
CGDataProviderRelease(imgDataProvider);
int width = (int)CGImageGetWidth(new->_cgImageRef);
int height = (int)CGImageGetHeight(new->_cgImageRef);
new.originalSize = NSMakeSize(width, height);
return new;
}
-(void)thumbImage:(int)length
{
/* simple logic to calculate new width and height */
//If the next line is commented out the problem doesn't exist.
//You just don't get the image resize.
[self resizeCGImageToWidth:newSize.width andHeight:newSize.height];
CFMutableDataRef workingData = (CFMutableDataRef)[[NSMutableData alloc] initWithCapacity:0];
CGImageDestinationRef dest;
dest = CGImageDestinationCreateWithData(workingData,kUTTypeJPEG,1,NULL);
CGImageDestinationAddImage(dest,_cgImageRef,NULL);
CGImageDestinationFinalize(dest);
CFRelease(dest);
self.imageData = (NSMutableData *)workingData;
}
This where I believe the problem exists:
- (void)resizeCGImageToWidth:(int)width andHeight:(int)height {
CGColorSpaceRef colorspace = CGImageGetColorSpace(_cgImageRef);
CGContextRef context = CGBitmapContextCreate(NULL, width, height,
CGImageGetBitsPerComponent(_cgImageRef),
CGImageGetBytesPerRow(_cgImageRef),
colorspace,
kCGImageAlphaPremultipliedFirst);
CGColorSpaceRelease(colorspace);
if(context == NULL)
return;
CGContextDrawImage(context, CGRectMake(0, 0, width, height), _cgImageRef);
_cgImageRef = CGBitmapContextCreateImage(context);
CGContextRelease(context);
}

CoreGraphics random crash

This is a little test program to duplicate an intermittent issue in a larger class. The real class creates 4 thumbs of various sizes.
This main.m program will crash 1 out of 5 times it's run with EXC_BAD_ACCESS and highlights CGImageRelease(imgRef); If i comment out CGImageRelease(imgRef) then the app experiences serious memory leaks but doesn't crash...
#import <Foundation/Foundation.h>
#import <Cocoa/Cocoa.h>
int main(int argc, const char * argv[])
{
#autoreleasepool {
NSString * image = #"/Users/xxx/Pictures/wallpaper/7gjMT.jpg";
NSData * imageData = [NSData dataWithContentsOfFile:image];
CFDataRef imgData = (__bridge CFDataRef)imageData;
CGImageRef imgRef;
CGDataProviderRef imgDataProvider = NULL;
imgDataProvider = CGDataProviderCreateWithCFData(imgData);
imgRef = CGImageCreateWithJPEGDataProvider(imgDataProvider, NULL, true, kCGRenderingIntentDefault);
CGDataProviderRelease(imgDataProvider);
for (int i = 0; i < 1000; i++) {
// create context, keeping original image properties
CGColorSpaceRef colorspace = CGImageGetColorSpace(imgRef);
CGContextRef context = CGBitmapContextCreate(NULL, 2560, 1440,
CGImageGetBitsPerComponent(imgRef),
CGImageGetBytesPerRow(imgRef),
colorspace,
kCGImageAlphaPremultipliedFirst);
CGColorSpaceRelease(colorspace);
// draw image to context (resizing it)
CGContextDrawImage(context, CGRectMake(0, 0, 2560, 1440), imgRef);
// extract resulting image from context
CGImageRef newImgRef;
newImgRef = CGBitmapContextCreateImage(context);
CGImageRelease(imgRef);
CGContextRelease(context);
imgRef = newImgRef;
}
}
return 0;
}
I found if I release the context first then 1 out of 10 failures it highlights CGImageGetBytesPerRow(imgRef) with the same error.
I added a breakpoint for malloc_error_break and got this on CGImageRelease:
Are CGImageRelease and CGImageRelease releasing a shared resource?
The main problem is almost certainly that you're releasing a colorspace that you don't own. CGImageGetColorSpace(imgRef) does not give you an ownership of the returned colorspace object, so you shouldn't be calling CGColorSpaceRelease(colorspace) later. (By the way, although I happened to get a different failure that clued me into the problem, the static analyzer catches this, too.)
As a secondary issue, I was getting failures to create the context because you're using an inappropriate bytes-per-row value. CGImageGetBytesPerRow(imgRef) is the bytes-per-row of that image, but that's only appropriate for the width of that image. Given that you're hard-coding a width rather than using the width of the image (since you're scaling), you should not be using the bytes-per-row of the image.
I guess it will work if you're scaling down, but it will waste space. If you're scaling up, it fails.
In any case, pass 0. That lets CGBitmapContextCreate() compute an optimal value.

I need help optimizing BGR888 blitting to NSView

This is best I've come up with for blitting a 24-bit BGR image out to an NSView.
I did trim a significant amount of CPU time by ensuring that the NSWindow host also had the same colorSpace.
I think there are 4 or 5 pixel copies going on here:
in the vImage conversion (required)
calling CGDataProviderCreateWithData
calling CGImageCreate
creating the NSBitmapImageRep bitmap
in the final blit with drawInRect (required)
Anyone want to chime in on improving it?
Any help would be much appreciated.
{
// one-time setup code
CGColorSpaceRef useColorSpace = nil;
int w = 1920;
int h = 1080;
[theWindow setColorSpace: [NSColorSpace genericRGBColorSpace]];
// setup vImage buffers (not listed here)
// srcBuffer is my 24-bit BGR image (malloc-ed to be w*h*3)
// dstBuffer is for the resulting 32-bit RGBA image (malloc-ed to be w*h*4)
...
// this is called # 30-60fps
if (!useColorSpace)
useColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
vImage_Error err = vImageConvert_BGR888toRGBA8888(srcBuffer, NULL, 0xff, dstBuffer, NO, 0);
CGDataProviderRef newProvider = CGDataProviderCreateWithData(NULL,dstBuffer->data,w*h*4,myReleaseProvider);
CGImageRef myImageRGBA = CGImageCreate(w, h, 8, 32, w*4, useColorSpace, kCGBitmapByteOrderDefault | kCGImageAlphaLast, newProvider, NULL, false, kCGRenderingIntentDefault);
CGDataProviderRelease(newProvider);
// store myImageRGBA in an array of frames (using NSObject wrappers) for later access (setNeedsDisplay:)
...
}
- (void)drawRect:(NSRect)dirtyRect
{
// this is called # 30-60fps
CGImageRef storedImage = ...; // retrieve from array
NSBitmapImageRep *repImg = [[NSBitmapImageRep alloc] initWithCGImage:storedImage];
CGRect myFrame = CGRectMake(0,0,CGImageGetWidth(storedImage),CGImageGetHeight(storedImage));
[repImg drawInRect:myFrame fromRect:myFrame operation:NSCompositeCopy fraction:1.0 respectFlipped:TRUE hints:nil];
// free image from array (not listed here)
}
// this is called when the CGDataProvider is ready to release its data
void myReleaseProvider (void *info, const void *data, size_t size)
{
if (data) {
free((void *)data);
data=nil;
}
}
Use CGColorSpaceCreateDeviceRGB instead of genericRGB to avoid colorspace conversion inside CG. Use kCGImageAlphaNoneSkipLast instead of kCGImageAlphaLast since we know alpha is opaque to allow for a copy instead of a blend.
After you make those changes, it would be useful to run an Instruments time profile on it to show where the time is going.