I need to capture the desktop image and process its RGB Data, i am using Quartz API to do the same,
The problem what i am facing is, high mem usage,
please refer the function ,
Edit here,
This function is getting called through pThread ; something like this,
void ImageProcessing::thread(){
NSAutoreleasePool *pool=[[NSAutoreleasePool alloc]init];
ImgRef sImageRef
while( active){
**strong text**
if ([currentWnd getCurrentRGBImage:¤tImg]){
/* do something here */
}
}
[pool release];
}
-(bool)getCurrentRGBImage:(ImgRef *)pImgRef{
CGImageRef pCGImageRef;
pCGImageRef = [self getDesktopImage];
if ( !pCGImageRef ){
NSLog(#" got NULL Image ");
CGImageRelease(pCGImageRef);
pCGImageRef = NULL;
return NO;
}
// Create a bitmap rep from the image...
size_t width = CGImageGetWidth(pCGImageRef);
size_t height = CGImageGetHeight(pCGImageRef);
int bytesPerRow = CGImageGetBytesPerRow(pCGImageRef);
int bytesPerPixel = CGImageGetBitsPerPixel(pCGImageRef)/8;
CGDataProviderRef provider = CGImageGetDataProvider(pCGImageRef);
CFDataRef pData = CGDataProviderCopyData(provider);
const uint8_t* bytes = (const uint8_t *)CFDataGetBytePtr(pData);
/***** ------------- *********
Copy RAW Bytes to pImgRef
****************************/
CGDataProviderRelease(provider);
CFRelease(pData);
CGImageRelease(pCGImageRef);
pCGImageRef = NULL;
return YES;
}
and getDesktopImage function is
-(CGImageRef)getDesktopImage{
CGImageRef screenShot;
screenShot = CGWindowListCreateImage(CGRectInfinite, kCGWindowListOptionOnScreenOnly, kCGNullWindowID, kCGWindowImageDefault);
return screenShot;
}
The problem is, overall this function is consuming lot of memory, *pImgRef is having one and only one instance, there only pixel , RGB Manipulation,
Memory usage i am referring Mac Activity Monitor, is it reliable ?
the document Documentation for CGDataProviderRelease it is mentioned Application needs to maintain retaining and releasing of it, so i am Calling CGDAtaProviderRelease but getting message in the console
malloc reference count underflow for break on auto_refcount_underflow_error to debug
Should i not call CGDataProviderRelease() ? if i comment i don't get this message, so not sure, whether its bug in Quartz or in my code,
Also, this function is consuming hell lot of memory, if i check in the Activity monitor, is it safe to assuming that is really using this much memory ?
Since you do not retain, create, or copy the data provider, you should not release it.
I don't see anything that would cause excess memory consumption, but you left some of the code out.
Related
I noticed that, while generating a texture from an MTLBuffer created from mmap() via newBufferWithBytesNoCopy, if the size requested by the len argument to mmap, page aligned, is larger than the actual size of the file, page aligned, the mmap call succeeds, and the newBufferWithBytesNoCopy message does not result in a nil return or error, but when I pass the buffer to the GPU to copy the data to an MTLTexture, the following is printed to the console, and all GPU commands fail to perform any action:
Execution of the command buffer was aborted due to an error during execution. Internal Error (IOAF code -536870211)
Here is code to demonstrate the problem:
static id<MTLDevice> Device;
static id<MTLCommandQueue> Queue;
static id<MTLTexture> BlockTexture[3];
#define TEX_LEN_1 1 // These are all made 1 in this question for simplicity
#define TEX_LEN_2 1
#define TEX_LEN_4 1
#define TEX_SIZE ((TEX_LEN_1<<10)+(TEX_LEN_2<<11)+(TEX_LEN_4<<12))
#define PAGE_ALIGN(S) ((S)+PAGE_SIZE-1&~(PAGE_SIZE-1))
int main(void) {
if (!(Queue = [Device = MTLCreateSystemDefaultDevice() newCommandQueue]))
return EXIT_FAILURE;
#autoreleasepool {
const id<MTLBuffer> data = ({
void *const map = ({
NSFileHandle *const file = [NSFileHandle fileHandleForReadingAtPath:[NSBundle.mainBundle pathForResource:#"Content" ofType:nil]];
if (!file)
return EXIT_FAILURE;
mmap(NULL, TEX_SIZE, PROT_READ, MAP_SHARED, file.fileDescriptor, 0);
});
if (map == MAP_FAILED)
return errno;
[Device newBufferWithBytesNoCopy:map length:PAGE_ALIGN(TEX_SIZE) options:MTLResourceStorageModeShared deallocator:^(void *const ptr, const NSUInteger len){
munmap(ptr, len);
}];
});
if (!data)
return EXIT_FAILURE;
const id<MTLCommandBuffer> buffer = [Queue commandBuffer];
const id<MTLBlitCommandEncoder> encoder = [buffer blitCommandEncoder];
if (!encoder)
return EXIT_FAILURE;
{
MTLTextureDescriptor *const descriptor = [MTLTextureDescriptor new];
descriptor.width = descriptor.height = 32;
descriptor.mipmapLevelCount = 6;
descriptor.textureType = MTLTextureType2DArray;
descriptor.storageMode = MTLStorageModePrivate;
const enum MTLPixelFormat format[] = {MTLPixelFormatR8Unorm, MTLPixelFormatRG8Unorm, MTLPixelFormatRGBA8Unorm};
const NSUInteger len[] = {TEX_LEN_1, TEX_LEN_2, TEX_LEN_4};
for (NSUInteger i = 3, off = 0; i--;) {
descriptor.pixelFormat = format[i];
const NSUInteger l = descriptor.arrayLength = len[i];
const id<MTLTexture> texture = [Device newTextureWithDescriptor:descriptor];
if (!texture)
return EXIT_FAILURE;
const NSUInteger br = 32<<i, bi = 1024<<i;
for (NSUInteger j = 0; j < l; off += bi)
[encoder copyFromBuffer:data sourceOffset:off sourceBytesPerRow:br sourceBytesPerImage:bi sourceSize:(const MTLSize){32, 32, 1} toTexture:texture destinationSlice:j++ destinationLevel:0 destinationOrigin:(const MTLOrigin){0}];
[encoder generateMipmapsForTexture:BlockTexture[i] = texture];
}
}
[encoder endEncoding];
[buffer commit];
}
// Rest of code to initialize application (omitted)
}
In this case, the command will fail if the size of the actual Content file is less than 4097 bytes, assuming a 4096 page size. What is the most strange is that neither the mmap() nor the newBufferWithBytesNoCopy fails in this case, but the GPU execution fails so badly that any/all subsequent GPU calls also fail.
Is there a way to cause predictable behavior? I thought that mmap() space beyond the file was just valid 0 memory. Why is this apparently not the case if the space is being used by the GPU? At the very least, how can I detect GPU execution errors or invalid buffers like this to handle them gracefully, besides manually checking if the file is too small? Am I using these functions incorrectly somehow? Is something here undefined behavior?
My research efforts including Google searching for terms such as newBufferWithBytesNoCopy and/or mmap together with 536870211, and got absolutely no results. Now, this question is the only result for such searches.
My guess is this problem has to do with the inner workings of the GPU and/or the MTLBuffer implementation and/or mmap() and its underlying facilities. Not having access to these inner workings, I have no idea how to even start figuring out a solution. I would appreciate an expert to enlighten me as to what is actually going on behind the scenes causing this error, and how to avoid it (besides manually checking if the file is too big, as this is a 'workaround' but does not really fix the problem at its base, or at the very least how to gracefully detect GPU crashes of this type and abort the application gracefully.
This is a little test program to duplicate an intermittent issue in a larger class. The real class creates 4 thumbs of various sizes.
This main.m program will crash 1 out of 5 times it's run with EXC_BAD_ACCESS and highlights CGImageRelease(imgRef); If i comment out CGImageRelease(imgRef) then the app experiences serious memory leaks but doesn't crash...
#import <Foundation/Foundation.h>
#import <Cocoa/Cocoa.h>
int main(int argc, const char * argv[])
{
#autoreleasepool {
NSString * image = #"/Users/xxx/Pictures/wallpaper/7gjMT.jpg";
NSData * imageData = [NSData dataWithContentsOfFile:image];
CFDataRef imgData = (__bridge CFDataRef)imageData;
CGImageRef imgRef;
CGDataProviderRef imgDataProvider = NULL;
imgDataProvider = CGDataProviderCreateWithCFData(imgData);
imgRef = CGImageCreateWithJPEGDataProvider(imgDataProvider, NULL, true, kCGRenderingIntentDefault);
CGDataProviderRelease(imgDataProvider);
for (int i = 0; i < 1000; i++) {
// create context, keeping original image properties
CGColorSpaceRef colorspace = CGImageGetColorSpace(imgRef);
CGContextRef context = CGBitmapContextCreate(NULL, 2560, 1440,
CGImageGetBitsPerComponent(imgRef),
CGImageGetBytesPerRow(imgRef),
colorspace,
kCGImageAlphaPremultipliedFirst);
CGColorSpaceRelease(colorspace);
// draw image to context (resizing it)
CGContextDrawImage(context, CGRectMake(0, 0, 2560, 1440), imgRef);
// extract resulting image from context
CGImageRef newImgRef;
newImgRef = CGBitmapContextCreateImage(context);
CGImageRelease(imgRef);
CGContextRelease(context);
imgRef = newImgRef;
}
}
return 0;
}
I found if I release the context first then 1 out of 10 failures it highlights CGImageGetBytesPerRow(imgRef) with the same error.
I added a breakpoint for malloc_error_break and got this on CGImageRelease:
Are CGImageRelease and CGImageRelease releasing a shared resource?
The main problem is almost certainly that you're releasing a colorspace that you don't own. CGImageGetColorSpace(imgRef) does not give you an ownership of the returned colorspace object, so you shouldn't be calling CGColorSpaceRelease(colorspace) later. (By the way, although I happened to get a different failure that clued me into the problem, the static analyzer catches this, too.)
As a secondary issue, I was getting failures to create the context because you're using an inappropriate bytes-per-row value. CGImageGetBytesPerRow(imgRef) is the bytes-per-row of that image, but that's only appropriate for the width of that image. Given that you're hard-coding a width rather than using the width of the image (since you're scaling), you should not be using the bytes-per-row of the image.
I guess it will work if you're scaling down, but it will waste space. If you're scaling up, it fails.
In any case, pass 0. That lets CGBitmapContextCreate() compute an optimal value.
This is best I've come up with for blitting a 24-bit BGR image out to an NSView.
I did trim a significant amount of CPU time by ensuring that the NSWindow host also had the same colorSpace.
I think there are 4 or 5 pixel copies going on here:
in the vImage conversion (required)
calling CGDataProviderCreateWithData
calling CGImageCreate
creating the NSBitmapImageRep bitmap
in the final blit with drawInRect (required)
Anyone want to chime in on improving it?
Any help would be much appreciated.
{
// one-time setup code
CGColorSpaceRef useColorSpace = nil;
int w = 1920;
int h = 1080;
[theWindow setColorSpace: [NSColorSpace genericRGBColorSpace]];
// setup vImage buffers (not listed here)
// srcBuffer is my 24-bit BGR image (malloc-ed to be w*h*3)
// dstBuffer is for the resulting 32-bit RGBA image (malloc-ed to be w*h*4)
...
// this is called # 30-60fps
if (!useColorSpace)
useColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
vImage_Error err = vImageConvert_BGR888toRGBA8888(srcBuffer, NULL, 0xff, dstBuffer, NO, 0);
CGDataProviderRef newProvider = CGDataProviderCreateWithData(NULL,dstBuffer->data,w*h*4,myReleaseProvider);
CGImageRef myImageRGBA = CGImageCreate(w, h, 8, 32, w*4, useColorSpace, kCGBitmapByteOrderDefault | kCGImageAlphaLast, newProvider, NULL, false, kCGRenderingIntentDefault);
CGDataProviderRelease(newProvider);
// store myImageRGBA in an array of frames (using NSObject wrappers) for later access (setNeedsDisplay:)
...
}
- (void)drawRect:(NSRect)dirtyRect
{
// this is called # 30-60fps
CGImageRef storedImage = ...; // retrieve from array
NSBitmapImageRep *repImg = [[NSBitmapImageRep alloc] initWithCGImage:storedImage];
CGRect myFrame = CGRectMake(0,0,CGImageGetWidth(storedImage),CGImageGetHeight(storedImage));
[repImg drawInRect:myFrame fromRect:myFrame operation:NSCompositeCopy fraction:1.0 respectFlipped:TRUE hints:nil];
// free image from array (not listed here)
}
// this is called when the CGDataProvider is ready to release its data
void myReleaseProvider (void *info, const void *data, size_t size)
{
if (data) {
free((void *)data);
data=nil;
}
}
Use CGColorSpaceCreateDeviceRGB instead of genericRGB to avoid colorspace conversion inside CG. Use kCGImageAlphaNoneSkipLast instead of kCGImageAlphaLast since we know alpha is opaque to allow for a copy instead of a blend.
After you make those changes, it would be useful to run an Instruments time profile on it to show where the time is going.
Im new developer on ObjC and trying to make a fill color app. When I touch on a image, the color will be changed but I got the merory leak with this function need your help:
-(void) updateImageWithColorSelected:(int) pos{
CGImageRef imageRef = self.basicImage.CGImage;
NSData *data = CGDataProviderCopyData(CGImagerGetDataProvider(imageRef));//leak here
Byte *pixels = (Byte *)[data bytes];
//change color...
for(int i = 0; i< IMG_SIZE; i++){
pixels[j] = 255;
}
CGDataProvider provider = CGDataProviderCreateWithData( NULL, pixels, [data length], NULL];
CGImageRef newImageRef = CGImageCreate(w,h....);
self.basicImage = [UIImage imageWithCGImage:newImageRef];
//release newImageRef
CGImagerRelease(newImageRef);
// set basic image to img
[self.img setImage:self.basicImage];
data = nil;
[data release];
}
I try to remove all the code except NSData *data = CGDataProviderCopyData and the app still leak.
Do you guys have any idea how to release "data" ?
Thank you in advance,
}
// set basic image to img
[self.img setImage:self.basicImage];
data = nil;
[data release];
}
You're sending release to a nil pointer.
[data release];
data = nil;
}
This will do better.
Edit: the issue with CGDataProviderCreateWithData
When data is released, the data pointer you passed to CGDataProviderCreateWithData becomes invalid. This is expected. The proper use of this function requires you allocate a buffer for the data and provide a callback to release the data when the provider is released.
The best solution for you is to use CGDataProviderCreateWithCFData instead, taking advantage of the toll-free bridging between Foundation and CoreFoundation objects.
Use:
CGDataProvider provider = CGDataProviderCreateWithCFData( (CFDataRef) data );
Note that at present the data provider created by the call to CGDataProviderCreateWithData() or CGDataProviderCreateWithCFData() is also being leaked, and should be released by calling CGDataProviderRelease(). (This leak is undoubtedly minor compared to the original leaked data.)
I have been working on reading in an audio asset using AVAssetReader so that I can later play back the audio with an AUGraph with an AudioUnit callback. I have the AUGraph and AudioUnit callback working but it reads files from disk and if the file is too big it would take up too much memory and crash the app. So I am instead reading the asset directly and only a limited size. I will then manage it as a double buffer and get the AUGraph what it needs when it needs it.
(Note: I would love know if I can use Audio Queue Services and still use an AUGraph with AudioUnit callback so memory is managed for me by the iOS frameworks.)
My problem is that I do not have a good understanding of arrays, structs and pointers in C. The part where I need help is taking the individual AudioBufferList which holds onto a single AudioBuffer and add that data to another AudioBufferList which holds onto all of the data to be used later. I believe I need to use memcpy but it is not clear how to use it or even initialize an AudioBufferList for my purposes. I am using MixerHost for reference which is the sample project from Apple which reads in the file from disk.
I have uploaded my work in progress if you would like to load it up in Xcode. I've figured out most of what I need to get this done and once I have the data being collected all in one place I should be good to go.
Sample Project: MyAssetReader.zip
In the header you can see I declare the bufferList as a pointer to the struct.
#interface MyAssetReader : NSObject {
BOOL reading;
signed long sampleTotal;
Float64 totalDuration;
AudioBufferList *bufferList; // How should this be handled?
}
Then I allocate bufferList this way, largely borrowing from MixerHost...
UInt32 channelCount = [asset.tracks count];
if (channelCount > 1) {
NSLog(#"We have more than 1 channel!");
}
bufferList = (AudioBufferList *) malloc (
sizeof (AudioBufferList) + sizeof (AudioBuffer) * (channelCount - 1)
);
if (NULL == bufferList) {NSLog (#"*** malloc failure for allocating bufferList memory"); return;}
// initialize the mNumberBuffers member
bufferList->mNumberBuffers = channelCount;
// initialize the mBuffers member to 0
AudioBuffer emptyBuffer = {0};
size_t arrayIndex;
for (arrayIndex = 0; arrayIndex < channelCount; arrayIndex++) {
// set up the AudioBuffer structs in the buffer list
bufferList->mBuffers[arrayIndex] = emptyBuffer;
bufferList->mBuffers[arrayIndex].mNumberChannels = 1;
// How should mData be initialized???
bufferList->mBuffers[arrayIndex].mData = malloc(sizeof(AudioUnitSampleType));
}
Finally I loop through the reads.
int frameCount = 0;
CMSampleBufferRef nextBuffer;
while (assetReader.status == AVAssetReaderStatusReading) {
nextBuffer = [assetReaderOutput copyNextSampleBuffer];
AudioBufferList localBufferList;
CMBlockBufferRef blockBuffer;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(nextBuffer, NULL, &localBufferList, sizeof(localBufferList), NULL, NULL,
kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &blockBuffer);
// increase the number of total bites
bufferList->mBuffers[0].mDataByteSize += localBufferList.mBuffers[0].mDataByteSize;
// carefully copy the data into the buffer list
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
// get information about duration and position
//CMSampleBufferGet
CMItemCount sampleCount = CMSampleBufferGetNumSamples(nextBuffer);
Float64 duration = CMTimeGetSeconds(CMSampleBufferGetDuration(nextBuffer));
Float64 presTime = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(nextBuffer));
if (isnan(duration)) duration = 0.0;
if (isnan(presTime)) presTime = 0.0;
//NSLog(#"sampleCount: %ld", sampleCount);
//NSLog(#"duration: %f", duration);
//NSLog(#"presTime: %f", presTime);
self.sampleTotal += sampleCount;
self.totalDuration += duration;
frameCount++;
free(nextBuffer);
}
I am unsure about the what that I handle mDataByteSize and mData, especially with memcpy. Since mData is a void pointer this is an extra tricky area.
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
In this line I think it should be copying the value from the data in localBufferList to the position in the bufferList plus the number of frames to position the pointer where it should write the data. I have a couple of ideas on what I need to change to get this to work.
Since a void pointer is just 1 and not the size of the pointer for an AudioUnitSampleType I may need to multiply it also by sizeof(AudioUnitSampleType) to get the memcpy into the right position
I may not be using malloc properly to prepare mData but since I am not sure how many frames there will be I am not sure what to do to initialize it
Currently when I run this app it ends this function with an invalid pointer for bufferList.
I appreciate your help with making me better understand how to manage an AudioBufferList.
I've come up with my own answer. I decided to use an NSMutableData object which allows me to appendBytes from the CMSampleBufferRef after calling CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer to get an AudioBufferList.
[data appendBytes:localBufferList.mBuffers[0].mData length:localBufferList.mBuffers[0].mDataByteSize];
Once the read loop is done I have all of the data in my NSMutableData object. I then create and populate the AudioBufferList this way.
audioBufferList = (AudioBufferList *)malloc(sizeof(AudioBufferList));
if (NULL == audioBufferList) {
NSLog (#"*** malloc failure for allocating audioBufferList memory");
[data release];
return;
}
audioBufferList->mNumberBuffers = 1;
audioBufferList->mBuffers[0].mNumberChannels = channelCount;
audioBufferList->mBuffers[0].mDataByteSize = [data length];
audioBufferList->mBuffers[0].mData = (AudioUnitSampleType *)malloc([data length]);
if (NULL == audioBufferList->mBuffers[0].mData) {
NSLog (#"*** malloc failure for allocating mData memory");
[data release];
return;
}
memcpy(audioBufferList->mBuffers[0].mData, [data mutableBytes], [data length]);
[data release];
I'd appreciate a little code review on how I use malloc to create the struct and populate it. I am getting a EXC_BAD_ACCESS error sporadically but I cannot pinpoint where the error is just yet. Since I am using malloc on the struct I should not have to retain it anywhere. I do call "free" to release child elements within the struct and finally the struct itself everywhere that I use malloc.