mmap() and newBufferWithBytesNoCopy causing IOAF code -536870211 error if the file is too small - objective-c

I noticed that, while generating a texture from an MTLBuffer created from mmap() via newBufferWithBytesNoCopy, if the size requested by the len argument to mmap, page aligned, is larger than the actual size of the file, page aligned, the mmap call succeeds, and the newBufferWithBytesNoCopy message does not result in a nil return or error, but when I pass the buffer to the GPU to copy the data to an MTLTexture, the following is printed to the console, and all GPU commands fail to perform any action:
Execution of the command buffer was aborted due to an error during execution. Internal Error (IOAF code -536870211)
Here is code to demonstrate the problem:
static id<MTLDevice> Device;
static id<MTLCommandQueue> Queue;
static id<MTLTexture> BlockTexture[3];
#define TEX_LEN_1 1 // These are all made 1 in this question for simplicity
#define TEX_LEN_2 1
#define TEX_LEN_4 1
#define TEX_SIZE ((TEX_LEN_1<<10)+(TEX_LEN_2<<11)+(TEX_LEN_4<<12))
#define PAGE_ALIGN(S) ((S)+PAGE_SIZE-1&~(PAGE_SIZE-1))
int main(void) {
if (!(Queue = [Device = MTLCreateSystemDefaultDevice() newCommandQueue]))
return EXIT_FAILURE;
#autoreleasepool {
const id<MTLBuffer> data = ({
void *const map = ({
NSFileHandle *const file = [NSFileHandle fileHandleForReadingAtPath:[NSBundle.mainBundle pathForResource:#"Content" ofType:nil]];
if (!file)
return EXIT_FAILURE;
mmap(NULL, TEX_SIZE, PROT_READ, MAP_SHARED, file.fileDescriptor, 0);
});
if (map == MAP_FAILED)
return errno;
[Device newBufferWithBytesNoCopy:map length:PAGE_ALIGN(TEX_SIZE) options:MTLResourceStorageModeShared deallocator:^(void *const ptr, const NSUInteger len){
munmap(ptr, len);
}];
});
if (!data)
return EXIT_FAILURE;
const id<MTLCommandBuffer> buffer = [Queue commandBuffer];
const id<MTLBlitCommandEncoder> encoder = [buffer blitCommandEncoder];
if (!encoder)
return EXIT_FAILURE;
{
MTLTextureDescriptor *const descriptor = [MTLTextureDescriptor new];
descriptor.width = descriptor.height = 32;
descriptor.mipmapLevelCount = 6;
descriptor.textureType = MTLTextureType2DArray;
descriptor.storageMode = MTLStorageModePrivate;
const enum MTLPixelFormat format[] = {MTLPixelFormatR8Unorm, MTLPixelFormatRG8Unorm, MTLPixelFormatRGBA8Unorm};
const NSUInteger len[] = {TEX_LEN_1, TEX_LEN_2, TEX_LEN_4};
for (NSUInteger i = 3, off = 0; i--;) {
descriptor.pixelFormat = format[i];
const NSUInteger l = descriptor.arrayLength = len[i];
const id<MTLTexture> texture = [Device newTextureWithDescriptor:descriptor];
if (!texture)
return EXIT_FAILURE;
const NSUInteger br = 32<<i, bi = 1024<<i;
for (NSUInteger j = 0; j < l; off += bi)
[encoder copyFromBuffer:data sourceOffset:off sourceBytesPerRow:br sourceBytesPerImage:bi sourceSize:(const MTLSize){32, 32, 1} toTexture:texture destinationSlice:j++ destinationLevel:0 destinationOrigin:(const MTLOrigin){0}];
[encoder generateMipmapsForTexture:BlockTexture[i] = texture];
}
}
[encoder endEncoding];
[buffer commit];
}
// Rest of code to initialize application (omitted)
}
In this case, the command will fail if the size of the actual Content file is less than 4097 bytes, assuming a 4096 page size. What is the most strange is that neither the mmap() nor the newBufferWithBytesNoCopy fails in this case, but the GPU execution fails so badly that any/all subsequent GPU calls also fail.
Is there a way to cause predictable behavior? I thought that mmap() space beyond the file was just valid 0 memory. Why is this apparently not the case if the space is being used by the GPU? At the very least, how can I detect GPU execution errors or invalid buffers like this to handle them gracefully, besides manually checking if the file is too small? Am I using these functions incorrectly somehow? Is something here undefined behavior?
My research efforts including Google searching for terms such as newBufferWithBytesNoCopy and/or mmap together with 536870211, and got absolutely no results. Now, this question is the only result for such searches.
My guess is this problem has to do with the inner workings of the GPU and/or the MTLBuffer implementation and/or mmap() and its underlying facilities. Not having access to these inner workings, I have no idea how to even start figuring out a solution. I would appreciate an expert to enlighten me as to what is actually going on behind the scenes causing this error, and how to avoid it (besides manually checking if the file is too big, as this is a 'workaround' but does not really fix the problem at its base, or at the very least how to gracefully detect GPU crashes of this type and abort the application gracefully.

Related

Const char * as block variable

Some relevant info I am on OSX using GCD in Objective-C. I have a background task that produces a very large const char * this is then re-introduced into a new background task. This cycle repeats essentially until the const char* is empty. Currently I am creating and using NSStrings in the blocks and then going back to char* immediately. As you can imagine this does a ton of unnecessary copying of all that.
I am wondering how __block variables work for non-objects or how I can get away from NSStrings?
Or
How is memory managed for non-object types?
It is currently just blowing up with ~2 gigs of memory all from the strings.
Here is how it currently looks:
-(void)doSomething:(NSString*)input{
__block NSString* blockCopy = input;
void (^blockTask)(void);
blockTask = ^{
const char* input = [blockCopy UTF8String];
//remainder will point to somewhere along input
const char* remainder = NULL;
myCoolCFunc(input,&remainder);
if(remainder != NULL && remainder[0] != '\0'){
//this is whats killing me the NSString creation of remainder
[self doSomething:#(remainder)];
}
}
/*...create background queue if needed */
dispatch_async(backgroundQueue,blockTask);
}
There is no need to use NSString at all and no need to use the __block attribute:
-(void)doSomething:(const char *)input{
void (^blockTask)(void);
blockTask = ^{
const char* remainder = NULL;
myCoolCFunc(input,&remainder);
if(remainder != NULL && remainder[0] != '\0'){
[self doSomething:remainder];
}
}
/*...create background queue if needed */
dispatch_async(backgroundQueue,blockTask);
}
There is also little need to use recursion either, as an iterative approach is also possible.
blockTask = ^{
const char* remainder = NULL;
while (YES) {
myCoolCFunc(input,&remainder);
if(remainder == NULL || remainder[0] == '\0')
break;
input = remainder;
}
}

Objective C stringWithCString "Method cache corrupted"

I am creating a method for executing shell commands. It looks like this:
NSString *cShellStr(NSString *command, int maxBufferSize) {
if (maxBufferSize<1) {
maxBufferSize = INT_MAX;
}
NSString *newCommand = [NSString stringWithFormat:#"%# 2>&1", command];
const char *cStrCommand = [newCommand cStringUsingEncoding:NSUTF8StringEncoding];
FILE *fp;
fp = popen(cStrCommand, "r");
if (fp == NULL) {
NSLog(#"Failed to open process");
return nil;
}
char *buffer = NULL;
buffer = (char*)malloc(4);
if (buffer == NULL) {
NSLog(#"Failed to allocate memory");
return nil;
}
while (!feof(fp)) {
buffer = realloc(buffer, sizeof(buffer)+1);
sprintf(buffer, "%s%c", buffer, fgetc(fp));
}
NSString *ret = [NSString stringWithCString:buffer encoding:NSASCIIStringEncoding];
fclose(fp);
free(buffer);
return ret;
}
However, it usually (strangely not always) gets this error at [NSString stringWithCString:buffer encoding:NSASCIIStringEncoding]:
objc[33674]: Method cache corrupted. This may be a message to an invalid object, or a memory error somewhere else.
objc[33674]: receiver 0x7fff7d62aec0, SEL 0x7fff99ac69f5, isa 0x7fff7d62aee8, cache 0x7fff7d62aef8, buckets 0x100200780, mask 0x3, occupied 0x2, wrap bucket 0x100200780
objc[33674]: receiver 0 bytes, buckets 64 bytes
objc[33674]: selector 'class'
objc[33674]: isa 'NSString'
objc[33674]: Method cache corrupted.
objc[33674]: Method cache corrupted. This may be a message to an invalid object, or a memory error somewhere else.
objc[33674]: receiver 0x7fff7d62aec0, SEL 0x7fff99aea2d4, isa 0x7fff7d62aee8, cache 0x7fff7d62aef8, buckets 0x100200780, mask 0x3, occupied 0x2, wrap bucket 0x100200780
objc[33674]: receiver 0 bytes, buckets 64 bytes
objc[33674]: selector 'stringWithCString:encoding:'
objc[33674]: isa 'NSString'
objc[33674]: Method cache corrupted.
I have stepped through it with lldb and confirmed that there are no other errors and that buffer contains exactly the text I would expect. Am I incorrectly using stringWithCString?
Extra info:
Xcode version:
Laptop info:
Clang version:
Apple LLVM version
6.0 (clang-600.0.51) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.3.0
Thread model: posix
while (!feof(fp)) {
buffer = realloc(buffer, sizeof(buffer)+1);
sprintf(buffer, "%s%c", buffer, fgetc(fp));
}
This loop should be rewritten as:
size_t len = 0, cap = 4;
buffer = malloc(cap); // TODO check NULL
int c;
while (EOF != (c = fgetc(fp))) {
if (len >= cap) {
cap += cap;
buffer = realloc(buffer, cap); // TODO check NULL
}
buffer[len++] = c;
}
// check feof/ferror
sprintf(buffer, "%s%c", buffer, fgetc(fp));
This is undefined behaviour because you are using buffer twice. Anything could go wrong, and lucky for you, it does. I'm saying "lucky for you" because with a bit of bad luck it would only go wrong in the hand of customers.
I didn't watch out; the issue that user3125367 found is ten times worse.

CGImageRef | Memory Consumption | Leak

I need to capture the desktop image and process its RGB Data, i am using Quartz API to do the same,
The problem what i am facing is, high mem usage,
please refer the function ,
Edit here,
This function is getting called through pThread ; something like this,
void ImageProcessing::thread(){
NSAutoreleasePool *pool=[[NSAutoreleasePool alloc]init];
ImgRef sImageRef
while( active){
**strong text**
if ([currentWnd getCurrentRGBImage:&currentImg]){
/* do something here */
}
}
[pool release];
}
-(bool)getCurrentRGBImage:(ImgRef *)pImgRef{
CGImageRef pCGImageRef;
pCGImageRef = [self getDesktopImage];
if ( !pCGImageRef ){
NSLog(#" got NULL Image ");
CGImageRelease(pCGImageRef);
pCGImageRef = NULL;
return NO;
}
// Create a bitmap rep from the image...
size_t width = CGImageGetWidth(pCGImageRef);
size_t height = CGImageGetHeight(pCGImageRef);
int bytesPerRow = CGImageGetBytesPerRow(pCGImageRef);
int bytesPerPixel = CGImageGetBitsPerPixel(pCGImageRef)/8;
CGDataProviderRef provider = CGImageGetDataProvider(pCGImageRef);
CFDataRef pData = CGDataProviderCopyData(provider);
const uint8_t* bytes = (const uint8_t *)CFDataGetBytePtr(pData);
/***** ------------- *********
Copy RAW Bytes to pImgRef
****************************/
CGDataProviderRelease(provider);
CFRelease(pData);
CGImageRelease(pCGImageRef);
pCGImageRef = NULL;
return YES;
}
and getDesktopImage function is
-(CGImageRef)getDesktopImage{
CGImageRef screenShot;
screenShot = CGWindowListCreateImage(CGRectInfinite, kCGWindowListOptionOnScreenOnly, kCGNullWindowID, kCGWindowImageDefault);
return screenShot;
}
The problem is, overall this function is consuming lot of memory, *pImgRef is having one and only one instance, there only pixel , RGB Manipulation,
Memory usage i am referring Mac Activity Monitor, is it reliable ?
the document Documentation for CGDataProviderRelease it is mentioned Application needs to maintain retaining and releasing of it, so i am Calling CGDAtaProviderRelease but getting message in the console
malloc reference count underflow for break on auto_refcount_underflow_error to debug
Should i not call CGDataProviderRelease() ? if i comment i don't get this message, so not sure, whether its bug in Quartz or in my code,
Also, this function is consuming hell lot of memory, if i check in the Activity monitor, is it safe to assuming that is really using this much memory ?
Since you do not retain, create, or copy the data provider, you should not release it.
I don't see anything that would cause excess memory consumption, but you left some of the code out.

Obscure memory error with ARC enabled

I'm creating an Objective-C library to talk with some external devices through USB.
When calling a certain method, it crashes at a random place inside the method or inside some C system functions (related to malloc or pthread) with one of the following error "invalid checksum for freed object", "autorelease pool page 0x1102032 corrupted", or even an unknown selector error (whereas the selector does exist).
Using Guard Malloc feature, it stops on this line with an EXEC_BAD_ACCESS error:
- (void)theMethod {
// some code
NSMutableData *payloads_pool = [NSMutableData dataWithLength:0x800];
NSUInteger payloads_pool_length = [payloads_pool length];
void *buffer = [payloads_pool mutableBytes];
memset(buffer, 0xCC, payloads_pool_length);
for (i = 0; i < 0x800; i += 0x40) {
unsigned int *buf = [payloads_pool mutableBytes];
(buf+i)[0] = 0x405; <==== STOP ON THIS LINE
(buf+i)[1] = 0x101;
(buf+i)[2] = 0x8402B001;
(buf+i)[3] = 0x8402EB01;
}
// some code
}
Since buf is unsigned int*, isn’t the pointer + in buf+i adding unsigned integers instead of bytes, thus seeking too far in memory?
Then I'd guess that the payloads_pool has fewer than 0x800 bytes in.
What is the value of i when it crashes?
Why isn't your for loop from 0 to payloads_pool.length?

How do you add to an AudioBufferList with an AVAssetReader?

I have been working on reading in an audio asset using AVAssetReader so that I can later play back the audio with an AUGraph with an AudioUnit callback. I have the AUGraph and AudioUnit callback working but it reads files from disk and if the file is too big it would take up too much memory and crash the app. So I am instead reading the asset directly and only a limited size. I will then manage it as a double buffer and get the AUGraph what it needs when it needs it.
(Note: I would love know if I can use Audio Queue Services and still use an AUGraph with AudioUnit callback so memory is managed for me by the iOS frameworks.)
My problem is that I do not have a good understanding of arrays, structs and pointers in C. The part where I need help is taking the individual AudioBufferList which holds onto a single AudioBuffer and add that data to another AudioBufferList which holds onto all of the data to be used later. I believe I need to use memcpy but it is not clear how to use it or even initialize an AudioBufferList for my purposes. I am using MixerHost for reference which is the sample project from Apple which reads in the file from disk.
I have uploaded my work in progress if you would like to load it up in Xcode. I've figured out most of what I need to get this done and once I have the data being collected all in one place I should be good to go.
Sample Project: MyAssetReader.zip
In the header you can see I declare the bufferList as a pointer to the struct.
#interface MyAssetReader : NSObject {
BOOL reading;
signed long sampleTotal;
Float64 totalDuration;
AudioBufferList *bufferList; // How should this be handled?
}
Then I allocate bufferList this way, largely borrowing from MixerHost...
UInt32 channelCount = [asset.tracks count];
if (channelCount > 1) {
NSLog(#"We have more than 1 channel!");
}
bufferList = (AudioBufferList *) malloc (
sizeof (AudioBufferList) + sizeof (AudioBuffer) * (channelCount - 1)
);
if (NULL == bufferList) {NSLog (#"*** malloc failure for allocating bufferList memory"); return;}
// initialize the mNumberBuffers member
bufferList->mNumberBuffers = channelCount;
// initialize the mBuffers member to 0
AudioBuffer emptyBuffer = {0};
size_t arrayIndex;
for (arrayIndex = 0; arrayIndex < channelCount; arrayIndex++) {
// set up the AudioBuffer structs in the buffer list
bufferList->mBuffers[arrayIndex] = emptyBuffer;
bufferList->mBuffers[arrayIndex].mNumberChannels = 1;
// How should mData be initialized???
bufferList->mBuffers[arrayIndex].mData = malloc(sizeof(AudioUnitSampleType));
}
Finally I loop through the reads.
int frameCount = 0;
CMSampleBufferRef nextBuffer;
while (assetReader.status == AVAssetReaderStatusReading) {
nextBuffer = [assetReaderOutput copyNextSampleBuffer];
AudioBufferList localBufferList;
CMBlockBufferRef blockBuffer;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(nextBuffer, NULL, &localBufferList, sizeof(localBufferList), NULL, NULL,
kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &blockBuffer);
// increase the number of total bites
bufferList->mBuffers[0].mDataByteSize += localBufferList.mBuffers[0].mDataByteSize;
// carefully copy the data into the buffer list
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
// get information about duration and position
//CMSampleBufferGet
CMItemCount sampleCount = CMSampleBufferGetNumSamples(nextBuffer);
Float64 duration = CMTimeGetSeconds(CMSampleBufferGetDuration(nextBuffer));
Float64 presTime = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(nextBuffer));
if (isnan(duration)) duration = 0.0;
if (isnan(presTime)) presTime = 0.0;
//NSLog(#"sampleCount: %ld", sampleCount);
//NSLog(#"duration: %f", duration);
//NSLog(#"presTime: %f", presTime);
self.sampleTotal += sampleCount;
self.totalDuration += duration;
frameCount++;
free(nextBuffer);
}
I am unsure about the what that I handle mDataByteSize and mData, especially with memcpy. Since mData is a void pointer this is an extra tricky area.
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
In this line I think it should be copying the value from the data in localBufferList to the position in the bufferList plus the number of frames to position the pointer where it should write the data. I have a couple of ideas on what I need to change to get this to work.
Since a void pointer is just 1 and not the size of the pointer for an AudioUnitSampleType I may need to multiply it also by sizeof(AudioUnitSampleType) to get the memcpy into the right position
I may not be using malloc properly to prepare mData but since I am not sure how many frames there will be I am not sure what to do to initialize it
Currently when I run this app it ends this function with an invalid pointer for bufferList.
I appreciate your help with making me better understand how to manage an AudioBufferList.
I've come up with my own answer. I decided to use an NSMutableData object which allows me to appendBytes from the CMSampleBufferRef after calling CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer to get an AudioBufferList.
[data appendBytes:localBufferList.mBuffers[0].mData length:localBufferList.mBuffers[0].mDataByteSize];
Once the read loop is done I have all of the data in my NSMutableData object. I then create and populate the AudioBufferList this way.
audioBufferList = (AudioBufferList *)malloc(sizeof(AudioBufferList));
if (NULL == audioBufferList) {
NSLog (#"*** malloc failure for allocating audioBufferList memory");
[data release];
return;
}
audioBufferList->mNumberBuffers = 1;
audioBufferList->mBuffers[0].mNumberChannels = channelCount;
audioBufferList->mBuffers[0].mDataByteSize = [data length];
audioBufferList->mBuffers[0].mData = (AudioUnitSampleType *)malloc([data length]);
if (NULL == audioBufferList->mBuffers[0].mData) {
NSLog (#"*** malloc failure for allocating mData memory");
[data release];
return;
}
memcpy(audioBufferList->mBuffers[0].mData, [data mutableBytes], [data length]);
[data release];
I'd appreciate a little code review on how I use malloc to create the struct and populate it. I am getting a EXC_BAD_ACCESS error sporadically but I cannot pinpoint where the error is just yet. Since I am using malloc on the struct I should not have to retain it anywhere. I do call "free" to release child elements within the struct and finally the struct itself everywhere that I use malloc.