memory is growing in audio buffer code - objective-c

I have a code that we use many times with our apps, its a class that take the buffer samples and process it ,then send back notification to the main class.
The code is c and objective-c.
It works just great, but there is a memory growing which i can see in instruments-allocations tool. the "overall bytes" is keep growing, in 100k a second. becuase of some parts of the code that i know who they are .
this is the callback function, with the line that makes problems.
it happens many times a second.
I also dont really understand where to put my *pool :
static OSStatus recordingCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
AudioBuffer buffer;
buffer.mNumberChannels = 1;
buffer.mDataByteSize = inNumberFrames * 2;
//NSLog(#"%ld",inNumberFrames);
buffer.mData = malloc( inNumberFrames * 2 );
// Put buffer in a AudioBufferList
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = buffer;
// block A
OSStatus status;
status = AudioUnitRender(audioUnit,
ioActionFlags,
inTimeStamp,
inBusNumber,
inNumberFrames,
&bufferList);
//end block A
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
int16_t *q = (int16_t *)(&bufferList)->mBuffers[0].mData;
int16_t average ;
for(int i=0; i < inNumberFrames; i++)
{
average=q[i];
if(average>100) // lineB
reducer++;
//blockC
if(reducer==150 )
{
average= preSignal + alpha*(average-preSignal);
//NSLog(#"average:%d",average);
//call scene
[dict setObject:[NSNumber numberWithInt:average] forKey:#"amp" ] ;
[[NSNotificationCenter defaultCenter] postNotificationName:#"DigitalArrived" object:nil userInfo:dict];
reducer=0;
preSignal=average;
}
//end blockC
}
free(buffer.mData);
[pool release];
return noErr;
}
OK:
ignore blockC for a second.
removing blockA and lineB solve it all.
removing only one of them- leaks.
i just cant undetstand what is growing here .

Just a guess, but allocating a new NSAutoreleasePool inside of your recording callback function (which is a super time-critical function) is probably a bad idea.
Actually, why are you doing this here at all? Shouldn't you just have one pool for the entire app, in your main.m? This is probably causing some of your leaks.

You should not do anything the requires memory allocation inside an Audio Unit render callback. The real-time requirements are too tight for using generic Objective C.
Since you should not allocate a pool, or any other memory, inside an audio unit callback, you should not use any Objective C methods that potentially or actually create any objects, such as dictionary modifications or notification creation. You may have to drop back to using plain C inside the render callback (set a flag), and do your Objective C messaging outside the render callback in another thread (after polling the flag(s) in a timer callback, for instance).

Related

mmap() and newBufferWithBytesNoCopy causing IOAF code -536870211 error if the file is too small

I noticed that, while generating a texture from an MTLBuffer created from mmap() via newBufferWithBytesNoCopy, if the size requested by the len argument to mmap, page aligned, is larger than the actual size of the file, page aligned, the mmap call succeeds, and the newBufferWithBytesNoCopy message does not result in a nil return or error, but when I pass the buffer to the GPU to copy the data to an MTLTexture, the following is printed to the console, and all GPU commands fail to perform any action:
Execution of the command buffer was aborted due to an error during execution. Internal Error (IOAF code -536870211)
Here is code to demonstrate the problem:
static id<MTLDevice> Device;
static id<MTLCommandQueue> Queue;
static id<MTLTexture> BlockTexture[3];
#define TEX_LEN_1 1 // These are all made 1 in this question for simplicity
#define TEX_LEN_2 1
#define TEX_LEN_4 1
#define TEX_SIZE ((TEX_LEN_1<<10)+(TEX_LEN_2<<11)+(TEX_LEN_4<<12))
#define PAGE_ALIGN(S) ((S)+PAGE_SIZE-1&~(PAGE_SIZE-1))
int main(void) {
if (!(Queue = [Device = MTLCreateSystemDefaultDevice() newCommandQueue]))
return EXIT_FAILURE;
#autoreleasepool {
const id<MTLBuffer> data = ({
void *const map = ({
NSFileHandle *const file = [NSFileHandle fileHandleForReadingAtPath:[NSBundle.mainBundle pathForResource:#"Content" ofType:nil]];
if (!file)
return EXIT_FAILURE;
mmap(NULL, TEX_SIZE, PROT_READ, MAP_SHARED, file.fileDescriptor, 0);
});
if (map == MAP_FAILED)
return errno;
[Device newBufferWithBytesNoCopy:map length:PAGE_ALIGN(TEX_SIZE) options:MTLResourceStorageModeShared deallocator:^(void *const ptr, const NSUInteger len){
munmap(ptr, len);
}];
});
if (!data)
return EXIT_FAILURE;
const id<MTLCommandBuffer> buffer = [Queue commandBuffer];
const id<MTLBlitCommandEncoder> encoder = [buffer blitCommandEncoder];
if (!encoder)
return EXIT_FAILURE;
{
MTLTextureDescriptor *const descriptor = [MTLTextureDescriptor new];
descriptor.width = descriptor.height = 32;
descriptor.mipmapLevelCount = 6;
descriptor.textureType = MTLTextureType2DArray;
descriptor.storageMode = MTLStorageModePrivate;
const enum MTLPixelFormat format[] = {MTLPixelFormatR8Unorm, MTLPixelFormatRG8Unorm, MTLPixelFormatRGBA8Unorm};
const NSUInteger len[] = {TEX_LEN_1, TEX_LEN_2, TEX_LEN_4};
for (NSUInteger i = 3, off = 0; i--;) {
descriptor.pixelFormat = format[i];
const NSUInteger l = descriptor.arrayLength = len[i];
const id<MTLTexture> texture = [Device newTextureWithDescriptor:descriptor];
if (!texture)
return EXIT_FAILURE;
const NSUInteger br = 32<<i, bi = 1024<<i;
for (NSUInteger j = 0; j < l; off += bi)
[encoder copyFromBuffer:data sourceOffset:off sourceBytesPerRow:br sourceBytesPerImage:bi sourceSize:(const MTLSize){32, 32, 1} toTexture:texture destinationSlice:j++ destinationLevel:0 destinationOrigin:(const MTLOrigin){0}];
[encoder generateMipmapsForTexture:BlockTexture[i] = texture];
}
}
[encoder endEncoding];
[buffer commit];
}
// Rest of code to initialize application (omitted)
}
In this case, the command will fail if the size of the actual Content file is less than 4097 bytes, assuming a 4096 page size. What is the most strange is that neither the mmap() nor the newBufferWithBytesNoCopy fails in this case, but the GPU execution fails so badly that any/all subsequent GPU calls also fail.
Is there a way to cause predictable behavior? I thought that mmap() space beyond the file was just valid 0 memory. Why is this apparently not the case if the space is being used by the GPU? At the very least, how can I detect GPU execution errors or invalid buffers like this to handle them gracefully, besides manually checking if the file is too small? Am I using these functions incorrectly somehow? Is something here undefined behavior?
My research efforts including Google searching for terms such as newBufferWithBytesNoCopy and/or mmap together with 536870211, and got absolutely no results. Now, this question is the only result for such searches.
My guess is this problem has to do with the inner workings of the GPU and/or the MTLBuffer implementation and/or mmap() and its underlying facilities. Not having access to these inner workings, I have no idea how to even start figuring out a solution. I would appreciate an expert to enlighten me as to what is actually going on behind the scenes causing this error, and how to avoid it (besides manually checking if the file is too big, as this is a 'workaround' but does not really fix the problem at its base, or at the very least how to gracefully detect GPU crashes of this type and abort the application gracefully.

Why does the position of #autoreleasepool matter?

I'm having trouble understanding how #autoreleasepool work. Consider the following example in which I am creating an AVAssetReader for an audiofile. To make the memory impact matter, I repeated this step 1000 times.
#import <Foundation/Foundation.h>
#import <AVFoundation/AVFoundation.h>
void memoryTest() {
NSURL *url = [[NSURL alloc] initWithString:#"path-to-mp3-file"];
AVURLAsset *asset = [[AVURLAsset alloc] initWithURL:url options:nil];
AVAssetReader *reader = [[AVAssetReader alloc] initWithAsset:asset error:NULL];
}
int main(int argc, const char * argv[]) {
// Breakpoint here (A)
#autoreleasepool {
for(int i = 0; i < 1000; i++) {
memoryTest();
}
}
// Breakpoint here (B)
return 0;
}
Before and after the loop at breakpoint A and breakpoint B I took a look at the memory usage of my application in the Xcode Debug Navigation. At point A my App consumes about 1.5MB of memory. At point B it is about 80MB. Interestingly the memory usage at B drops to about 4MB when I put the autoreleasepool inside the loop like so:
for(int i = 0; i < 1000; i++) {
#autoreleasepool { memoryTest(); }
}
Why does the position matter? (The breakpoint - in both cases - is outside of the autoreleasepool!) In either case the consumed memory at point B is proportional to the number of loops. Am I missing something else here to free up its memory?
I thought of the memory chart in the navigator to be delayed, but adding usleep just before the breakpoint does not change anything. If I change memoryTest() to
void memoryTest() {
NSURL *url = [NSURL URLWithString:#"path-to-mp3-file"];
}
the position of #autoreleasepool does not matter. Isn't that weird?
I am using Xcode 6, OS X SDK 10.10 with ARC enabled.
Temporary memory is being created and autorelease'ed by the code inside the loop by your code and the API calls it makes. This memory is not released until the pool is drained. The usual point it is is drained is in the run loop. But your loop does not allow the run loop to execute so by placing the #autoreleasepool inside the loop the pool can drain as your code runs.

CGImageRef | Memory Consumption | Leak

I need to capture the desktop image and process its RGB Data, i am using Quartz API to do the same,
The problem what i am facing is, high mem usage,
please refer the function ,
Edit here,
This function is getting called through pThread ; something like this,
void ImageProcessing::thread(){
NSAutoreleasePool *pool=[[NSAutoreleasePool alloc]init];
ImgRef sImageRef
while( active){
**strong text**
if ([currentWnd getCurrentRGBImage:&currentImg]){
/* do something here */
}
}
[pool release];
}
-(bool)getCurrentRGBImage:(ImgRef *)pImgRef{
CGImageRef pCGImageRef;
pCGImageRef = [self getDesktopImage];
if ( !pCGImageRef ){
NSLog(#" got NULL Image ");
CGImageRelease(pCGImageRef);
pCGImageRef = NULL;
return NO;
}
// Create a bitmap rep from the image...
size_t width = CGImageGetWidth(pCGImageRef);
size_t height = CGImageGetHeight(pCGImageRef);
int bytesPerRow = CGImageGetBytesPerRow(pCGImageRef);
int bytesPerPixel = CGImageGetBitsPerPixel(pCGImageRef)/8;
CGDataProviderRef provider = CGImageGetDataProvider(pCGImageRef);
CFDataRef pData = CGDataProviderCopyData(provider);
const uint8_t* bytes = (const uint8_t *)CFDataGetBytePtr(pData);
/***** ------------- *********
Copy RAW Bytes to pImgRef
****************************/
CGDataProviderRelease(provider);
CFRelease(pData);
CGImageRelease(pCGImageRef);
pCGImageRef = NULL;
return YES;
}
and getDesktopImage function is
-(CGImageRef)getDesktopImage{
CGImageRef screenShot;
screenShot = CGWindowListCreateImage(CGRectInfinite, kCGWindowListOptionOnScreenOnly, kCGNullWindowID, kCGWindowImageDefault);
return screenShot;
}
The problem is, overall this function is consuming lot of memory, *pImgRef is having one and only one instance, there only pixel , RGB Manipulation,
Memory usage i am referring Mac Activity Monitor, is it reliable ?
the document Documentation for CGDataProviderRelease it is mentioned Application needs to maintain retaining and releasing of it, so i am Calling CGDAtaProviderRelease but getting message in the console
malloc reference count underflow for break on auto_refcount_underflow_error to debug
Should i not call CGDataProviderRelease() ? if i comment i don't get this message, so not sure, whether its bug in Quartz or in my code,
Also, this function is consuming hell lot of memory, if i check in the Activity monitor, is it safe to assuming that is really using this much memory ?
Since you do not retain, create, or copy the data provider, you should not release it.
I don't see anything that would cause excess memory consumption, but you left some of the code out.

How do you add to an AudioBufferList with an AVAssetReader?

I have been working on reading in an audio asset using AVAssetReader so that I can later play back the audio with an AUGraph with an AudioUnit callback. I have the AUGraph and AudioUnit callback working but it reads files from disk and if the file is too big it would take up too much memory and crash the app. So I am instead reading the asset directly and only a limited size. I will then manage it as a double buffer and get the AUGraph what it needs when it needs it.
(Note: I would love know if I can use Audio Queue Services and still use an AUGraph with AudioUnit callback so memory is managed for me by the iOS frameworks.)
My problem is that I do not have a good understanding of arrays, structs and pointers in C. The part where I need help is taking the individual AudioBufferList which holds onto a single AudioBuffer and add that data to another AudioBufferList which holds onto all of the data to be used later. I believe I need to use memcpy but it is not clear how to use it or even initialize an AudioBufferList for my purposes. I am using MixerHost for reference which is the sample project from Apple which reads in the file from disk.
I have uploaded my work in progress if you would like to load it up in Xcode. I've figured out most of what I need to get this done and once I have the data being collected all in one place I should be good to go.
Sample Project: MyAssetReader.zip
In the header you can see I declare the bufferList as a pointer to the struct.
#interface MyAssetReader : NSObject {
BOOL reading;
signed long sampleTotal;
Float64 totalDuration;
AudioBufferList *bufferList; // How should this be handled?
}
Then I allocate bufferList this way, largely borrowing from MixerHost...
UInt32 channelCount = [asset.tracks count];
if (channelCount > 1) {
NSLog(#"We have more than 1 channel!");
}
bufferList = (AudioBufferList *) malloc (
sizeof (AudioBufferList) + sizeof (AudioBuffer) * (channelCount - 1)
);
if (NULL == bufferList) {NSLog (#"*** malloc failure for allocating bufferList memory"); return;}
// initialize the mNumberBuffers member
bufferList->mNumberBuffers = channelCount;
// initialize the mBuffers member to 0
AudioBuffer emptyBuffer = {0};
size_t arrayIndex;
for (arrayIndex = 0; arrayIndex < channelCount; arrayIndex++) {
// set up the AudioBuffer structs in the buffer list
bufferList->mBuffers[arrayIndex] = emptyBuffer;
bufferList->mBuffers[arrayIndex].mNumberChannels = 1;
// How should mData be initialized???
bufferList->mBuffers[arrayIndex].mData = malloc(sizeof(AudioUnitSampleType));
}
Finally I loop through the reads.
int frameCount = 0;
CMSampleBufferRef nextBuffer;
while (assetReader.status == AVAssetReaderStatusReading) {
nextBuffer = [assetReaderOutput copyNextSampleBuffer];
AudioBufferList localBufferList;
CMBlockBufferRef blockBuffer;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(nextBuffer, NULL, &localBufferList, sizeof(localBufferList), NULL, NULL,
kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &blockBuffer);
// increase the number of total bites
bufferList->mBuffers[0].mDataByteSize += localBufferList.mBuffers[0].mDataByteSize;
// carefully copy the data into the buffer list
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
// get information about duration and position
//CMSampleBufferGet
CMItemCount sampleCount = CMSampleBufferGetNumSamples(nextBuffer);
Float64 duration = CMTimeGetSeconds(CMSampleBufferGetDuration(nextBuffer));
Float64 presTime = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(nextBuffer));
if (isnan(duration)) duration = 0.0;
if (isnan(presTime)) presTime = 0.0;
//NSLog(#"sampleCount: %ld", sampleCount);
//NSLog(#"duration: %f", duration);
//NSLog(#"presTime: %f", presTime);
self.sampleTotal += sampleCount;
self.totalDuration += duration;
frameCount++;
free(nextBuffer);
}
I am unsure about the what that I handle mDataByteSize and mData, especially with memcpy. Since mData is a void pointer this is an extra tricky area.
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
In this line I think it should be copying the value from the data in localBufferList to the position in the bufferList plus the number of frames to position the pointer where it should write the data. I have a couple of ideas on what I need to change to get this to work.
Since a void pointer is just 1 and not the size of the pointer for an AudioUnitSampleType I may need to multiply it also by sizeof(AudioUnitSampleType) to get the memcpy into the right position
I may not be using malloc properly to prepare mData but since I am not sure how many frames there will be I am not sure what to do to initialize it
Currently when I run this app it ends this function with an invalid pointer for bufferList.
I appreciate your help with making me better understand how to manage an AudioBufferList.
I've come up with my own answer. I decided to use an NSMutableData object which allows me to appendBytes from the CMSampleBufferRef after calling CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer to get an AudioBufferList.
[data appendBytes:localBufferList.mBuffers[0].mData length:localBufferList.mBuffers[0].mDataByteSize];
Once the read loop is done I have all of the data in my NSMutableData object. I then create and populate the AudioBufferList this way.
audioBufferList = (AudioBufferList *)malloc(sizeof(AudioBufferList));
if (NULL == audioBufferList) {
NSLog (#"*** malloc failure for allocating audioBufferList memory");
[data release];
return;
}
audioBufferList->mNumberBuffers = 1;
audioBufferList->mBuffers[0].mNumberChannels = channelCount;
audioBufferList->mBuffers[0].mDataByteSize = [data length];
audioBufferList->mBuffers[0].mData = (AudioUnitSampleType *)malloc([data length]);
if (NULL == audioBufferList->mBuffers[0].mData) {
NSLog (#"*** malloc failure for allocating mData memory");
[data release];
return;
}
memcpy(audioBufferList->mBuffers[0].mData, [data mutableBytes], [data length]);
[data release];
I'd appreciate a little code review on how I use malloc to create the struct and populate it. I am getting a EXC_BAD_ACCESS error sporadically but I cannot pinpoint where the error is just yet. Since I am using malloc on the struct I should not have to retain it anywhere. I do call "free" to release child elements within the struct and finally the struct itself everywhere that I use malloc.

Manipulating audio data in C++ for DSP purposes

I hope this question is not too vague. I'm trying to take info from an audio buffer in this Xcode project and use it to do some DSP.
framebuffer points to an array of values that I would like to pass to a function, loop through and finally plug into the original buffer. The method would act like a sound filter or effect.
Maybe to keep my question as clear as possible, could we get an example of a sub-routine that would add 0.25 to each sample in the buffer?
Here's the code so far:
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
EAGLView *remoteIOplayer = (EAGLView *)inRefCon;
for (int i = 0 ; i < ioData->mNumberBuffers; i++){
//get the buffer to be filled
AudioBuffer buffer = ioData->mBuffers[i];
short *frameBuffer = (short*)buffer.mData;
for (int j = 0; j < inNumberFrames; j++){
// get NextPacket returns a 32 bit value, one frame.
frameBuffer[j] = [[remoteIOplayer inMemoryAudioFile] getNextPacket];
}
EAGLView* thisView = [[EAGLView alloc] init];
[thisView DoStuffWithTheRecordedAudio:ioData];
[thisView release];
}
return noErr;
}
Trying to do UI or Open GL stuff inside an audio callback is a bad idea on iOS devices. You need to decouple the callback and UI execution using queues or fifos, and the like.
Trying to do Objective C messaging inside the inner loop of real-time audio may also a very bad idea in term of device performance. Sticking to plain C/C++ works far better in performance critical inner loops.
Also, adding a constant to audio data will likely just result in an inaudible DC offset.