I have an audio app i need to capture mic samples to encode into mp3 with ffmpeg
First configure the audio:
/**
* We need to specifie our format on which we want to work.
* We use Linear PCM cause its uncompressed and we work on raw data.
* for more informations check.
*
* We want 16 bits, 2 bytes (short bytes) per packet/frames at 8khz
*/
AudioStreamBasicDescription audioFormat;
audioFormat.mSampleRate = SAMPLE_RATE;
audioFormat.mFormatID = kAudioFormatLinearPCM;
audioFormat.mFormatFlags = kAudioFormatFlagIsPacked | kAudioFormatFlagIsSignedInteger;
audioFormat.mFramesPerPacket = 1;
audioFormat.mChannelsPerFrame = 1;
audioFormat.mBitsPerChannel = audioFormat.mChannelsPerFrame*sizeof(SInt16)*8;
audioFormat.mBytesPerPacket = audioFormat.mChannelsPerFrame*sizeof(SInt16);
audioFormat.mBytesPerFrame = audioFormat.mChannelsPerFrame*sizeof(SInt16);
The recording callback is:
static OSStatus recordingCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData)
{
NSLog(#"Log record: %lu", inBusNumber);
NSLog(#"Log record: %lu", inNumberFrames);
NSLog(#"Log record: %lu", (UInt32)inTimeStamp);
// the data gets rendered here
AudioBuffer buffer;
// a variable where we check the status
OSStatus status;
/**
This is the reference to the object who owns the callback.
*/
AudioProcessor *audioProcessor = (__bridge AudioProcessor*) inRefCon;
/**
on this point we define the number of channels, which is mono
for the iphone. the number of frames is usally 512 or 1024.
*/
buffer.mDataByteSize = inNumberFrames * sizeof(SInt16); // sample size
buffer.mNumberChannels = 1; // one channel
buffer.mData = malloc( inNumberFrames * sizeof(SInt16) ); // buffer size
// we put our buffer into a bufferlist array for rendering
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = buffer;
// render input and check for error
status = AudioUnitRender([audioProcessor audioUnit], ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, &bufferList);
[audioProcessor hasError:status:__FILE__:__LINE__];
// process the bufferlist in the audio processor
[audioProcessor processBuffer:&bufferList];
// clean up the buffer
free(bufferList.mBuffers[0].mData);
//NSLog(#"RECORD");
return noErr;
}
With data:
inBusNumber = 1
inNumberFrames = 1024
inTimeStamp = 80444304 // All the time same inTimeStamp, this is strange
However, the framesize that i need to encode mp3 is 1152. How can i configure it?
If i do buffering, that implies a delay, but i would like to avoid this because is a real time app. If i use this configuration, each buffer i get trash trailing samples, 1152 - 1024 = 128 bad samples. All samples are SInt16.
You can configure the number of frames per slice an AudioUnit will use with the property kAudioUnitProperty_MaximumFramesPerSlice. However, I think the best solution in your case is to buffer the incoming audio to a ring buffer and then signal your encoder that audio is available. Since you're transcoding to MP3 I'm not sure what real-time means in this case.
I'm writing an iOS app that takes input from the microphone, runs it through a high-pass filter audio unit, and plays it back through the speakers. I've been able to do this successfully by using the AUGraph API. In it, I put two nodes: a Remote I/O unit, and an effect audio unit (kAudioUnitType_Effect, kAudioUnitSubType_HighPassFilter), and connected the io unit's input element's output scope to the effect unit's input, and the effect node's output to the io unit's output element's input scope. But now I have the need to do some analysis based on the processed audio samples, so I need direct access to the buffer. This means (and please correct me if I'm wrong) I can no longer use AUGraphConnectNodeInput to make the connection between the effect node's output and the io unit's output element, and have to attach an render callback function for the io unit's output element, so that I can access the buffer whenever the speakers need new samples.
I've done so, but I get a -50 error when I call the AudioUnitRender function in the render callback. I believe I have a case of ASBDs mismatch between the two audio units, since I'm not doing anything about that in the render callback (and the AUGraph took care of it before).
Here's the code:
AudioController.h:
#interface AudioController : NSObject
{
AUGraph mGraph;
AudioUnit mEffects;
AudioUnit ioUnit;
}
#property (readonly, nonatomic) AudioUnit mEffects;
#property (readonly, nonatomic) AudioUnit ioUnit;
-(void)initializeAUGraph;
-(void)startAUGraph;
-(void)stopAUGraph;
#end
AudioController.mm:
#implementation AudioController
…
static OSStatus renderInput(void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData)
{
AudioController *THIS = (__bridge AudioController*)inRefCon;
AudioBuffer buffer;
AudioStreamBasicDescription fxOutputASBD;
UInt32 fxOutputASBDSize = sizeof(fxOutputASBD);
AudioUnitGetProperty([THIS mEffects], kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 0, &fxOutputASBD, &fxOutputASBDSize);
buffer.mDataByteSize = inNumberFrames * fxOutputASBD.mBytesPerFrame;
buffer.mNumberChannels = fxOutputASBD.mChannelsPerFrame;
buffer.mData = malloc(buffer.mDataByteSize);
AudioBufferList bufferList;
bufferList.mNumberBuffers = 1;
bufferList.mBuffers[0] = buffer;
//TODO prender ARM y solucionar problema de memoria
OSStatus result = AudioUnitRender([THIS mEffects], ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, &bufferList);
[THIS hasError:result:__FILE__:__LINE__];
memcpy(ioData, buffer.mData, buffer.mDataByteSize);
return noErr;
}
- (void)initializeAUGraph
{
OSStatus result = noErr;
// create a new AUGraph
result = NewAUGraph(&mGraph);
AUNode outputNode;
AUNode effectsNode;
AudioComponentDescription effects_desc;
effects_desc.componentType = kAudioUnitType_Effect;
effects_desc.componentSubType = kAudioUnitSubType_LowPassFilter;
effects_desc.componentFlags = 0;
effects_desc.componentFlagsMask = 0;
effects_desc.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponentDescription output_desc;
output_desc.componentType = kAudioUnitType_Output;
output_desc.componentSubType = kAudioUnitSubType_RemoteIO;
output_desc.componentFlags = 0;
output_desc.componentFlagsMask = 0;
output_desc.componentManufacturer = kAudioUnitManufacturer_Apple;
// Add nodes to the graph to hold the AudioUnits
result = AUGraphAddNode(mGraph, &output_desc, &outputNode);
[self hasError:result:__FILE__:__LINE__];
result = AUGraphAddNode(mGraph, &effects_desc, &effectsNode );
[self hasError:result:__FILE__:__LINE__];
// Connect the effect node's output to the output node's input
// This is no longer the case, as I need to access the buffer
// result = AUGraphConnectNodeInput(mGraph, effectsNode, 0, outputNode, 0);
[self hasError:result:__FILE__:__LINE__];
// Connect the output node's input scope's output to the effectsNode input
result = AUGraphConnectNodeInput(mGraph, outputNode, 1, effectsNode, 0);
[self hasError:result:__FILE__:__LINE__];
// open the graph AudioUnits
result = AUGraphOpen(mGraph);
[self hasError:result:__FILE__:__LINE__];
// Get a link to the effect AU
result = AUGraphNodeInfo(mGraph, effectsNode, NULL, &mEffects);
[self hasError:result:__FILE__:__LINE__];
// Same for io unit
result = AUGraphNodeInfo(mGraph, outputNode, NULL, &ioUnit);
[self hasError:result:__FILE__:__LINE__];
// Enable input on io unit
UInt32 flag = 1;
result = AudioUnitSetProperty(ioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, 1, &flag, sizeof(flag));
[self hasError:result:__FILE__:__LINE__];
// Setup render callback struct
AURenderCallbackStruct renderCallbackStruct;
renderCallbackStruct.inputProc = &renderInput;
renderCallbackStruct.inputProcRefCon = (__bridge void*)self;
// Set a callback for the specified node's specified input
result = AUGraphSetNodeInputCallback(mGraph, outputNode, 0, &renderCallbackStruct);
[self hasError:result:__FILE__:__LINE__];
// Get fx unit's input current stream format...
AudioStreamBasicDescription fxInputASBD;
UInt32 sizeOfASBD = sizeof(AudioStreamBasicDescription);
result = AudioUnitGetProperty(mEffects, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, 0, &fxInputASBD, &sizeOfASBD);
[self hasError:result:__FILE__:__LINE__];
// ...and set it on the io unit's input scope's output
result = AudioUnitSetProperty(ioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
1,
&fxInputASBD,
sizeof(fxInputASBD));
[self hasError:result:__FILE__:__LINE__];
// Set fx unit's output sample rate, just in case
Float64 sampleRate = 44100.0;
result = AudioUnitSetProperty(mEffects,
kAudioUnitProperty_SampleRate,
kAudioUnitScope_Output,
0,
&sampleRate,
sizeof(sampleRate));
[self hasError:result:__FILE__:__LINE__];
// Once everything is set up call initialize to validate connections
result = AUGraphInitialize(mGraph);
[self hasError:result:__FILE__:__LINE__];
}
#end
As I said before, I'm getting a -50 error on the AudioUnitRender call, and I'm finding little to no documentation at all about it.
Any help will be much appreciated.
Thanks to Tim Bolstad (http://timbolstad.com/2010/03/14/core-audio-getting-started/) for providing an excellent starting point tutorial.
There are simpler working examples of using RemoteIO for just playing buffers of audio. Perhaps start with one of those first rather than a graph.
Check to make sure that you are actually making all of the necessary connections. It appears that you are initializing most everything necessary, but if you simply want to passthrough audio you don't need the render callback function.
Now, if you want to do the filter, you may need one, but even so, make sure that you are actually connecting the components together properly.
Here's a snippet from an app I'm working on:
AUGraphConnectNodeInput(graph, outputNode, kInputBus, mixerNode, kInputBus);
AUGraphConnectNodeInput(graph, mixerNode, kOutputBus, outputNode, kOutputBus);
This connects the input from the RemoteIO unit to a Multichannel Mixer unit, then connects the output from the mixer to the RemoteIO's output to the speaker.
It looks to me like you're passing in the wrong audio unit to AudioUnitRender. I think you need to pass in ioUnit instead of mEffects. In any case, double check all of the parameters you're passing in to AudioUnitRender. When I see -50 returned it's because I botched one of them.
I need to get audio from microphone and pass to other device using tcp connection.
I not sure if is possible to play an AudioBuffer from the AudioQueue.
Currently I'm get the buffer with the code below:
void AudioInputCallback(
void *inUserData,
AudioQueueRef inAQ,
AudioQueueBufferRef inBuffer,
const AudioTimeStamp *inStartTime,
UInt32 inNumberPacketDescriptions,
const AudioStreamPacketDescription *inPacketDescs)
{
AudioRecorder *self = (AudioRecorder *)inUserData;
if(!self.recordState.recording)
{
NSLog(#"[ AudioRecorder ] AudioInputCallback not recording");
return;
}
if(inNumberPacketDescriptions > 0)
{
NSData *data = [[NSData alloc] initWithBytes:inBuffer->mAudioData length:inBuffer->mAudioDataByteSize * 2];
[self.delegate didReceivedAudioData:data];
[data release];
}
}
This block just push mAudioData to NSData, the NSData is the key data type to transfer data between devices.
Now I don't know how to run this mAudioData in other device. Other thing is how to get audio continuously, when all buffers are full, I not receive data anymore, just need to clean the buffer I guess.
Any idea are welcome, thanks!
I have been working on reading in an audio asset using AVAssetReader so that I can later play back the audio with an AUGraph with an AudioUnit callback. I have the AUGraph and AudioUnit callback working but it reads files from disk and if the file is too big it would take up too much memory and crash the app. So I am instead reading the asset directly and only a limited size. I will then manage it as a double buffer and get the AUGraph what it needs when it needs it.
(Note: I would love know if I can use Audio Queue Services and still use an AUGraph with AudioUnit callback so memory is managed for me by the iOS frameworks.)
My problem is that I do not have a good understanding of arrays, structs and pointers in C. The part where I need help is taking the individual AudioBufferList which holds onto a single AudioBuffer and add that data to another AudioBufferList which holds onto all of the data to be used later. I believe I need to use memcpy but it is not clear how to use it or even initialize an AudioBufferList for my purposes. I am using MixerHost for reference which is the sample project from Apple which reads in the file from disk.
I have uploaded my work in progress if you would like to load it up in Xcode. I've figured out most of what I need to get this done and once I have the data being collected all in one place I should be good to go.
Sample Project: MyAssetReader.zip
In the header you can see I declare the bufferList as a pointer to the struct.
#interface MyAssetReader : NSObject {
BOOL reading;
signed long sampleTotal;
Float64 totalDuration;
AudioBufferList *bufferList; // How should this be handled?
}
Then I allocate bufferList this way, largely borrowing from MixerHost...
UInt32 channelCount = [asset.tracks count];
if (channelCount > 1) {
NSLog(#"We have more than 1 channel!");
}
bufferList = (AudioBufferList *) malloc (
sizeof (AudioBufferList) + sizeof (AudioBuffer) * (channelCount - 1)
);
if (NULL == bufferList) {NSLog (#"*** malloc failure for allocating bufferList memory"); return;}
// initialize the mNumberBuffers member
bufferList->mNumberBuffers = channelCount;
// initialize the mBuffers member to 0
AudioBuffer emptyBuffer = {0};
size_t arrayIndex;
for (arrayIndex = 0; arrayIndex < channelCount; arrayIndex++) {
// set up the AudioBuffer structs in the buffer list
bufferList->mBuffers[arrayIndex] = emptyBuffer;
bufferList->mBuffers[arrayIndex].mNumberChannels = 1;
// How should mData be initialized???
bufferList->mBuffers[arrayIndex].mData = malloc(sizeof(AudioUnitSampleType));
}
Finally I loop through the reads.
int frameCount = 0;
CMSampleBufferRef nextBuffer;
while (assetReader.status == AVAssetReaderStatusReading) {
nextBuffer = [assetReaderOutput copyNextSampleBuffer];
AudioBufferList localBufferList;
CMBlockBufferRef blockBuffer;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(nextBuffer, NULL, &localBufferList, sizeof(localBufferList), NULL, NULL,
kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &blockBuffer);
// increase the number of total bites
bufferList->mBuffers[0].mDataByteSize += localBufferList.mBuffers[0].mDataByteSize;
// carefully copy the data into the buffer list
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
// get information about duration and position
//CMSampleBufferGet
CMItemCount sampleCount = CMSampleBufferGetNumSamples(nextBuffer);
Float64 duration = CMTimeGetSeconds(CMSampleBufferGetDuration(nextBuffer));
Float64 presTime = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(nextBuffer));
if (isnan(duration)) duration = 0.0;
if (isnan(presTime)) presTime = 0.0;
//NSLog(#"sampleCount: %ld", sampleCount);
//NSLog(#"duration: %f", duration);
//NSLog(#"presTime: %f", presTime);
self.sampleTotal += sampleCount;
self.totalDuration += duration;
frameCount++;
free(nextBuffer);
}
I am unsure about the what that I handle mDataByteSize and mData, especially with memcpy. Since mData is a void pointer this is an extra tricky area.
memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));
In this line I think it should be copying the value from the data in localBufferList to the position in the bufferList plus the number of frames to position the pointer where it should write the data. I have a couple of ideas on what I need to change to get this to work.
Since a void pointer is just 1 and not the size of the pointer for an AudioUnitSampleType I may need to multiply it also by sizeof(AudioUnitSampleType) to get the memcpy into the right position
I may not be using malloc properly to prepare mData but since I am not sure how many frames there will be I am not sure what to do to initialize it
Currently when I run this app it ends this function with an invalid pointer for bufferList.
I appreciate your help with making me better understand how to manage an AudioBufferList.
I've come up with my own answer. I decided to use an NSMutableData object which allows me to appendBytes from the CMSampleBufferRef after calling CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer to get an AudioBufferList.
[data appendBytes:localBufferList.mBuffers[0].mData length:localBufferList.mBuffers[0].mDataByteSize];
Once the read loop is done I have all of the data in my NSMutableData object. I then create and populate the AudioBufferList this way.
audioBufferList = (AudioBufferList *)malloc(sizeof(AudioBufferList));
if (NULL == audioBufferList) {
NSLog (#"*** malloc failure for allocating audioBufferList memory");
[data release];
return;
}
audioBufferList->mNumberBuffers = 1;
audioBufferList->mBuffers[0].mNumberChannels = channelCount;
audioBufferList->mBuffers[0].mDataByteSize = [data length];
audioBufferList->mBuffers[0].mData = (AudioUnitSampleType *)malloc([data length]);
if (NULL == audioBufferList->mBuffers[0].mData) {
NSLog (#"*** malloc failure for allocating mData memory");
[data release];
return;
}
memcpy(audioBufferList->mBuffers[0].mData, [data mutableBytes], [data length]);
[data release];
I'd appreciate a little code review on how I use malloc to create the struct and populate it. I am getting a EXC_BAD_ACCESS error sporadically but I cannot pinpoint where the error is just yet. Since I am using malloc on the struct I should not have to retain it anywhere. I do call "free" to release child elements within the struct and finally the struct itself everywhere that I use malloc.
I hope this question is not too vague. I'm trying to take info from an audio buffer in this Xcode project and use it to do some DSP.
framebuffer points to an array of values that I would like to pass to a function, loop through and finally plug into the original buffer. The method would act like a sound filter or effect.
Maybe to keep my question as clear as possible, could we get an example of a sub-routine that would add 0.25 to each sample in the buffer?
Here's the code so far:
static OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
EAGLView *remoteIOplayer = (EAGLView *)inRefCon;
for (int i = 0 ; i < ioData->mNumberBuffers; i++){
//get the buffer to be filled
AudioBuffer buffer = ioData->mBuffers[i];
short *frameBuffer = (short*)buffer.mData;
for (int j = 0; j < inNumberFrames; j++){
// get NextPacket returns a 32 bit value, one frame.
frameBuffer[j] = [[remoteIOplayer inMemoryAudioFile] getNextPacket];
}
EAGLView* thisView = [[EAGLView alloc] init];
[thisView DoStuffWithTheRecordedAudio:ioData];
[thisView release];
}
return noErr;
}
Trying to do UI or Open GL stuff inside an audio callback is a bad idea on iOS devices. You need to decouple the callback and UI execution using queues or fifos, and the like.
Trying to do Objective C messaging inside the inner loop of real-time audio may also a very bad idea in term of device performance. Sticking to plain C/C++ works far better in performance critical inner loops.
Also, adding a constant to audio data will likely just result in an inaudible DC offset.