i'm recording Video/Audio using AVAssetwriter and want to be able to write silent Sample Buffers. I'm not that experienced in CoreAudio, so i'm having trouble coming up with a working solution.
The idea is to keep recording video when an audio device is disconnected until it's reconnected. The problem is, that AVFoundation somehow push the audio to the front, so the resulting movie file is massivley out of sync.
My current implementation tries to create an empty/silent CMSampleBuffer to place in between segments where there's no audio device connected.
if (audioOutput == captureOutput && audioWriterInput.readyForMoreMediaData) {
if (needToFillAudioGap) {
CMTime temp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
CMItemCount numSamples = temp.value - lastAudioDisconnect.value;
OSStatus status;
CMBlockBufferRef bbuf = NULL;
CMSampleBufferRef sbuf = NULL;
int nchans = 2;
size_t buflen = numSamples * nchans * sizeof(float);
NSMutableData* data = [NSMutableData dataWithLength:buflen];
void* samples = [data mutableBytes];
status = CMBlockBufferCreateWithMemoryBlock(
kCFAllocatorDefault,
samples,
buflen,
kCFAllocatorNull,
NULL,
0,
buflen,
0,
&bbuf);
if (status != noErr) {
NSLog(#"CMBlockBufferCreateWithMemoryBlock error: %d", (int)status);
return;
}
CMBlockBufferRef blockBufferContiguous;
status = CMBlockBufferCreateContiguous(kCFAllocatorDefault,
bbuf,
kCFAllocatorNull,
NULL,
0,
buflen,
0,
&blockBufferContiguous);
CFRelease(bbuf);
if(status != noErr)
{
printf("CMBlockBufferCreateContiguous failed with error %d\n", (int)status);
return;
}
status = CMAudioSampleBufferCreateReadyWithPacketDescriptions(kCFAllocatorDefault, blockBufferContiguous, CMSampleBufferGetFormatDescription(sampleBuffer), numSamples, lastAudioDisconnect, NULL, &sbuf);
CFRelease(blockBufferContiguous);
if (status != noErr) {
NSLog(#"CMSampleBufferCreate error: %d", (int)status);
return;
}
BOOL r = [audioWriterInput appendSampleBuffer:sbuf];
if (!r) {
NSLog(#"appendSampleBuffer error: %d", (int)status);
return;
}
CFRelease(sbuf);
NSLog(#"Filling Audio Gap");
needToFillAudioGap = false;
} else {
if ([audioWriterInput appendSampleBuffer:sampleBuffer])
lastAudioDisconnect = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
}
}
The sampleBuffer at the top is the first samplebuffer after the audio device is reconnected. That gives me information on how long the gap I have to fill should be. LastAudioDisconnect always holds the Presentation Timestamp from the last audio samplebuffer that was written.
With Guard Malloc enabled the program crashes with: CrashIfClientProvidedBogusAudioBufferList
EDIT:
With Guard Malloc disabled i am able to reconnect the audio device multiple times while recording and when i stop to record, the gap is there without problems.
The problem then is that i only have a couple a minutes to stop the recording after reconnecting a device, because the AVAssetWriter fails randomly with the error code 11800 (AVErrorUnknown).
The error is because your created CMSampleBuffer is too long.
The created CMSampleBuffer should be the same length (contain the same number of samples) as the proceeding sample buffers that are filled with live audio. You can create multiple silent buffers, if necessary, and deliver them at the same rate as the live audio buffers.
Related
I'm trying to determine whether an audio file uses VBR or not in a Cocoa app (OS X 10.8+).
AVFoundation doesn't seem to be able to answer the question at all, and AudioToolbox is lying to me.
The code below adamantly claims that any mp3 file I throw at it is VBR; even for files where I know that isn't the case (cross-checked with MediaInfo)
OSStatus result = noErr;
UInt32 size;
AudioFileID audioFile;
AudioStreamBasicDescription audioFormat;
AudioFormatPropertyID vbrInfo;
// Open audio file.
result = AudioFileOpenURL( (__bridge CFURLRef)originalURL, kAudioFileReadPermission, 0, &audioFile );
if( result != noErr )
{
NSLog( #"Error in AudioFileOpenURL: %d", (int)result );
return;
}
// Get data format
size = sizeof( audioFormat );
result = AudioFileGetProperty( audioFile, kAudioFilePropertyDataFormat, &size, &audioFormat );
if( result != noErr )
{
NSLog( #"Error in AudioFileGetProperty: %d", (int)result );
return;
}
// Get vbr info
size = sizeof( vbrInfo );
result = AudioFormatGetProperty( kAudioFormatProperty_FormatIsVBR, sizeof(audioFormat), &audioFormat, &size, &vbrInfo);
if( result != noErr )
{
NSLog( #"Error getting vbr info: %d", (int)result );
return;
}
NSLog(#"%# is VBR: %d", originalURL.lastPathComponent, vbrInfo);
Why is it lying? What am I doing wrong?
How can I realiably determine the VBR-iness of an audio file in a Cocoa Application?
VBR and CBR sample files can be obtained here (not my page).
Your best bet is to read the MPEG frame headers from the file directly. If they all specify the same encoding parameters (version, layer, bitrate, samplerate, & channels), the file is CBR. Otherwise, the file is VBR / ABR.
There are several MP3 libraries out there that can either be used to do this directly or have the correct code for you to borrow to do it. If the file is on fast storage, performance is also well within your requirements.
If you need further assistance, I'm a major contributor to a C# MP3 decoder project and can point you to it for some examples.
In VBR mp3 first frame is different. So one doesn't need to read the whole file to find out if it is VBR or not.
Previously I read audio samples from a complete audio file using CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer. Right now I would like to do the same using ranges (ie i specify the range in time.. read a small chunk of audio as per the time, and then go back and read again). The reason why I want to use time range is b/c I want to control the size of each read (to fit in a packet with a max size).
for some reason, there is always a bump between each read. In my code you'll notice that I start the AVAssetReader and end it every time I set a time range, and that's b/c I cannot dynamically adjust the time range after the reader has started (see here for more details).
Could it be that starting and ending a reader is just too expensive to produce a continuous real time experience? Or are there other ways of doing this that I'm not aware of?
Also note that this jitter or lag happens at whatever point I set the time interval to be.. which makes me believe that starting and ending a reader the way I am is too expensive for real time audio playback.
- (void) setupReader
{
NSURL *assetURL = [NSURL URLWithString:#"ipod-library://item/item.m4a?id=1053020204400037178"];
songAsset = [AVURLAsset URLAssetWithURL:assetURL options:nil];
track = [songAsset.tracks objectAtIndex:0];
nativeTrackASBD = [self getTrackNativeSettings:track];
// set CM time parameters
assetCMTime = songAsset.duration;
CMTimeReadDurationInSeconds = CMTimeMakeWithSeconds(1, assetCMTime.timescale);
currentCMTime = CMTimeMake(0,assetCMTime.timescale);
}
-(void)readVBRPackets
{
// make sure assetCMTime is greater than currentCMTime
while (CMTimeCompare(assetCMTime,currentCMTime) == 1 )
{
NSError * error = nil;
reader = [[AVAssetReader alloc] initWithAsset:songAsset error:&error];
readerOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:track
outputSettings:nil];
[reader addOutput:readerOutput];
reader.timeRange = CMTimeRangeMake(currentCMTime, CMTimeReadDurationInSeconds);
[reader startReading];
while ((sample = [readerOutput copyNextSampleBuffer])) {
CMItemCount numSamples = CMSampleBufferGetNumSamples(sample);
if (numSamples == 0) {
continue;
}
NSLog(#"reading sample");
CMBlockBufferRef CMBuffer = CMSampleBufferGetDataBuffer( sample );
AudioBufferList audioBufferList;
OSStatus err = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
sample,
NULL,
&audioBufferList,
sizeof(audioBufferList),
NULL,
NULL,
kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
&CMBuffer
);
const AudioStreamPacketDescription * inPacketDescriptions;
size_t packetDescriptionsSizeOut;
size_t inNumberPackets;
CheckError(CMSampleBufferGetAudioStreamPacketDescriptionsPtr(sample,
&inPacketDescriptions,
&packetDescriptionsSizeOut),
"could not read sample packet descriptions");
inNumberPackets = packetDescriptionsSizeOut/sizeof(AudioStreamPacketDescription);
AudioBuffer audioBuffer = audioBufferList.mBuffers[0];
for (int i = 0; i < inNumberPackets; ++i)
{
SInt64 dataOffset = inPacketDescriptions[i].mStartOffset;
UInt32 packetSize = inPacketDescriptions[i].mDataByteSize;
size_t packetSpaceRemaining;
packetSpaceRemaining = bufferByteSize - bytesFilled;
// if the space remaining in the buffer is not
// enough for the data contained in this packet
// then just write it
if (packetSpaceRemaining < packetSize)
{
[self enqueueBuffer];
}
// copy data to the audio queue buffer
AudioQueueBufferRef fillBuf = audioQueueBuffers[fillBufferIndex];
memcpy((char*)fillBuf->mAudioData + bytesFilled,
(const char*)(audioBuffer.mData + dataOffset), packetSize);
// fill out packet description
packetDescs[packetsFilled] = inPacketDescriptions[i];
packetDescs[packetsFilled].mStartOffset = bytesFilled;
bytesFilled += packetSize;
packetsFilled += 1;
// if this is the last packet, then ship it
size_t packetsDescsRemaining = kAQMaxPacketDescs - packetsFilled;
if (packetsDescsRemaining == 0) {
[self enqueueBuffer];
}
}
CFRelease(CMBuffer);
CMSampleBufferInvalidate(sample);
CFRelease(sample);
}
[reader cancelReading];
reader = NULL;
readerOutput = NULL;
currentCMTime = CMTimeAdd(currentCMTime, CMTimeReadDurationInSeconds);
}
}
I know what happens :-D It took me near a whole day to figure it out.
In fact AVAssetReader fades the first 1024 samples (maybe a little more) in. That's why you hear the jitter effect.
I fixed it by reading 1024 samples before the position I really want to read, then skip that 1024 samples.
I hope it'll work for you also.
I'm trying to take a video created using the iVidCap plugin and add audio to it. Basically the exact same thing as in this question: Writing video + generated audio to AVAssetWriterInput, audio stuttering. I've used the code from this post as a basis to try and modify the iVidCap.mm file myself, but the app always crashes in endRecordingSession.
I'm not sure how I need to modify endRecordingSession to accomodate for the audio (the original plugin just creates a video file). Here is the function:
- (int) endRecordingSession: (VideoDisposition) action {
NSLog(#"Start endRecordingSession");
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
NSLog(#"Auto released pool");
NSString *filePath;
BOOL success = false;
[videoWriterInput markAsFinished];
NSLog(#"Mark video writer input as finished");
//[audioWriterInput markAsFinished];
// Wait for the video status to become known.
// Is this really doing anything?
int status = videoWriter.status;
while (status == AVAssetWriterStatusUnknown) {
NSLog(#"Waiting for video to complete...");
[NSThread sleepForTimeInterval:0.5f];
status = videoWriter.status;
}
NSLog(#"Video completed");
#synchronized(self) {
success = [videoWriter finishWriting];
NSLog(#"Success: %#", success);
if (!success) {
// We failed to successfully finalize the video file.
NSLog(#"finishWriting returned NO");
} else {
// The video file was successfully written to the Documents folder.
filePath = [[self getDocumentsFileURL:videoFileName] path];
if (action == Save_Video_To_Album) {
// Move the video to an accessible location on the device.
NSLog(#"Temporary video filePath=%#", filePath);
if (UIVideoAtPathIsCompatibleWithSavedPhotosAlbum(filePath)) {
NSLog(#"Video IS compatible. Adding it to photo album.");
UISaveVideoAtPathToSavedPhotosAlbum(filePath, self, #selector(copyToPhotoAlbumCompleteFromVideo: didFinishSavingWithError: contextInfo:), nil);
} else {
NSLog(#"Video IS NOT compatible. Could not be added to the photo album.");
success = NO;
}
} else if (action == Discard_Video) {
NSLog(#"Video cancelled. Removing temporary video file: %#", filePath);
[self removeFile:filePath];
}
}
[self cleanupWriter];
}
isRecording = false;
[pool drain];
return success; }
Right now it crashes on [videoWriter finishWriting]. I tried adding [audioWriterInput markAsFinished], but then it crashes on that. I would contact the original poster since it seems like they got it working, but there doesn't seem to be a way to send private messages.
Does anyone have any suggestions on how I can get this to work or why it's crashing? I've tried my best to figure this out but I'm pretty new to Obj-C. I can post the rest of the code if needed (a lot of it is in the original post referenced earlier).
The issue might actually be in the writeAudioBuffer function.
If you copied the code from that post but didnlt change it then you will certainly have some problems.
You need to do something like this:
if ( ![self waitForAudioWriterReadiness]) {
NSLog(#"WARNING: writeAudioBuffer dropped frame after wait limit reached.");
return 0;
}
OSStatus status;
CMBlockBufferRef bbuf = NULL;
CMSampleBufferRef sbuf = NULL;
size_t buflen = n * nchans * sizeof(float);
CMBlockBufferRef tmp_bbuf = NULL;
status = CMBlockBufferCreateWithMemoryBlock(
kCFAllocatorDefault,
samples,
buflen,
kCFAllocatorDefault,
NULL,
0,
buflen,
0,
&tmp_bbuf);
if (status != noErr || !tmp_bbuf) {
NSLog(#"CMBlockBufferCreateWithMemoryBlock error");
return -1;
}
// Copy the buffer so that we get a copy of the samples in memory.
// CMBlockBufferCreateWithMemoryBlock does not actually copy the data!
//
status = CMBlockBufferCreateContiguous(kCFAllocatorDefault, tmp_bbuf, kCFAllocatorDefault, NULL, 0, buflen, kCMBlockBufferAlwaysCopyDataFlag, &bbuf);
//CFRelease(tmp_bbuf); // causes abort?!
if (status != noErr) {
NSLog(#"CMBlockBufferCreateContiguous error");
//CFRelease(bbuf);
return -1;
}
CMTime timestamp = CMTimeMake(sample_position_, 44100);
status = CMAudioSampleBufferCreateWithPacketDescriptions(
kCFAllocatorDefault, bbuf, TRUE, 0, NULL, audio_fmt_desc_, 1, timestamp, NULL, &sbuf);
sample_position_ += n;
if (status != noErr) {
NSLog(#"CMSampleBufferCreate error");
return -1;
}
BOOL r = [audioWriterInput appendSampleBuffer:sbuf];
if (!r) {
NSLog(#"appendSampleBuffer error");
}
//CFRelease(bbuf); // crashes, don't know why.. Is there a leak here?
//CFRelease(sbuf);
return 0;
There are a few things to do with memory management that I am unsure on here.
Additionally be sure to use:
audioWriterInput.expectsMediaDataInRealTime = YES;
I've been pulling my hair out over this.
I've found a few things here, but nothing actually seems to work. And the documentation is really limited.
What I'm trying to figure out here is how to get the start time code of a Quicktime movie in Objective-C from the timecode track, and getting a human-readable output from that.
I've found this:
SMPTE TimeCode from Quick Time
It works perfectly in 32-bit mode. But it doesn't work in 64-bit mode because of the Quicktime API. The software I need to incorporate it into already has been and must continue to run 64-bit.
I'm losing my mind here. Anyone out there know about these APIs?
Ultimately, the goal here is to figure out the start timecode of the Quicktime because its needed to set the OFFSET in FCP-X XML files. Without it, the video files are brought in without audio (or, really, its just slipped a lot).
Use AVFoundation framework instead of QuickTime. The player initialisation is well explained in the documentation: https://developer.apple.com/library/mac/#documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/02_Playback.html#//apple_ref/doc/uid/TP40010188-CH3-SW2
Once your AVAsset is loaded in memory, you can extract the first sample frame number (timeStampFrame) by reading the content of the timecode track if present:
long timeStampFrame = 0;
for (AVAssetTrack * track in [_asset tracks]) {
if ([[track mediaType] isEqualToString:AVMediaTypeTimecode]) {
AVAssetReader *assetReader = [AVAssetReader assetReaderWithAsset:_asset error:nil];
AVAssetReaderTrackOutput *assetReaderOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:track outputSettings:nil];
if ([assetReader canAddOutput:assetReaderOutput]) {
[assetReader addOutput:assetReaderOutput];
if ([assetReader startReading] == YES) {
int count = 0;
while ( [assetReader status]==AVAssetReaderStatusReading ) {
CMSampleBufferRef sampleBuffer = [assetReaderOutput copyNextSampleBuffer];
if (sampleBuffer == NULL) {
if ([assetReader status] == AVAssetReaderStatusFailed)
break;
else
continue;
}
count++;
CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
size_t length = CMBlockBufferGetDataLength(blockBuffer);
if (length>0) {
unsigned char *buffer = malloc(length);
memset(buffer, 0, length);
CMBlockBufferCopyDataBytes(blockBuffer, 0, length, buffer);
for (int i=0; i<length; i++) {
timeStampFrame = (timeStampFrame << 8) + buffer[i];
}
free(buffer);
}
CFRelease(sampleBuffer);
}
if (count == 0) {
NSLog(#"No sample in the timecode track: %#", [assetReader error]);
}
NSLog(#"Processed %d sample", count);
}
}
if ([assetReader status] != AVAssetReaderStatusCompleted)
[assetReader cancelReading];
}
}
This is a little more tricky than the QuickTime API and there must be some improvement to the code above but it works for me.
I'm trying to encode the raw PCM data from microphone to MP3 using AudioToolbox framework and Lame. And although everything seems to run fine, there is this problem with "clicks" and "distortions" present in the encoded stream.
I'm not sure that I setup AudioQueue correctly and also that I process the encoded buffer in the right wat...
My code to setup audio recording:
AudioStreamBasicDescription streamFormat;
memset(&streamFormat, 0, sizeof(AudioStreamBasicDescription));
streamFormat.mSampleRate = 44100;
streamFormat.mFormatID = kAudioFormatLinearPCM;
streamFormat.mFormatFlags = kLinearPCMFormatFlagIsSignedInteger|kLinearPCMFormatFlagIsPacked;
streamFormat.mBitsPerChannel = 16;
streamFormat.mChannelsPerFrame = 1;
streamFormat.mBytesPerPacket = 2;
streamFormat.mBytesPerFrame = 2;
streamFormat.mFramesPerPacket = 1;
streamFormat.mReserved = 0;
AudioQueueNewInput(&streamFormat, InputBufferCallback, (__bridge void*)(self), nil, nil, 0, &mQueue);
UInt32 bufferByteSize = 44100;
memset(&mEncodedBuffer, 0, sizeof(mEncodedBuffer)); //mEncoded buffer is
//unsigned char [72000]
AudioQueueBufferRef buffer;
for (int i=0; i<3; i++) {
AudioQueueAllocateBuffer(mQueue, bufferByteSize, &buffer);
AudioQueueEnqueueBuffer(mQueue, buffer, 0, NULL);
}
AudioQueueStart(mQueue, nil);
Then the AudioQueue callback function calls to lame_encode_buffer and then writes the encoded buffer to file:
void InputBufferCallback (void *inUserData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer, const AudioTimeStamp *inStartTime, UInt32 inNumPackets, const AudioStreamPacketDescription* inPacketDesc) {
memset(&mEncodedBuffer, 0, sizeof(mEncodedBuffer));
int encodedBytes = lame_encode_buffer(glf, (short*)inBuffer->mAudioData, NULL, inBuffer->mAudioDataByteSize, mEncodedBuffer, 72000);
//What I don't understand is that if I write the full 'encodedBytes' data, then there are A LOT of distortions and original sound is seriously broken
NSData* data = [NSData dataWithBytes:mEncodedBuffer length:encodedBytes/2];
[mOutputFile writeData:data];
}
And when I afterward try to play the file which contains Lame encoded data with AVAudioPlayer I clearly hear original sound but with some clicks and distortions around.
Can anybody advise what's wrong here?
Your code does not appear to be paying attention to inNumPackets, which is the amount of actual audio data given the callback.
Also, doing a long operation, such as running an encoder, inside an audio callback might not be fast enough and thus may violate response requirements. Any long function calls should be done outside the callback.