Merge multi channel audio buffers into one CMSampleBuffer - objective-c

Question

I am using FFmpeg to access an RTSP stream in my macOS app.
REACHED GOALS: I have created a tone generator which creates single channel audio and returns a CMSampleBuffer. The tone generator is used to test my audio pipeline when the video's fps and audio sample rates are changed.
GOAL: The goal is to merge multi-channel audio buffers into a single CMSampleBuffer.
Audio data lifecyle:
AVCodecContext* audioContext = self.rtspStreamProvider.audioCodecContext;
if (!audioContext) { return; }
// Getting audio settings from FFmpegs audio context (AVCodecContext).
int samplesPerChannel = audioContext->frame_size;
int frameNumber = audioContext->frame_number;
int sampleRate = audioContext->sample_rate;
int fps = [self.rtspStreamProvider fps];
int calculatedSampleRate = sampleRate / fps;
// NSLog(#"\nSamples per channel = %i, frames = %i.\nSample rate = %i, fps = %i.\ncalculatedSampleRate = %i.", samplesPerChannel, frameNumber, sampleRate, fps, calculatedSampleRate);
// Decoding the audio data from a encoded AVPacket into a AVFrame.
AVFrame* audioFrame = [self.rtspStreamProvider readDecodedAudioFrame];
if (!audioFrame) { return; }
// Extracting my audio buffers from FFmpegs AVFrame.
uint8_t* leftChannelAudioBufRef = audioFrame->data[0];
uint8_t* rightChannelAudioBufRef = audioFrame->data[1];
// Creating the CMSampleBuffer with audio data.
CMSampleBufferRef leftSampleBuffer = [CMSampleBufferFactory createAudioSampleBufferUsingData:leftChannelAudioBufRef channelCount:1 framesCount:samplesPerChannel sampleRate:sampleRate];
// CMSampleBufferRef rightSampleBuffer = [CMSampleBufferFactory createAudioSampleBufferUsingData:packet->data[1] channelCount:1 framesCount:samplesPerChannel sampleRate:sampleRate];
if (!leftSampleBuffer) { return; }
if (!self.audioQueue) { return; }
if (!self.audioDelegates) { return; }
// All audio consumers will receive audio samples via delegation.
dispatch_sync(self.audioQueue, ^{
NSHashTable *audioDelegates = self.audioDelegates;
for (id<AudioDataProviderDelegate> audioDelegate in audioDelegates)
{
[audioDelegate provider:self didOutputAudioSampleBuffer:leftSampleBuffer];
// [audioDelegate provider:self didOutputAudioSampleBuffer:rightSampleBuffer];
}
});
CMSampleBuffer containing audio data creation:
import Foundation
import CoreMedia
#objc class CMSampleBufferFactory: NSObject
{
#objc static func createAudioSampleBufferUsing(data: UnsafeMutablePointer<UInt8> ,
channelCount: UInt32,
framesCount: CMItemCount,
sampleRate: Double) -> CMSampleBuffer? {
/* Prepare for sample Buffer creation */
var sampleBuffer: CMSampleBuffer! = nil
var osStatus: OSStatus = -1
var audioFormatDescription: CMFormatDescription! = nil
var absd: AudioStreamBasicDescription! = nil
let sampleDuration = CMTimeMake(value: 1, timescale: Int32(sampleRate))
let presentationTimeStamp = CMTimeMake(value: 0, timescale: Int32(sampleRate))
// NOTE: Change bytesPerFrame if you change the block buffer value types. Currently we are using double.
let bytesPerFrame: UInt32 = UInt32(MemoryLayout<Float32>.size) * channelCount
let memoryBlockByteLength = framesCount * Int(bytesPerFrame)
// var acl = AudioChannelLayout()
// acl.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo
/* Sample Buffer Block buffer creation */
var blockBuffer: CMBlockBuffer?
osStatus = CMBlockBufferCreateWithMemoryBlock(
allocator: kCFAllocatorDefault,
memoryBlock: nil,
blockLength: memoryBlockByteLength,
blockAllocator: nil,
customBlockSource: nil,
offsetToData: 0,
dataLength: memoryBlockByteLength,
flags: 0,
blockBufferOut: &blockBuffer
)
assert(osStatus == kCMBlockBufferNoErr)
guard let eBlock = blockBuffer else { return nil }
osStatus = CMBlockBufferFillDataBytes(with: 0, blockBuffer: eBlock, offsetIntoDestination: 0, dataLength: memoryBlockByteLength)
assert(osStatus == kCMBlockBufferNoErr)
TVBlockBufferHelper.fillAudioBlockBuffer(blockBuffer,
audioData: data,
frames: Int32(framesCount))
/* Audio description creations */
absd = AudioStreamBasicDescription(
mSampleRate: sampleRate,
mFormatID: kAudioFormatLinearPCM,
mFormatFlags: kLinearPCMFormatFlagIsPacked | kLinearPCMFormatFlagIsFloat,
mBytesPerPacket: bytesPerFrame,
mFramesPerPacket: 1,
mBytesPerFrame: bytesPerFrame,
mChannelsPerFrame: channelCount,
mBitsPerChannel: 32,
mReserved: 0
)
guard absd != nil else {
print("\nCreating AudioStreamBasicDescription Failed.")
return nil
}
osStatus = CMAudioFormatDescriptionCreate(allocator: kCFAllocatorDefault,
asbd: &absd,
layoutSize: 0,
layout: nil,
// layoutSize: MemoryLayout<AudioChannelLayout>.size,
// layout: &acl,
magicCookieSize: 0,
magicCookie: nil,
extensions: nil,
formatDescriptionOut: &audioFormatDescription)
guard osStatus == noErr else {
print("\nCreating CMFormatDescription Failed.")
return nil
}
/* Create sample Buffer */
var timmingInfo = CMSampleTimingInfo(duration: sampleDuration, presentationTimeStamp: presentationTimeStamp, decodeTimeStamp: .invalid)
osStatus = CMSampleBufferCreate(allocator: kCFAllocatorDefault,
dataBuffer: eBlock,
dataReady: true,
makeDataReadyCallback: nil,
refcon: nil,
formatDescription: audioFormatDescription,
sampleCount: framesCount,
sampleTimingEntryCount: 1,
sampleTimingArray: &timmingInfo,
sampleSizeEntryCount: 0, // Must be 0, 1, or numSamples.
sampleSizeArray: nil, // Pointer ot Int. Don't know the size. Don't know if its bytes or bits?
sampleBufferOut: &sampleBuffer)
return sampleBuffer
}
}
CMSampleBuffer gets filled with raw audio data from FFmpeg's data:
#import Foundation;
#import CoreMedia;
#interface BlockBufferHelper : NSObject
+(void)fillAudioBlockBuffer:(CMBlockBufferRef)blockBuffer
audioData:(uint8_t *)data
frames:(int)framesCount;
#end
#import "TVBlockBufferHelper.h"
#implementation BlockBufferHelper
+(void)fillAudioBlockBuffer:(CMBlockBufferRef)blockBuffer
audioData:(uint8_t *)data
frames:(int)framesCount
{
// Possibly dev error.
if (framesCount == 0) {
NSAssert(false, #"\nfillAudioBlockBuffer/audioData/frames will not be able to fill an blockBuffer which has no frames.");
return;
}
char *rawBuffer = NULL;
size_t size = 0;
OSStatus status = CMBlockBufferGetDataPointer(blockBuffer, 0, &size, NULL, &rawBuffer);
if(status != noErr)
{
return;
}
memcpy(rawBuffer, data, framesCount);
}
#end
The LEARNING Core Audio book from Chris Adamson/Kevin Avila points me toward a multi channel mixer.
The multi channel mixer should have 2-n inputs and 1 output. I assume the output could be a buffer or something that could be put into a CMSampleBuffer for further consumption.
This direction should lead me to AudioUnits, AUGraph and the AudioToolbox. I don't understand all of these classes and how they work together. I have found some code snippets on SO which could help me but most of them use AudioToolBox classes and don't use CMSampleBuffers as much as I need.
Is there another way to merge audio buffers into a new one?
Is creating a multi channel mixer using AudioToolBox the right direction?