AVAssetWriterInput Settings Quality vs. Filesize - objective-c

Here's a rather specific question that's left me stumped. I'm writing a video recording software that captures video data from a webcam to an MP4. My software is going to replace a script that's already in place that triggers QuickTime Player to do the same but output to a MOV.
I'm using AVFoundation and have the capturing and saving in place but, after repeated tweaks and tests, I've found that the script that's already in place consistently creates MOV files with a higher video quality and lower file sizes than my software.
Here is a link to two samples, a MOV created by the on-site script and an MP4 created by my software: https://www.dropbox.com/sh/1qnn5afmrquwfcr/AADQvDMWkbYJwVNlio9_vbeNa?dl=0
The MOV was created by a colleague and is the quality and file size I'm trying to match with my software. The MP4 was recording in my office, obviously a different lighting situation, but with the same camera as is used on-site.
Comparing the two videos, I can see that they have the same dimensions, duration, and video codec but differ in both file size and quality.
Here is the code where I set up my AVAssetWriter and AVAssetWriter input:
NSDictionary *settings = nil;
settings = [NSDictionary dictionaryWithObjectsAndKeys:
// Specify H264 (MPEG4) as the codec
AVVideoCodecH264, AVVideoCodecKey,
// Specify the video width and height
[NSNumber numberWithInt:self.recordSize.width], AVVideoWidthKey,
[NSNumber numberWithInt:self.recordSize.height], AVVideoHeightKey,
// Specify the video compression
[NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInteger:2500000], AVVideoAverageBitRateKey,
[NSNumber numberWithInt:1], AVVideoMaxKeyFrameIntervalKey,
//AVVideoProfileLevelH264Baseline30, AVVideoProfileLevelKey, Not available on 10.7
nil], AVVideoCompressionPropertiesKey,
// Specify the HD output color
[NSDictionary dictionaryWithObjectsAndKeys:
AVVideoColorPrimaries_ITU_R_709_2, AVVideoColorPrimariesKey,
AVVideoTransferFunction_ITU_R_709_2, AVVideoTransferFunctionKey,
AVVideoYCbCrMatrix_ITU_R_709_2, AVVideoYCbCrMatrixKey, nil], AVVideoColorPropertiesKey,
nil];
self.assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:settings];// sourceFormatHint:self.formatHint];
/*self.pixelBufferAdaptor = [[AVAssetWriterInputPixelBufferAdaptor alloc]
initWithAssetWriterInput:self.assetWriterInput
sourcePixelBufferAttributes:[NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange],//kCVPixelFormatType_32BGRA],
kCVPixelBufferPixelFormatTypeKey,
nil]];*/
NSError *error;
self.videoFile = [NSURL fileURLWithPath:file];
//[self.movieFileOutput startRecordingToOutputFileURL:self.videoFile recordingDelegate:self];
self.assetWriter = [[AVAssetWriter alloc]
initWithURL:self.videoFile
fileType:AVFileTypeMPEG4//AVFileTypeQuickTimeMovie//
error:&error];
if (self.assetWriter){
[self.assetWriter addInput:self.assetWriterInput];
self.assetWriterInput.expectsMediaDataInRealTime = YES;
And the code where I set up my AVCaptureVideoDataOutput and add it to my capture session:
AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init];
output.alwaysDiscardsLateVideoFrames = NO;
output.videoSettings = nil;
self.videoQueue = dispatch_queue_create("ca.blackboxsoftware.avcapturequeue", NULL);
[output setSampleBufferDelegate:self queue:self.videoQueue];
[self.session addOutput:output];
This quality issue is the big stumbling block of my software and I desperately need your help with it. I'll be happy to post any other code you might need to see and test out changes you feel would make the difference.
Thank you for your time.

Related

Creating a CVPixelBufferRef from a IDeckLinkVideoInputFrame

I'm using the BlackMagic DeckLink SDK to try capture frames from a BM device.
I'm trying to grab the pixel data from a IDeckLinkVideoInputFrame in the DeckLinkController::VideoInputFrameArrived callback and convert it to a CVPixelBufferRef to be able to write it to disk with AVFoundation's AVAssetWriterInputPixelBufferAdaptor and AVAssetWriter. The code I'm using seems to be working, apart from the fact that all frames written to disk are green. (BlackMagic's example code that generates a preview on screen does show an image, so the device and device settings should be OK).
The AVAssetWriter is set up as follows:
writer = [[AVAssetWriter assetWriterWithURL:destinationUrl
fileType:AVFileTypeAppleM4V
error:&error] retain];
if(error)
NSLog(#"ERROR: %#", [error localizedDescription]);
NSMutableDictionary * outputSettings = [[NSMutableDictionary alloc] init];
[outputSettings setObject: AVVideoCodecH264
forKey: AVVideoCodecKey];
[outputSettings setObject: [NSNumber numberWithInt:1920]
forKey: AVVideoWidthKey];
[outputSettings setObject: [NSNumber numberWithInt:1080]
forKey: AVVideoHeightKey];
NSMutableDictionary * compressionProperties = [[NSMutableDictionary alloc] init];
[compressionProperties setObject: [NSNumber numberWithInt: 1000000]
forKey: AVVideoAverageBitRateKey];
[compressionProperties setObject: [NSNumber numberWithInt: 16]
forKey: AVVideoMaxKeyFrameIntervalKey];
[compressionProperties setObject: AVVideoProfileLevelH264Main31
forKey: AVVideoProfileLevelKey];
[outputSettings setObject: compressionProperties
forKey: AVVideoCompressionPropertiesKey];
writerVideoInput = [[AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:outputSettings] retain];
NSMutableDictionary * pixBufSettings = [[NSMutableDictionary alloc] init];
[pixBufSettings setObject: [NSNumber numberWithInt: kCVPixelFormatType_422YpCbCr8_yuvs]
forKey: (NSString *) kCVPixelBufferPixelFormatTypeKey];
[pixBufSettings setObject: [NSNumber numberWithInt: 1920]
forKey: (NSString *) kCVPixelBufferWidthKey];
[pixBufSettings setObject: [NSNumber numberWithInt: 1080]
forKey: (NSString *) kCVPixelBufferHeightKey];
writerVideoInput.expectsMediaDataInRealTime = YES;
writer.shouldOptimizeForNetworkUse = NO;
adaptor = [[AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerVideoInput
sourcePixelBufferAttributes:pixBufSettings] retain];
[writer addInput:writerVideoInput];
For reference, these output settings and compression options should be correct, but I have tried several different alternatives.
When a frame comes in from the device, I convert it to a CVPixelBufferRef as follows:
void *videoData;
int64_t frameTime;
int64_t frameDuration;
videoFrame->GetBytes(&videoData);
videoFrame->GetStreamTime(&frameTime, &frameDuration, 3000);
CMTime presentationTime = CMTimeMake(frameDuration, 3000);
CVPixelBufferRef buffer = NULL;
CVPixelBufferPoolCreatePixelBuffer(NULL, adaptor.pixelBufferPool, &buffer);
CVPixelBufferLockBaseAddress(buffer, 0);
void *rasterData = CVPixelBufferGetBaseAddress(buffer);
memcpy(rasterData, videoData, (videoFrame->GetRowBytes()*videoFrame->GetHeight()));
CVPixelBufferUnlockBaseAddress(buffer, 0);
if (buffer)
{
if(![adaptor appendPixelBuffer:buffer withPresentationTime:presentationTime]) {
NSLog(#"ERROR appending pixelbuffer: %#", writer.error);
[writerVideoInput markAsFinished];
if(![writer finishWriting])
NSLog(#"ERROR finishing writing: %#", [writer.error localizedDescription]);
}
else {
NSLog(#"SUCCESS");
if(buffer)
CVPixelBufferRelease(buffer);
}
}
This code is appending frames to the AVAssetWriterInputPixelBufferAdaptor, but all the frames are green.
Can anybody see what I'm doing wrong here, or does anybody have any experience using AVFoundation capturing and compressing frames using the BlackMagic Decklink SDK?
When you see 'green' and are working in the YUV color space, you are seeing values of 0 in the buffer. Since AVWriter is writing frames, the odds are that 'buffer' contains values of 0. I see a couple of ways that could happen.
1) The buffer your are appending is most likely initialized with 0, so it is possible your copy is failing. In your code that could happen if (videoFrame->GetRowBytes()*videoFrame->GetHeight()) somehow evaluates to 0. It seems impossible, but I'd check that.
2) The CVPixelBufferGetBaseAddress is either returning the wrong pointer, the PixelBuffer itself is the wrong format or possible invalid (yet didn't crash because of safeguards in the API).
3) 'videoData' is, for whatever reason, itself full of 0. DeckLinkCaptureDelegate returns frames with nothing in them when it doesn't like the input format (usually this is because the BMDDisplayMode passed to EnableVideoInput doesn't match your video source.
int flags=videoFrame->GetFlags();
if (flags & bmdFrameHasNoInputSource)
{
//our input format doesn't match the source
}
Other than changing your source mode and trying again, a quick check would be to change the memcpy line to the following:
memset(rasterData, 0x3f, 1920*1080*2);
If you still see green frames then take a hard look at #2. If you see different colored frames, then your problem is #1 or #3 and most likely the resolution of your video input doesn't match the BMDDisplayMode that you chose.
One other thing to note. I think the line where you create the presentation time is wrong. It probably should be (note changing frameDuration to frameTime:
CMTime presentationTime = CMTimeMake(frameTime, 3000);

Video capture with 1:1 aspect ratio in iOS

I want to capture videos with an iOS camera with 1:1 aspect ratio.
I tried with UIImagePickerController, but it doesn't provide changing aspect ratio.
Could anyone give me ideas?
Additionally, the iPhone app "Viddy" provides 1:1 aspect ratio video capturing:
AVCaptureVideoPreviewLayer *_preview = [AVVideoCaptureVideoPreviewLayer layerWithSession:_session];
_preview.frame = CGRectMake(0,0,320,320);
_preview.videoGravity = AVLayerVideoGravityResizeAspectFill;
NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:320], AVVideoWidthKey,
[NSNumber numberWithInt:320], AVVideoHeightKey,
AVVideoScalingModeResizeAspectFill,AVVideoScalingModeKey,
nil];
self.videoInput = [AVAssetWriterInput assetWriterInputWithMediaType: AVMediaTypeVideo
outputSettings: videoSettings];
self.videoInput.transform = CGAffineTransformMakeRotation(M_PI);
if([_writer canAddInput:_videoInput]) // AVAssetWriter *_writer
[_writer addInput:_videoInput];
Note:
_preview's videoGravity and the videoSettings AVVideoScalingModeKey should be same to get the output as 320 x 320.
GPUImageMovie* movieFile = [[GPUImageMovie alloc] initWithAsset:asset];
GPUImageCropFilter *cropFilter = [[GPUImageCropFilter alloc] initWithCropRegion:CGRectMake(0.0, 0.1, 1.0, 0.8)];
[movieFile addTarget:cropFilter];
GPUImageMovieWriter* movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:movieURL size:CGSizeMake(320.0, 320.0)];
[cropFilter addTarget:movieWriter];
[movieWriter startRecording];
[movieFile startProcessing];
[movieWriter finishRecordingWithCompletionHandler:^{
NSLog(#"Completed Successfully");
[cropFilter removeTarget:movieWriter];
[movieFile removeTarget:cropFilter];
}];
where
asset is the input movie file.
cropRegion is the area to crop.
movieUrl is the target url to save the cropped movie.
I don't think it's possible to do so without help of some app, or even if it's possible with an app, you can capture video then crop it to 1:1

AVAudioPlayer not playing recording made with AVAudioRecorder

This may be a really simple problem but I've made a recording using AVAudioRecorder, then I stopped the recorder. After stopping the AVAudioRecorder, I press another button to play the recording but it doesn't play. The file exists, I can play it on my computer, even the code knows it exists, there is no error, but refuses to play. What can be the issue?
NSError *error = nil;
if ([[NSFileManager defaultManager] fileExistsAtPath:[self.recorder.url path]]) {
self.recordingPlayer = [[AVAudioPlayer alloc] initWithContentsOfURL:self.recorder.url error:&error];
self.recordingPlayer.delegate = self;
if (error) {
NSLog(#"error: %#", [error localizedDescription]);
} else {
[self.recordingPlayer play];
}
} else {
NSLog(#"Recording file doesn't exist");
}
EDIT: just tried it on my device and it works fine, plays the recording. It just doesn't work on iOS simulator
The problem was with my record settings, I had the number of AVNumberOfChannelsKey set to 2, for some reason iOS simulator didn't like this, but iPhone was fine with it. Either way, my recording shouldn't have 2 channels in the first place, so good thing I spotted this.
NSDictionary *recordSettings = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:AVAudioQualityMax], AVEncoderAudioQualityKey,
[NSNumber numberWithInt:16], AVEncoderBitRateKey,
[NSNumber numberWithInt:1], AVNumberOfChannelsKey,
[NSNumber numberWithFloat:44100.0], AVSampleRateKey,
nil];
Try this link's code. In this you can record an audio and retrieve from document folder.
Record audio file and save locally on iPhone

AQRecorder AAC play in AVPlayer

I have some problem with recorded aac audio file on iOS.
I need record audio and play it with AVPlayer. Recording work fine, file created. i can play it on Mac. But when i try play it(file) in AVPlayer - no sound.
property for record audio
mRecordFormat.mSampleRate = 8000;
mRecordFormat.mFormatID = kAudioFormatMPEG4AAC;
mRecordFormat.mFormatFlags = kAudioFormatMPEG4AAC_LD;
mRecordFormat.mFramesPerPacket = 0;
mRecordFormat.mChannelsPerFrame = 1;
mRecordFormat.mBitsPerChannel = 0;
mRecordFormat.mBytesPerPacket = 0;
i try different setting but no results. Help please
P.S. Other files(mp3 etc) playing normaly
Ok, i find solution couple years ago.
I change record logic to
NSDictionary *settings = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:kAudioFormatMPEG4AAC], AVFormatIDKey,
[NSNumber numberWithFloat:16000], AVSampleRateKey,
[NSNumber numberWithInt:2], AVNumberOfChannelsKey,
[NSNumber numberWithInt:AVAudioQualityMedium], AVSampleRateConverterAudioQualityKey,
[NSNumber numberWithInt:64000], AVEncoderBitRateKey,
[NSNumber numberWithInt:8], AVEncoderBitDepthHintKey,
nil];
error = nil;
_recorder = [[AVAudioRecorder alloc] initWithURL:url settings:settings error:&error];
and this recorded file play on iOS, Android and MacOS/Win. So it's all.

4 second lag in AVAudioRecorder record

I'm using AVAudioRecorder to record audio but I'm experiencing a 4 second delay between button press and beginning to record.
Here's my setup code:
NSDictionary *recordSettings = [NSDictionary
dictionaryWithObjectsAndKeys:
[NSNumber numberWithFloat: 16000.0],AVSampleRateKey,
[NSNumber numberWithInt: kAudioFormatAppleIMA4],AVFormatIDKey,
[NSNumber numberWithInt: 1], AVNumberOfChannelsKey,
[NSNumber numberWithInt: AVAudioQualityMax],AVEncoderAudioQualityKey,nil];
NSError *error = nil;
audioRecorder = [[AVAudioRecorder alloc]
initWithURL:soundFileURL
settings:recordSettings
error:&error];
if (error)
{
NSLog(#"error: %#", [error localizedDescription]);
} else {
NSLog(#"prepare to record");
[audioRecorder prepareToRecord];
}
-(void)record {
NSLog(#"Record");
//[audioRecorder prepareToRecord];
if (!audioRecorder.recording)
{
NSLog(#"Record 2");
[audioRecorder record];
NSLog(#"Record 3");
}
}
Record is the function called on button press. I know prepareToRecord is called implicitly via 'record' but I wanted to see if it would affect the delay at all. It does not.
Here's the console log:
2011-10-18 21:48:06.508 [2949:707] Record
2011-10-18 21:48:06.509 [2949:707] Record 2
2011-10-18 21:48:10.047 [2949:707] Record 3
There's about 3.5 seconds before it starts recording.
Are these settings too much for the iPhone? (iPhone 4). Am I initializing it wrong?
I'm seeing the same issue on the iOS5/iPhone4 combo. iOS4 and iPhone4S are fine. There seems to be an issue when using the new OS on older hardware. I've tried many combinations of settings as well.
Try setting the AVAudioSession category when you initialize the Audio Recorder:
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord error:nil];
I had the same issue on iPod touch 4th generation (iOS4, iOS5 and JailBreak - I used more devices) and on different devices it runs different, but I think #Charles Chase is right, it does depend of device's iOS you are testing on. Your recordSettings dictionary is ok, code also looks right except one thing, you need to add this:
audioRecorder.delegate = self;
right after you alloc and init your recorder:
audioRecorder = [[AVAudioRecorder alloc]
initWithURL:soundFileURL
settings:recordSettings
error:&error];
audioRecorder.delegate = self;
I had the same problem. Also, since I was switching between Playback and Record mode, this was causing long delays constantly. But when I switched to Play and Record it refused to use the loudspeaker on iPhone, it would only use the headset speaker.
Solution:
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord error:nil];
UInt32 audioRouteOverride = kAudioSessionOverrideAudioRoute_Speaker;
AudioSessionSetProperty(kAudioSessionProperty_OverrideAudioRoute,
sizeof(audioRouteOverride), &audioRouteOverride);