Creating a CVPixelBufferRef from a IDeckLinkVideoInputFrame - objective-c

Question

I'm using the BlackMagic DeckLink SDK to try capture frames from a BM device.
I'm trying to grab the pixel data from a IDeckLinkVideoInputFrame in the DeckLinkController::VideoInputFrameArrived callback and convert it to a CVPixelBufferRef to be able to write it to disk with AVFoundation's AVAssetWriterInputPixelBufferAdaptor and AVAssetWriter. The code I'm using seems to be working, apart from the fact that all frames written to disk are green. (BlackMagic's example code that generates a preview on screen does show an image, so the device and device settings should be OK).
The AVAssetWriter is set up as follows:
writer = [[AVAssetWriter assetWriterWithURL:destinationUrl
fileType:AVFileTypeAppleM4V
error:&error] retain];
if(error)
NSLog(#"ERROR: %#", [error localizedDescription]);
NSMutableDictionary * outputSettings = [[NSMutableDictionary alloc] init];
[outputSettings setObject: AVVideoCodecH264
forKey: AVVideoCodecKey];
[outputSettings setObject: [NSNumber numberWithInt:1920]
forKey: AVVideoWidthKey];
[outputSettings setObject: [NSNumber numberWithInt:1080]
forKey: AVVideoHeightKey];
NSMutableDictionary * compressionProperties = [[NSMutableDictionary alloc] init];
[compressionProperties setObject: [NSNumber numberWithInt: 1000000]
forKey: AVVideoAverageBitRateKey];
[compressionProperties setObject: [NSNumber numberWithInt: 16]
forKey: AVVideoMaxKeyFrameIntervalKey];
[compressionProperties setObject: AVVideoProfileLevelH264Main31
forKey: AVVideoProfileLevelKey];
[outputSettings setObject: compressionProperties
forKey: AVVideoCompressionPropertiesKey];
writerVideoInput = [[AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:outputSettings] retain];
NSMutableDictionary * pixBufSettings = [[NSMutableDictionary alloc] init];
[pixBufSettings setObject: [NSNumber numberWithInt: kCVPixelFormatType_422YpCbCr8_yuvs]
forKey: (NSString *) kCVPixelBufferPixelFormatTypeKey];
[pixBufSettings setObject: [NSNumber numberWithInt: 1920]
forKey: (NSString *) kCVPixelBufferWidthKey];
[pixBufSettings setObject: [NSNumber numberWithInt: 1080]
forKey: (NSString *) kCVPixelBufferHeightKey];
writerVideoInput.expectsMediaDataInRealTime = YES;
writer.shouldOptimizeForNetworkUse = NO;
adaptor = [[AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerVideoInput
sourcePixelBufferAttributes:pixBufSettings] retain];
[writer addInput:writerVideoInput];
For reference, these output settings and compression options should be correct, but I have tried several different alternatives.
When a frame comes in from the device, I convert it to a CVPixelBufferRef as follows:
void *videoData;
int64_t frameTime;
int64_t frameDuration;
videoFrame->GetBytes(&videoData);
videoFrame->GetStreamTime(&frameTime, &frameDuration, 3000);
CMTime presentationTime = CMTimeMake(frameDuration, 3000);
CVPixelBufferRef buffer = NULL;
CVPixelBufferPoolCreatePixelBuffer(NULL, adaptor.pixelBufferPool, &buffer);
CVPixelBufferLockBaseAddress(buffer, 0);
void *rasterData = CVPixelBufferGetBaseAddress(buffer);
memcpy(rasterData, videoData, (videoFrame->GetRowBytes()*videoFrame->GetHeight()));
CVPixelBufferUnlockBaseAddress(buffer, 0);
if (buffer)
{
if(![adaptor appendPixelBuffer:buffer withPresentationTime:presentationTime]) {
NSLog(#"ERROR appending pixelbuffer: %#", writer.error);
[writerVideoInput markAsFinished];
if(![writer finishWriting])
NSLog(#"ERROR finishing writing: %#", [writer.error localizedDescription]);
}
else {
NSLog(#"SUCCESS");
if(buffer)
CVPixelBufferRelease(buffer);
}
}
This code is appending frames to the AVAssetWriterInputPixelBufferAdaptor, but all the frames are green.
Can anybody see what I'm doing wrong here, or does anybody have any experience using AVFoundation capturing and compressing frames using the BlackMagic Decklink SDK?

Answer 1

When you see 'green' and are working in the YUV color space, you are seeing values of 0 in the buffer. Since AVWriter is writing frames, the odds are that 'buffer' contains values of 0. I see a couple of ways that could happen.
1) The buffer your are appending is most likely initialized with 0, so it is possible your copy is failing. In your code that could happen if (videoFrame->GetRowBytes()*videoFrame->GetHeight()) somehow evaluates to 0. It seems impossible, but I'd check that.
2) The CVPixelBufferGetBaseAddress is either returning the wrong pointer, the PixelBuffer itself is the wrong format or possible invalid (yet didn't crash because of safeguards in the API).
3) 'videoData' is, for whatever reason, itself full of 0. DeckLinkCaptureDelegate returns frames with nothing in them when it doesn't like the input format (usually this is because the BMDDisplayMode passed to EnableVideoInput doesn't match your video source.
int flags=videoFrame->GetFlags();
if (flags & bmdFrameHasNoInputSource)
{
//our input format doesn't match the source
}
Other than changing your source mode and trying again, a quick check would be to change the memcpy line to the following:
memset(rasterData, 0x3f, 1920*1080*2);
If you still see green frames then take a hard look at #2. If you see different colored frames, then your problem is #1 or #3 and most likely the resolution of your video input doesn't match the BMDDisplayMode that you chose.
One other thing to note. I think the line where you create the presentation time is wrong. It probably should be (note changing frameDuration to frameTime:
CMTime presentationTime = CMTimeMake(frameTime, 3000);