Rendering QCRenderer for displaying onto DeckLink Cards - objective-c

Question

I'm writing an application in xcode on a MacMini (Late 2012).
It's an app where I load some QuartzComposer files (with QCRenderer class). Then render those files into videomemory and read them back using glReadPixels to get all the pixel data. This pixel data is then pushed to a decklink frame (I'm using BlackMagic Decklink SDK) for playback onto a DeckLink Quad. Everything is working great. It's even possible to render 3 outputs without dropping frames on HD (1080i50). But after a while (like 5 mins) it is dropping frames even when I'm only rendering 1 output.
So I think there are 2 possible reasons. First. When there is a completed frame (frame dit played out) callback I'm receiving the bmdOutputFrameDisplayedLate from the SDK which means the frame was not played at the time it was scheduled for. So when this happens I'm pushing the next frame by 1 into the future.
Second. I've set a frameBuffer Size (3 frames are rendered out before playback will be started). So maybe after a while rendering is falling behind the scheduling which will cause the dropping/late frames. Maybe i'm not doing the openGL rendering process like it should be?
So here's my code:
-> first I'm loading a QCRenderer into memory
- (id)initWithPath:(NSString*)path forResolution:(NSSize)resolution
{
if (self = [super init])
{
NSOpenGLPixelFormatAttribute attributes[] = {
NSOpenGLPFAPixelBuffer,
NSOpenGLPFANoRecovery,
NSOpenGLPFAAccelerated,
NSOpenGLPFADepthSize, 24,
(NSOpenGLPixelFormatAttribute) 0
};
NSOpenGLPixelFormat* format = [[NSOpenGLPixelFormat alloc] initWithAttributes:attributes];
quartzPixelBuffer = nil;
quartzPixelBuffer = [[NSOpenGLPixelBuffer alloc] initWithTextureTarget:GL_TEXTURE_2D textureInternalFormat:GL_RGBA textureMaxMipMapLevel:0 pixelsWide:resolution.width pixelsHigh:resolution.height];
if(quartzPixelBuffer == nil)
{
NSLog(#"Cannot create OpenGL pixel buffer");
}
//Create the OpenGL context to render with (with color and depth buffers)
quartzOpenGLContext = [[NSOpenGLContext alloc] initWithFormat:format shareContext:nil];
if(quartzOpenGLContext == nil)
{
NSLog(#"Cannot create OpenGL context");
}
[quartzOpenGLContext setPixelBuffer:quartzPixelBuffer cubeMapFace:0 mipMapLevel:0 currentVirtualScreen:[quartzOpenGLContext currentVirtualScreen]];
//Create the QuartzComposer Renderer with that OpenGL context and the specified composition file
NSString* correctPath = [path substringWithRange:NSMakeRange(0, path.length - 1)];
quartzRenderer = [[QCRenderer alloc] initWithOpenGLContext:quartzOpenGLContext pixelFormat:format file:correctPath];
if(quartzRenderer == nil)
{
NSLog(#"Cannot create QCRenderer");
}
}
return self;
}
-> next step is to render 3 frames (BUFFER_DEPTH is set to 3 currently) before starting playback
- (void) preRollFrames;
{
// reset scheduled
[self resetScheduled];
totalFramesScheduled = 0;
if (isRunning == TRUE)
{
[self stopPlayback];
}
#autoreleasepool
{
for (double i = 0.0; i < ((1.0 / framesPerSecond) * BUFFER_DEPTH); i += 1.0/framesPerSecond)
{
// render image at given time
[self createVideoFrame:TRUE];
}
}
}
-> this is the createVideoFrame function. When scheduleBlack is set to true there must be rendered a black frame. If false the function renderFrameAtTime for the QCRenderer class is called. The return of this function is then passed to the decklinkVideoFrame object. Next, this frame will be pushed into the schedule queue of the DeckLink Card (SDK).
- (void) createVideoFrame:(BOOL)schedule
{
#autoreleasepool
{
// get displaymode
IDeckLinkDisplayMode* decklinkdisplaymode = (IDeckLinkDisplayMode*)CurrentRes;
// create new videoframe on output
if (deckLinkOutput->CreateVideoFrame((int)decklinkdisplaymode->GetWidth(), (int)decklinkdisplaymode->GetHeight(), (int)decklinkdisplaymode->GetWidth() * 4, bmdFormat8BitARGB, bmdFrameFlagFlipVertical, &videoFrame) != S_OK)
{
// failed to create new video frame on output
// display terminal message
sendMessageToTerminal = [[mogiTerminalMessage alloc] initWithSendNotification:#"terminalErrorMessage" forMessage:[NSString stringWithFormat:#"DeckLink: Output %d -> Failed to create new videoframe", outputID]];
}
unsigned frameBufferRowBytes = ((int)decklinkdisplaymode->GetWidth() * 4 + 63) & ~63;
void* frameBufferPtr = valloc((int)decklinkdisplaymode->GetHeight() * frameBufferRowBytes);
// set videoframe pointer
if (videoFrame != NULL)
{
videoFrame->GetBytes((void**)&frameBufferPtr);
}
// fill pointer with pixel data
if (scheduleBlack == TRUE)
{
[qClear renderFrameAtTime:1.0 forBuffer:(void**)frameBufferPtr forScreen:0];
// render first frame qRenderer
if (qRender != NULL)
{
[qRender renderFirstFrame];
}
}
else
{
[qRender renderFrameAtTime:totalSecondsScheduled forBuffer:(void**)frameBufferPtr forScreen:screenID];
schedule = TRUE;
}
// if playback -> schedule frame
if (schedule == TRUE)
{
// schedule frame
if (videoFrame != NULL)
{
if (deckLinkOutput->ScheduleVideoFrame(videoFrame, (totalFramesScheduled * frameDuration), frameDuration, frameTimescale) != S_OK)
{
// failed to schedule new frame
// display message to terminal
sendMessageToTerminal = [[mogiTerminalMessage alloc] initWithSendNotification:#"terminalErrorMessage" forMessage:[NSString stringWithFormat:#"DeckLink: Output %d -> Failed to schedule new videoframe", outputID]];
}
else
{
// increase totalFramesScheduled
totalFramesScheduled ++;
// increase totalSecondsScheduled
totalSecondsScheduled += 1.0/framesPerSecond;
}
// clear videoframe
videoFrame->Release();
videoFrame = NULL;
}
}
}
}
-> render frameAtTime function from QCRenderer class
- (void) renderFrameAtTime:(double)time forBuffer:(void*)frameBuffer forScreen:(int)screen
{
#autoreleasepool
{
CGLContextObj cgl_ctx = [quartzOpenGLContext CGLContextObj];
// render frame at time
[quartzRenderer renderAtTime:time arguments:NULL];
glReadPixels(0, 0, [quartzPixelBuffer pixelsWide], [quartzPixelBuffer pixelsHigh], GL_BGRA, GL_UNSIGNED_INT_8_8_8_8, frameBuffer);
}
}
-> After perolling frames Playback is started. Each time there's a frame played out. This Callback method is called (Decklink SDK). If there's a late frame. I'm pushing the totalFrames 1 frame into future.
PlaybackDelegate::PlaybackDelegate (DecklinkDevice* owner)
{
pDecklinkDevice = owner;
}
HRESULT PlaybackDelegate::ScheduledFrameCompleted (IDeckLinkVideoFrame* completedFrame, BMDOutputFrameCompletionResult result)
{
if (result == bmdOutputFrameDisplayedLate)
{
// if displayed late bump scheduled time further into the future by one frame
[pDecklinkDevice increaseScheduledFrames];
NSLog(#"bumped %d", [pDecklinkDevice getOutputID]);
}
if ([pDecklinkDevice getIsRunning] == TRUE)
{
[pDecklinkDevice createVideoFrame:TRUE];
}
return S_OK;
}
So my question. Am I doing the openGL rendering process correct? Maybe that is causing the delay after some minutes. Or I'm a handling the displayedLate frame incorrect so the timing of the scheduling queue is messed up after some time?
Thx!
Thomas

Answer 1

When late, try advancing the frame counter according to the completion result specified by the ScheduledFrameCompleted callback. Consider an extra increment of two.
At least on Windows and for many years now, only workstation boards provide unthrottled pixel readback when using NVidia's products. My iMac has a GeForce series card but I haven't measured its performance. I wouldn't be surprised if glReadPixels is throttled.
Also try using GL_BGRA_EXT and GL_UNSIGNED_INT_8_8_8_8_REV.
You should have precision timing metrics for glReadPixels and the writes to hardware. I assume you're reading back progressive or interlaced frames but not fields. Ideally, pixel readback should be less than 10 ms. And obviously, the entire render cycle needs be faster than the video hardware's frame rate.