AVPlayer -seekToTime: Returns Black Frames - objective-c

Question

I have a Metal-based application that utilizes AVFoundation for movie playback + seeking. To start, I am only processing .mov files and nothing else, and even the app in question will not process any other format. While it has been working effectively in the past, I recently received feedback from some M1 users about black frames only showing up on their app regardless of which time they set their seek bar to.
I have performed the following troubleshooting in my attempts find the root source for this black texture bug:
Verified that the video being processed is .mov movie file type.
Verified that the CVPixelBufferRef object returned from AVPlayerItemVideoOutput's -copyPixelBufferForItemTime: is valid, i.e. not nil.
Verified that the MTLTexture created from the CVPixelBufferRef is also valid, i.e. also not nil.
Converted the MTLTexture to a bitmap and saved it as a .JPEG image to the user's disk.
The last part is probably the most important step here as the images saved are also all black (for users experiencing the bug) when viewed using Finder, and made me come to the assumption that I might be using AVFoundation somewhat wrong. While I hope that my opening post might not be too long in regards to code, below are the following steps I am performing in order to process videos to be rendered using Metal for your reference.
Inside my reader class, I have the following properties:
VideoReaderClass.m
#property (retain) AVPlayer *vidPlayer;
#property (retain) AVPlayerItem *vidPlayerItem;
#property (retain) AVPlayerItemVideoOutput *playerItemVideoOutput;
#property (retain) AVMutableComposition *videoMutableComp;
#property (assign) AVPlayerItemStatus playerItemStatus;
// The frame duration of the composition, as well as the media being processed.
#property (assign) CMTime frameDuration;
// Tracks the time to insert a new footage to
#property (assign) CMTime startInsertTime;
// Weak reference to a delegate responsible for processing seek:
#property (weak) id<VideoReaderRenderer> vidReaderRenderer;
A method called before reading starts. Handles observer cleanup as well.
- (void)initializeReadMediaPipeline
{
[_lock lock];
_startInsertTime = kCMTimeZero;
_playerItemStatus = AVPlayerItemStatusUnknown;
if(_videoMutableComp)
{
[_videoMutableComp release];
}
_videoMutableComp = [AVMutableComposition composition];
[_videoMutableComp retain];
if(_vidPlayer)
{
[[_vidPlayer currentItem] removeObserver:self
forKeyPath:#"status"
context:MyAddAVPlayerItemKVOContext];
[[_vidPlayer currentItem] removeObserver:self
forKeyPath:#"playbackBufferFull"
context:MyAVPlayerItemBufferFullKVOContext];
[[_vidPlayer currentItem] removeObserver:self
forKeyPath:#"playbackBufferEmpty"
context:MyAVPlayerItemBufferEmptyKVOContext];
[[_vidPlayer currentItem] removeObserver:self
forKeyPath:#"playbackLikelyToKeepUp"
context:MyAVPlayerItemBufferKeepUpKVOContext];
[_vidPlayer release];
_vidPlayer = nil;
}
[_lock unlock];
}
In this class is a public method to be called in a background queue when the app should begin processing a video, or a set of videos.
- (BOOL)addMediaAtLocation:(NSURL *)location
{
BOOL result = NO;
[_lock lock];
NSDictionary* optsInfo =
#{
AVURLAssetPreferPreciseDurationAndTimingKey : #(YES)
};
AVURLAsset* assetURL = [AVURLAsset URLAssetWithURL:location options:optsInfo];
AVAssetTrack* assetTrack = [assetURL tracksWithMediaType:AVMediaTypeVideo].firstObject;
NSError* error = nil;
[_videoMutableComp insertTimeRange:CMTimeRangeMake(kCMTimeZero, assetTrack.timeRange.duration)
ofAsset:assetURL
atTime:_startInsertTime
error:&error];
if(!error)
{
[_vidPlayerItem addObserver:self
forKeyPath:#"status"
options:NSKeyValueObservingOptionOld | NSKeyValueObservingOptionNew
context:MyAddAVPlayerItemKVOContext];
[_vidPlayerItem addObserver:self
forKeyPath:#"playbackBufferFull"
options:NSKeyValueObservingOptionOld | NSKeyValueObservingOptionNew
context:MyAVPlayerItemBufferFullKVOContext];
[_vidPlayerItem addObserver:self
forKeyPath:#"playbackBufferEmpty"
options:NSKeyValueObservingOptionOld | NSKeyValueObservingOptionNew
context:MyAVPlayerItemBufferEmptyKVOContext];
[_vidPlayerItem addObserver:self
forKeyPath:#"playbackLikelyToKeepUp"
options:NSKeyValueObservingOptionOld | NSKeyValueObservingOptionNew
context:MyAVPlayerItemBufferKeepUpKVOContext];
[_vidPlayer replaceCurrentItemWithPlayerItem:_vidPlayerItem];
}
_vidPlayer = [[AVPlayer alloc] init];
if(_playerItemVideoOutput)
{
[_playerItemVideoOutput release];
_playerItemVideoOutput = nil;
}
[_lock unlock];
}
Called externally by our view controller when seeking is needed.
- (void)seekToFrame:(CMTime)time
{
if(_vidReaderRenderer)
{
if(_vidPlayerItem && _playerItemVideoOutput)
{
[_vidPlayerItem seekTimeTime:time
toleranceBefore:kCMTimeZero
toleranceAfter:kCMTimeZero
completionHandler:^(BOOL finished){
if(finished)
{
CVPixelBufferRef p_buffer = [_playerItemVideoOutput copyPixelBufferForItemTime:time itemTimeForDisplay:nil];
if(p_buffer)
{
[_vidReaderRenderer seekOperationFinished:p_buffer];
}
}
}];
}
}
}
Lastly, this is where I handle the notification when the AVPlayerItem's status is set for "ReadyToPlay."
- (void)observeValueForKeyPath:(NSString *)keyPath
ofObject:(id)object
change:(NSDictionary<NSKeyValueChangeKey,id> *)change
context:(void *)context
{
// ... Checking for right contexts here, omitting for this example.
if([keyPath isEqualToString:#"status"])
{
AVPlayerItemStatus status = AVPlayerItemStatusUnknown;
// Get the status change from the change dictionary
NSNumber *statusNumber = change[NSKeyValueChangeNewKey];
if([statusNumber isKindOfClass:[NSNumber class]])
{
status = statusNumber.integerValue;
}
// Switch over the status
switch(status)
{
case AVPlayerItemStatusReadyToPlay:
{
// Ready to Play
if(_vidPlayerItem)
{
[_lock lock];
NSDictionary* attribs =
#{
(NSString*)kCVPixelBufferPixelFormatTypeKey : #(kCVPixelFormatType_64RGBAHalf)
};
_playerItemVideoOutput = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:attribs];
[_vidPlayerItem addOutput:_playerItemVideoOutput];
[_vidPlayer setRate:0.0];
_playerItemStatus = status;
[_lock unlock];
// "Wake up" the AVPlayer/AVPlayerItem here.
[self seekToFrame:kCMTimeZero];
}
break;
}
}
}
}
The code listing below is the class that also acts the delegate for a custom protocol called VideoReaderRenderer, which handles the seekToTime: completion block, as well as converting the pixel buffer to a MTLTexture:
RendererDelegate.m
At some point in its initialization and before performing any seek operations, I instantiate the CVMetalTextureCacheRef.
CVReturn ret_val = CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, _mtlDevice, nil, &_metalTextureCache);
The method in which my VideoReader class calls inside seekToTime's completion block:
-(void)seekOperationFinished:(CVPixelBufferRef)pixelBuffer
{
CVMetalTextureRef mtl_tex = NULL;
size_t w = CVPixelBufferGetWidth(pixelBuffer);
size_t h = CVPixelBufferGetHeight(pixelBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
CVReturn ret_val = CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
_metalTextureCache,
pixelBuffer,
nil,
MTLPixelFormatRGBA16Float,
w,
h,
0,
&mtl_tex);
CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
if(ret_val != kCVReturnSuccess)
{
if(mtl_tex != NULL)
{
CVBufferRelease(mtl_tex);
CVPixelBufferRelease(pixelBuffer);
}
return;
}
id<MTLTexture> inputTexture = CVMetalTextureGetTexture(mtl_tex);
if(!inputTexture)
return;
NSSize texSize = NSMakeSize(inputTexture.width, inputTexture.height);
_viewPortSize = (simd_uint2){(uint)texSize.width, (uint)texSize.height};
// Create the texture here.
[_textureLock lock];
if(NSEqualSizes(_projectSize, texSize))
{
if(!_inputFrameTex)
{
MTLTextureDescriptor* texDescriptor = [MTLTextureDescriptor new];
texDescriptor.width = texSize.width;
texDescriptor.height = texSize.height;
texDescriptor.pixelFormat = MTLPixelFormatRGBA16Float;
texDescriptor.usage = MTLTextureUsageShaderWrite | MTLTextureUsageShaderRead;
_inputFrameTex = [_mtlDevice newTextureWithDescriptor:texDescriptor];
}
id<MTLCommandBuffer> commandBuffer = [_drawCopyCommandQueue commandBuffer];
id<MTLBlitCommandEncoder> blitEncoder = [commandBuffer blitCommandEncoder];
[blitEncoder copyFromTexture:inputTexture
sourceSlice:0
sourceLevel:0
sourceOrigin:MTLOriginMake(0, 0, 0)
sourceSize:MTLSizeMake(inputTexture.width, inputTexture.height, 1)
toTexture:_inputFrameTex
destinationSlice:0
destinationLevel:0
destinationOrigin:MTLOriginMake(0, 0, 0)];
[blitEncoder endEncoding];
[commandBuffer commit];
[commandBuffer waitUntilCompleted];
}
[_textureLock unlock];
CVBufferRelease(mtl_tex);
CVPixelBufferRelease(pixelBuffer);
// Added "trouble-shooting" code to save the contents of the MTLTexture as a JPEG onto the user's disk.
if(_inputFrameTex)
{
// ... Save contents of texture to disk as JPEG.
}
}
Once again, my apologies for the rather long post. Additional details include having our rendering being displayed on a custom NSView backed by a CAMetalLayer, where its subsequent draw calls are called by a CVDisplayLink for background rendering. Though I don't think the latter seems to be the source of the problem of seekToTime: returning black frames. Can anyone care to shed some light to my situation? Thank you very much in advance.