I'm trying to play a video (MP4/H.263) on iOS, but getting really fuzzy results.
Here's the code to initialize the asset reading:
mTextureHandle = [self createTexture:CGSizeMake(400,400)];
NSURL * url = [NSURL fileURLWithPath:file];
mAsset = [[AVURLAsset alloc] initWithURL:url options:NULL];
NSArray * tracks = [mAsset tracksWithMediaType:AVMediaTypeVideo];
mTrack = [tracks objectAtIndex:0];
NSLog(#"Tracks: %i", [tracks count]);
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
NSDictionary * settings = [[NSDictionary alloc] initWithObjectsAndKeys:value, key, nil];
mOutput = [[AVAssetReaderTrackOutput alloc]
initWithTrack:mTrack outputSettings:settings];
mReader = [[AVAssetReader alloc] initWithAsset:mAsset error:nil];
[mReader addOutput:mOutput];
So much for the reader init, now the actual texturing:
CMSampleBufferRef sampleBuffer = [mOutput copyNextSampleBuffer];
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress( pixelBuffer, 0 );
glBindTexture(GL_TEXTURE_2D, mTextureHandle);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 600, 400, 0, GL_BGRA_EXT, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress( pixelBuffer ));
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
CFRelease(sampleBuffer);
Everything works well ... except the rendered image looks like this; sliced and skewed?
I've even tried looking into AVAssetTrack's preferred transformation matrix, to no avail, since it always returns CGAffineTransformIdentity.
Side-note: If I switch the source to camera, the image gets rendered fine. Am I missing some decompression step? Shouldn't that be handled by the asset reader?
Thanks!
Code: https://github.com/shaded-enmity/objcpp-opengl-video
I think the CMSampleBuffer uses a padding for performance reason, so you need to have the right width for the texture.
Try to set width of the texture with : CVPixelBufferGetBytesPerRow(pixelBuffer) / 4
(if your video format uses 4 bytes per pixel, change if other)
Related
I'm reading large video files, frame by frame, and hoping there is a faster way to get pixel buffers.
For those codecs which are supported I'm using AVFoundation APIs and then for the remainder (e.g. DNxHD) I'm using Quicktime techniques.
Here's a simplified version of my AVFoundation based approach.
Is there some API I have not noticed, or some attribute I could set to speed things up?
AVURLAsset *myAVURLAsset = [AVURLAsset URLAssetWithURL:#"blahblah" options:nil];
AVAssetTrack * videoTrack = [[myAVURLAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
AVAssetReader *movieReader = [[AVAssetReader alloc] initWithAsset:myAVURLAsset error:nil];
NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:#(kCVPixelFormatType_32ARGB) forKey:#"kCVPixelBufferPixelFormatTypeKey"];
AVAssetReaderTrackOutput * output = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:videoTrack outputSettings:videoSettings];
[movieReader addOutput:output];
[movieReader startReading];
bool atEnd = false;
while (!atEnd) {
CMSampleBufferRef sampleBuffer = [output copyNextSampleBuffer];
if (sampleBuffer)
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0);
uint8_t *data = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
// ...do lots of stuff with pixel data
// ...release and tidy up sampleBuffer etc
}
else{
atEnd = true;
}
}
I've been able to adapt some code found on SO to produce an animated GIF from the "screenshots" of my view, but the results are unpredictable. GIF frames are sometimes full images, full frames ("replace" mode, as GIMP marks it), other times are just a "diff" from previous layer ("combine" mode).
From what I've seen, when there are fewer and/or smaller frames involved, the CG writes the GIF in "combine" mode, but failing to get the colors right. Actually, the moving parts are colored correctly, the background is wrong.
When CG saves the GIF as full frames, the colors are ok. The file size is larger, but hey, obviously you cannot have the best of both worlds. :)
Is there a way to either:
a) force CG to create "full frames" when saving the GIF
b) fix the colors (color table?)
What I do is (ARC mode):
capture the visible part of the view with
[[scrollView contentView] dataWithPDFInsideRect:[[scrollView contentView] visibleRect]];
convert and resize it to NSImageBitmapRep of PNG type
-(NSMutableDictionary*) pngImageProps:(int)quality {
NSMutableDictionary *pngImageProps;
pngImageProps = [[NSMutableDictionary alloc] init];
[pngImageProps setValue:[NSNumber numberWithBool:NO] forKey:NSImageInterlaced];
double compressionF = 1;
[pngImageProps setValue:[NSNumber numberWithFloat:compressionF] forKey:NSImageCompressionFactor];
return pngImageProps;
}
-(NSData*) resizeImageToData:(NSData*)data toDimX:(int)xdim andDimY:(int)ydim withQuality:(int)quality{
NSImage *image = [[NSImage alloc] initWithData:data];
NSRect inRect = NSZeroRect;
inRect.size = [image size];
NSRect outRect = NSMakeRect(0, 0, xdim, ydim);
NSImage *outImage = [[NSImage alloc] initWithSize:outRect.size];
[outImage lockFocus];
[image drawInRect:outRect fromRect:inRect operation:NSCompositeCopy fraction:1];
NSBitmapImageRep* bitmapRep = [[NSBitmapImageRep alloc] initWithFocusedViewRect:outRect];
[outImage unlockFocus];
NSMutableDictionary *imageProps = [self pngImageProps:quality];
NSData* imageData = [bitmapRep representationUsingType:NSPNGFileType properties:imageProps];
return [imageData copy];
}
get the array of BitmapReps and create the GIF
-(CGImageRef) pngRepDataToCgImageRef:(NSData*)data {
CFDataRef imgData = (__bridge CFDataRef)data;
CGDataProviderRef imgDataProvider = CGDataProviderCreateWithCFData (imgData);
CGImageRef image = CGImageCreateWithPNGDataProvider(imgDataProvider, NULL, true, kCGRenderingIntentDefault);
return image;
}
////////// create GIF from
NSArray *images; // holds all BitmapReps
CGImageDestinationRef destination = CGImageDestinationCreateWithURL((__bridge CFURLRef)[NSURL fileURLWithPath:pot],
kUTTypeGIF,
allImages,
NULL);
// set frame delay
NSDictionary *frameProperties = [NSDictionary
dictionaryWithObject:[NSDictionary
dictionaryWithObject:[NSNumber numberWithFloat:0.2f]
forKey:(NSString *) kCGImagePropertyGIFDelayTime]
forKey:(NSString *) kCGImagePropertyGIFDictionary];
// set gif color properties
NSMutableDictionary *gifPropsDict = [[NSMutableDictionary alloc] init];
[gifPropsDict setObject:(NSString *)kCGImagePropertyColorModelRGB forKey:(NSString *)kCGImagePropertyColorModel];
[gifPropsDict setObject:[NSNumber numberWithBool:YES] forKey:(NSString *)kCGImagePropertyGIFHasGlobalColorMap];
// set gif loop
NSDictionary *gifProperties = [NSDictionary
dictionaryWithObject:gifPropsDict
forKey:(NSString *) kCGImagePropertyGIFDictionary];
// loop through frames and add them to GIF
for (int i=0; i < [images count]; i++) {
NSData *imageData = [images objectAtIndex:i];
CGImageRef imageRef = [self pngRepDataToCgImageRef:imageData];
CGImageDestinationAddImage(destination, imageRef, (__bridge CFDictionaryRef) (frameProperties));
}
// save the GIF
CGImageDestinationSetProperties(destination, (__bridge CFDictionaryRef)(gifProperties));
CGImageDestinationFinalize(destination);
CFRelease(destination);
I've checked the ImageBitmapReps, when saved as PNG individually, they are just fine.
As I understood, the color tables should be handled by CG or am I responsible to produce the dithered colors? How to do that?
Even when doing the same animation repeatedly, the GIFs produced may vary.
This is a single BitmapRep
(source: andraz.eu)
And this is the GIF with the invalid colors ("combine" mode)
(source: andraz.eu)
I read your code. Please double check the "allImages" while you are creating the CGImageDestinationRef, and the "[images count]".
the follow test code works fine:
NSDictionary *prep = [NSDictionary dictionaryWithObject:[NSDictionary dictionaryWithObject:[NSNumber numberWithFloat:0.2f] forKey:(NSString *) kCGImagePropertyGIFDelayTime] forKey:(NSString *) kCGImagePropertyGIFDictionary];
CGImageDestinationRef dst = CGImageDestinationCreateWithURL((__bridge CFURLRef)(fileURL), kUTTypeGIF, [filesArray count], nil);
for (int i=0;i<[filesArray count];i++)
{
//load anImage from array
...
CGImageRef imageRef=[anImage CGImageForProposedRect:nil context:nil hints:nil];
CGImageDestinationAddImage(dst, imageRef,(__bridge CFDictionaryRef)(prep));
}
bool fileSave = CGImageDestinationFinalize(dst);
CFRelease(dst);
I am using AGImagePickerController to pick multiple pictures from album, and then push the selected assets to a viewController where it tries to convert each asset into an UIImage.
However, I found out that if I selected more than 20 images, I will start to get memory low warning and the app exited. Here is my code of the conversion
for(int i =0 ; i < [self.selectedPictures count] ; i++)
{
NSLog(#"Object %d",i);
ALAsset *asset = [self.selectedPictures objectAtIndex:i];
ALAssetRepresentation *rep = [asset defaultRepresentation];
CGImageRef iref = [rep fullResolutionImage];
UIImage *anImage = [UIImage imageWithCGImage:iref scale:[rep scale] orientation:(UIImageOrientation)[rep orientation]];
float newHeight = anImage.size.height / (anImage.size.width / 1280);
UIImage *resizedImage = [anImage resizedImageWithContentMode:UIViewContentModeScaleAspectFit bounds:CGSizeMake(newHeight, 1280.f) interpolationQuality:kCGInterpolationHigh];
UIImage *resizedThumbnailImage = [anImage resizedImageWithContentMode:UIViewContentModeScaleAspectFill bounds:CGSizeMake(290.0f, 300.f) interpolationQuality:kCGInterpolationHigh];
// JPEG to decrease file size and enable faster uploads & downloads
NSData *imageData = UIImageJPEGRepresentation(resizedImage, 0.6f);
//NSData *thumbnailImageData = UIImagePNGRepresentation(thumbnailImage);
NSData *thumbnailImageData = UIImageJPEGRepresentation(resizedThumbnailImage, 0.6f);
PFFile *photoFile = [PFFile fileWithData:imageData];
PFFile *thumbnailFile = [PFFile fileWithData:thumbnailImageData];
[photoFile saveinbackground];
[thumbnailFile saveinbackground];
}
So i figured out that I should add CGImageRelease(iref); after anImage to release the iref, and the memory warning is gone. However, my app will crash after the last asset is converted to UIImage. And so far i could not find out why it is crashing.
You shouldn't be doing CGImageRelease(iref); unless you use CGImageCreate,
CGImageCreateCopy or CGImageRetain. That is the reason why it is crashing.
I found a way to fix this.
use #autoreleasepool
I'm writing a custom Movie Recording app and have implemented AVAssetWriter and AVAssetWriterInputPixelBufferAdaptor for writing frames to a file. In the DataOutputDelegate callback I am trying to apply a CIFilter to the sampleBuffer. First I get a CVPixelBufferRef and then create a CIImage. I then apply the CIIFilter and grab the resulting CIImage:
CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
CIImage *image = [CIImage imageWithCVPixelBuffer:pixelBuffer];
CIFilter *hueAdjust = [CIFilter filterWithName:#"CIHueAdjust"];
[hueAdjust setDefaults];
[hueAdjust setValue: image forKey: #"inputImage"];
[hueAdjust setValue: [NSNumber numberWithFloat: 2.094]
forKey: #"inputAngle"];
CIImage *result = [hueAdjust valueForKey: #"outputImage"];
CVPixelBufferRef newBuffer = //Convert CIImage...
How would I go about converting that CIImage so I can:
[self.filteredImageWriter appendPixelBuffer:newBuffer withPresentationTime:lastSampleTime];
UPDATED ANSWER
While my previous method works, I was able to tweak it to simplify the code. It also seems to runs slightly faster.
samplePixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(samplePixelBuffer, 0); // NOT SURE IF NEEDED // NO PERFORMANCE IMPROVEMENTS IF REMOVED
NSDictionary *options = [NSDictionary dictionaryWithObject:(__bridge id)rgbColorSpace forKey:kCIImageColorSpace];
outputImage = [CIImage imageWithCVPixelBuffer:samplePixelBuffer options:options];
//-----------------
// FILTER OUTPUT IMAGE
#autoreleasepool {
outputImage = [self applyEffectToCIImage:outputImage dict:dict];
}
CVPixelBufferUnlockBaseAddress(samplePixelBuffer, 0); // NOT SURE IF NEEDED // NO PERFORMANCE IMPROVEMENTS IF REMOVED
//-----------------
// RENDER OUTPUT IMAGE BACK TO PIXEL BUFFER
[self.filterContext render:outputImage toCVPixelBuffer:samplePixelBuffer bounds:[outputImage extent] colorSpace:CGColorSpaceCreateDeviceRGB()]; // DOES NOT SEEM TO WORK USING rgbColorSpace
PREVIOUS ANSWER
I just implemented the following to get the pixel buffer from a CIImage. Make sure your pixel formats are consistent otherwise you will have color issues. Also the CFDictionaries are very important. http://allmybrain.com/2011/12/08/rendering-to-a-texture-with-ios-5-texture-cache-api/
CFDictionaryRef empty = CFDictionaryCreate(kCFAllocatorDefault, // EMPTY IOSURFACE DICT
NULL,
NULL,
0,
&kCFTypeDictionaryKeyCallBacks,
&kCFTypeDictionaryValueCallBacks);
CFMutableDictionaryRef attributes = CFDictionaryCreateMutable(kCFAllocatorDefault,
1,
&kCFTypeDictionaryKeyCallBacks,
&kCFTypeDictionaryValueCallBacks);
CFDictionarySetValue(attributes,kCVPixelBufferIOSurfacePropertiesKey,empty);
CVPixelBufferRef pixelBuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault,
outputImage.extent.size.width,
outputImage.extent.size.height,
kCVPixelFormatType_32BGRA,
attributes,
&pixelBuffer);
if (status == kCVReturnSuccess && pixelBuffer != NULL) {
CVPixelBufferLockBaseAddress(pixelBuffer, 0); // NOT SURE IF NEEDED // KEPT JUST IN CASE
[self.filterContext render:outputImage toCVPixelBuffer:pixelBuffer bounds:[outputImage extent] colorSpace:CGColorSpaceCreateDeviceRGB()];
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0); // NOT SURE IF NEEDED // KEPT JUST IN CASE
}
I have a Cocoa Mac image editing app which lets users export JPEG images. I'm currently using the following code to export these images as JPEG files:
//this is user specified
NSInteger resolution;
NSImage* savedImage = [[NSImage alloc] initWithSize:NSMakeSize(600, 600)];
[savedImage lockFocus];
//draw here
[savedImage unlockFocus];
NSBitmapImageRep* savedImageBitmapRep = [NSBitmapImageRep imageRepWithData:[savedImage TIFFRepresentationUsingCompression:NSTIFFCompressionNone factor:1.0]];
NSDictionary* properties = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithFloat:1.0], NSImageCompressionFactor, nil];
//holds the jpeg file
NSData * imageData = nil;
imageData = [savedImageBitmapRep representationUsingType:NSJPEGFileType properties:properties];
However, I would like for the user to be able to provide the pixels per inch for this JPEG image (like you can in Photoshop's export options). What would I need to modify in the above code to adjust this value for the exported JPEG?
I couldn't find a way to do it with the NSImage APIs but CGImage can by setting kCGImagePropertyDPIHeight/Width.
I also set kCGImageDestinationLossyCompressionQuality which I think is the same as NSImageCompressionFactor.
//this is user specified
NSInteger resolution = 100;
NSImage* savedImage = [[NSImage alloc] initWithSize:NSMakeSize(600, 600)];
[savedImage lockFocus];
//draw here
[savedImage unlockFocus];
NSBitmapImageRep* savedImageBitmapRep = [NSBitmapImageRep imageRepWithData:[savedImage TIFFRepresentationUsingCompression:NSTIFFCompressionNone factor:1.0]];
NSDictionary* properties = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithFloat:1.0], kCGImageDestinationLossyCompressionQuality,
[NSNumber numberWithInteger:resolution], kCGImagePropertyDPIHeight,
[NSNumber numberWithInteger:resolution], kCGImagePropertyDPIWidth,
nil];
NSMutableData* imageData = [NSMutableData data];
CGImageDestinationRef imageDest = CGImageDestinationCreateWithData((CFMutableDataRef) imageData, kUTTypeJPEG, 1, NULL);
CGImageDestinationAddImage(imageDest, [savedImageBitmapRep CGImage], (CFDictionaryRef) properties);
CGImageDestinationFinalize(imageDest);
// Do something with imageData
if (![imageData writeToFile:[#"~/Desktop/test.jpg" stringByExpandingTildeInPath] atomically:NO])
NSLog(#"Failed to write imageData");
For NSImage or NSImageRep you do not set the resolution directly but set the size instead.
For size, numberOfPixels and resolution the following equation holds:
size = numberOfPixels * 72.0 / resolution
size is a length and is expressed in dots with the unit inch/72.
(size and resolution are floats). You can see that for an image with dpi=72 size and numberOfPixels are numerally the same (but the meaning is very different).
After creating an NSBitmapImageRep the size with the desired resolution can be set:
NSBitmapImageRep* savedImageBitmapRep = . . . ; // create the new rep
NSSize newSize;
newSize.width = [savedImageBitmapRep pixelsWide] * 72.0 / resolution; // x-resolution
newSize.height = [savedImageBitmapRep pixelsHigh] * 72.0 / resolution; // y-resolution
[savedImageBitmapRep setSize:newSize];
// save the rep
Two remarks: do you really need the lockFocus / unlockFocus way? The preferred way to build a new NSBitmapImageRep is to use NSGraphicsContext. see : http://www.mail-archive.com/cocoa-dev#lists.apple.com/msg74857.html
And: to use TIFFRepresentation for an NSBitmapImageRep is very time and space consuming. Since 10.6 another
way exists and costs nothing, because lockFocus and unlockFocus create an object of class NSCGImageSnapshotRep which under the hood is a CGImage. (In OS versions before 10.6 it was an NSCachedImageRep.) The following does it:
[anImage lockFocus];
// draw something
[anImage unlockFocus];
// now anImage contains an NSCGImageSnapshotRep
CGImageRef cg = [anImage CGImageForProposedRect:NULL context:nil hints:nil];
NSBitmapImageRep *newRep = [[NSBitmapImageRep alloc] initWithCGImage:cg];
// set the resolution
// here you may NSLog anImage, cg and newRep
// save the newRep
// release the newRep if needed