AVCaptureSession resolution doesn't change with AVCaptureSessionPreset - objective-c

I want to change the resolution of pictures I take with the camera on OS X with AV Foundation.
But even if I change the resolution of my AVCaptureSession, the output picture size doesn't change. I always have a 1280x720 picture.
I want a lower resolution because I use these pictures in a real time process and I want the program to be faster.
This is a sample of my code:
session = [[AVCaptureSession alloc] init];
if([session canSetSessionPreset:AVCaptureSessionPreset640x360]) {
[session setSessionPreset:AVCaptureSessionPreset640x360];
}
AVCaptureDeviceInput *device_input = [[AVCaptureDeviceInput alloc] initWithDevice:
[AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo][0] error:nil];
if([session canAddInput:device_input])
[session addInput:device_input];
still_image = [[AVCaptureStillImageOutput alloc] init];
NSDictionary *output_settings = [[NSDictionary alloc] initWithObjectsAndKeys:AVVideoCodecJPEG, AVVideoCodecKey, nil];
[still_image setOutputSettings : output_settings];
[session addOutput:still_image];
What should I change in my code?

I have also run into this issue and found a solution that seems to work. For some reason on OS X, StillImageOutput breaks the capture session presets.
What I did was change the AVCaptureDevice's active format directly. Try this code right after you add your StillImageOutput to your Capture Session.
//Get a list of supported formats for the device
NSArray *supportedFormats = [[AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo][0] formats];
//Find the format closest to what you are looking for
// this is just one way of finding it
NSInteger desiredWidth = 640;
AVCaptureDeviceFormat *bestFormat;
for (AVCaptureDeviceFormat *format in supportedFormats) {
CMVideoDimensions dimensions = CMVideoFormatDescriptionGetDimensions((CMVideoFormatDescriptionRef)[format formatDescription]);
if (dimensions.width <= desiredWidth) {
bestFormat = format;
}
}
[[AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo][0] lockForConfiguration:nil];
[[AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo][0] setActiveFormat:bestFormat];
[[AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo][0] unlockForConfiguration];
It is possible that there are other ways of fixing this issue but this is fixed it for me.

So I had this problem also, but using the raw AVCaptureVideoDataOutput instead of JPG.
The issue is that the session presets Low/Medium/High actually affects the capture device in some ways, for example framerate, but it won't change the hardware capture resolution - it will always capture at 1280x720. The idea, I think, is that a normal Quicktime output device will figure this out and add a scaling step to 640x480 (for example) if the session preset is to Medium.
But when using the raw outputs, they won't care about the preset desired dimensions.
The solution, contrary to Apples documentation on videoSettings, is to add the requested dimensions to the videoSettings:
NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithDouble:640], (id)kCVPixelBufferWidthKey,
[NSNumber numberWithDouble:480], (id)kCVPixelBufferHeightKey,
[NSNumber numberWithInt:kCMPixelFormat_422YpCbCr8_yuvs], (id)kCVPixelBufferPixelFormatTypeKey,
nil];
[captureoutput setVideoSettings:outputSettings];
I say contrary to Apples docs, because the docs say that the FormatTypeKey is the only key allowed here. But the bufferheight/width keys actually do work and are needed.

Related

How to snapshot and save a rendered Metal drawable in macOS Cocoa Objective-C?

I've got an app that uses Metal to do some rendering to screen (with a CAMetalLayer, not a MTKView, by necessity), and I'd like to provide the user with the option of saving a snapshot of the result to disk. Attempting to follow the answer at https://stackoverflow.com/a/47632198/2752221 while translating to Objective-C, I first wrote a commandBuffer completion callback like so (note this is manual retain/release code, not ARC; sorry, legacy code):
[commandBuffer addCompletedHandler:^(id<MTLCommandBuffer> buffer) {
[self performSelectorOnMainThread:#selector(saveImageTakingDrawable:) withObject:[drawable retain] waitUntilDone:NO];
optionKeyPressedAndUnhandled_ = NO;
}];
I do this immediately after calling [commandBuffer presentDrawable:drawable]; my id <CAMetalDrawable> drawable is still in scope. Here is my implementation of saveImageTakingDrawable::
- (void)saveImageTakingDrawable:(id <CAMetalDrawable>)drawable
{
// We need to have an image to save
if (!drawable) { NSBeep(); return; }
id<MTLTexture> displayTexture = drawable.texture;
if (!displayTexture) { NSBeep(); return; }
CIImage *ciImage = [CIImage imageWithMTLTexture:displayTexture options:nil];
// release the metal texture as soon as we can, to free up the system resources
[drawable release];
if (!ciImage) { NSBeep(); return; }
NSCIImageRep *rep = [NSCIImageRep imageRepWithCIImage:ciImage];
if (!rep) { NSBeep(); return; }
NSImage *nsImage = [[[NSImage alloc] initWithSize:rep.size] autorelease];
[nsImage addRepresentation:rep];
NSData *tiffData = [nsImage TIFFRepresentation];
if (!tiffData) { NSBeep(); return; }
... filesystem cruft culminating in ...
if ([tiffData writeToFile:filePath options:NSDataWritingWithoutOverwriting error:nil])
{
// play a sound to acknowledge saving
[[NSSound soundNamed:#"Tink"] play];
return;
}
NSBeep();
return;
}
The result is a "Tink" sound and a 7.8 MB .tif file of sensible dimensions (1784x1090), but it's transparent, and there is no usable image data in it; viewing the file in Hex Fiend shows that the whole file is all zeros except fairly brief header and footer sections.
I suspect that the fundamental method is flawed for some reason. I get several console logs when I attempt this snapshot:
2020-06-04 18:20:40.203669-0400 MetalTest[37773:1065740] [CAMetalLayerDrawable texture] should not be called after already presenting this drawable. Get a nextDrawable instead.
Input Metal texture was created with a device that does not match the current context device.
Input Metal texture was created with a device that does not match the current context device.
2020-06-04 18:20:40.247637-0400 MetalTest[37773:1065740] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x600000297260> F8BB1C28-BAE8-11D6-9C31-00039315CD46
2020-06-04 18:20:40.281161-0400 MetalTest[37773:1065740] HALC_ShellDriverPlugIn::Open: Can't get a pointer to the Open routine
That first log seems to suggest that I'm really not even allowed to get the texture out of the drawable after it has been presented in the first place. So... what's the right way to do this?
UPDATE:
Note that I am not wedded to the later parts of saveImageTakingDrawable:'s code. I would be happy to write out a PNG instead of a TIFF, and if there's a way to get where I'm going without using CIImage, NSCIImageRep, or NSImage, so much the better. I just want to save the drawable's texture image out as a PNG or TIFF, somehow.
I just want to save the drawable's texture image out as a PNG or TIFF, somehow.
Here is an alternative approach which you may test; will need to set the path to wherever you want the image file saved.
- (void) windowCapture: (id)sender {
NSTask *task = [[NSTask alloc]init];
[task setLaunchPath:#"/bin/sh"];
NSArray *args = [NSArray arrayWithObjects: #"-c", #"screencapture -i -c -Jwindow", nil];
[task setArguments:args];
NSPipe *pipe = [NSPipe pipe];
[task setStandardOutput:pipe];
[task launch];
[task waitUntilExit];
int status = [task terminationStatus];
NSData *dataRead = [[pipe fileHandleForReading] readDataToEndOfFile];
NSString *pipeOutput = [[[NSString alloc] initWithData:dataRead encoding:NSUTF8StringEncoding]autorelease];
// Tell us if there was a problem
if (!(status == 0)){NSLog(#"Error: %#",pipeOutput);}
[task release];
// Get image data from pasteboard and write to file
NSPasteboard *pboard = [NSPasteboard generalPasteboard];
NSData *pngData = [pboard dataForType:NSPasteboardTypePNG];
NSError *err;
BOOL success = [pngData writeToFile:#"/Users/xxxx/Desktop/ABCD.png" options:NSDataWritingAtomic error:&err];
if(!success){NSLog(#"Unable to write to file: %#",err);} else {NSLog(#"File written to desktop.");}
}
I think you should inject a blit into the command buffer (before submitting it) to copy the texture to a texture of your own. (The drawable's texture is not safe to use after it has been presented, as you've found.)
One strategy is to set the layer's framebufferOnly to false for the pass, and then use a MTLBlitCommandEncoder to encode a copy from the drawable texture to a texture of your own. Or, if the pixel formats are different, encode a draw of a quad from the drawable texture to your own, using a render command encoder.
The other strategy is to substitute your own texture as the render target color attachment where your code is currently using the drawable's texture. Render to your texture primarily and then draw that to the drawable's texture.
Either way, your texture's storageMode can be MTLStorageModeManaged, so you can access its data with one of the -getBytes:... methods. You just have to make sure that the command buffer encodes a synchronizeResource: command of a blit encoder at the end.
You can then use the bytes to construct an NSBitmapImageRep using the -initWithBitmapDataPlanes:pixelsWide:pixelsHigh:bitsPerSample:samplesPerPixel:hasAlpha:isPlanar:colorSpaceName:bitmapFormat:bytesPerRow:bitsPerPixel: method. Then, get PNG data from it using -representationUsingType:properties: and save that to file.

Objective c - How to programmatically set UIImage DPI of JPEG file while saving it

I have read that you can change the meta data of an image to set the dpi to another value other than the default 72. I tried the solution in this question but had the same problem as the author of that question. The image metadata properties in the original image seems to take precedence over modifications.
I am using the ALAssetsLibrary to write an image to the photo library on my iPhone. I need to dpi to be 500 instead of the standard 72dpi. I know it is possible to change the properties by manipulating the bits directly (as shown here), but am hoping that iOS gives a better solution.
Thank you in advance for your help.
This piece of code for manipulating image metadata -if exists- and you can use this to change any values in image meta data. Please be aware changing DPI value in metadata does not actually process image and change DPI.
#import <ImageIO/ImageIO.h>
-(NSData *)changeMetaDataInImage
{
NSData *sourceImageData = [[NSData alloc] initWithContentsOfFile:#"~/Desktop/1.jpg"];
if (sourceImageData != nil)
{
CGImageSourceRef source = CGImageSourceCreateWithData((__bridge CFDataRef)sourceImageData, NULL);
NSDictionary *metadata = (__bridge_transfer NSDictionary *)CGImageSourceCopyPropertiesAtIndex(source, 0, NULL);
NSMutableDictionary *tempMetadata = [metadata mutableCopy];
[tempMetadata setObject:[NSNumber numberWithInt:300] forKey:#"DPIHeight"];
[tempMetadata setObject:[NSNumber numberWithInt:300] forKey:#"DPIWidth"];
NSMutableDictionary *EXIFDictionary = [[tempMetadata objectForKey:(NSString *)kCGImagePropertyTIFFDictionary] mutableCopy];
[EXIFDictionary setObject:[NSNumber numberWithInt:300] forKey:(NSString *)kCGImagePropertyTIFFXResolution];
[EXIFDictionary setObject:[NSNumber numberWithInt:300] forKey:(NSString *)kCGImagePropertyTIFFYResolution];
NSMutableDictionary *JFIFDictionary = [[NSMutableDictionary alloc] init];
[JFIFDictionary setObject:[NSNumber numberWithInt:300] forKey:(NSString *)kCGImagePropertyJFIFXDensity];
[JFIFDictionary setObject:[NSNumber numberWithInt:300] forKey:(NSString *)kCGImagePropertyJFIFYDensity];
[JFIFDictionary setObject:#"1" forKey:(NSString *)kCGImagePropertyJFIFVersion];
[tempMetadata setObject:EXIFDictionary forKey:(NSString *)kCGImagePropertyTIFFDictionary];
[tempMetadata setObject:JFIFDictionary forKey:(NSString *)kCGImagePropertyJFIFDictionary];
NSMutableData *destinationImageData = [NSMutableData data];
CFStringRef UTI = CGImageSourceGetType(source);
CGImageDestinationRef destination = CGImageDestinationCreateWithData((__bridge CFMutableDataRef)destinationImageData, UTI, 1, NULL);
CGImageDestinationAddImageFromSource(destination, source,0, (__bridge CFDictionaryRef) tempMetadata);
CGImageDestinationFinalize(destination);
return destinationImageData;
}
}
While trying to employ this solution for exporting JPEGs, I found that Photoshop wouldn't honour the DPI as exported. It turns out that Photoshop actively ignores the JFIF header information (and, in fact, doesn't even export it itself when it writes out JPEGs).
I ditched the JFIF section entirely. Most modern image libraries look at the TIFF/EXIF data first, so including that (and only that) seems to work well. Your mileage may vary, so be sure to test as many export destinations as you can!
However, the TIFF data is missing the ResolutionUnit tag, without which Photoshop will ignore XResolution and YResolution. It can have three values:
1 = No absolute unit of measurement. Used for images that may have a non-square aspect ratio, but no meaningful absolute dimensions.
2 = Inch.
3 = Centimeter.
Setting ResolutionUnit to 2 is required for DPI (dots per inch), for example:
[EXIFDictionary setObject:#(2) forKey:(NSString *)kCGImagePropertyTIFFResolutionUnit];

Is there a way to be get the brightness level on iOS of a stream of the camera?

I am using the iPhone/iPad camera to get a video stream and doing recognition on the stream, but with lighting changes it has a negative impact on the robustness. I have tested different settings in different light and can get it to work, but trying to get the settings to adjust at run time is what I need.
I can calculate a simple brightness check on each frame, but the camera adjusts and throws my results off. I can watch for sharp changes and run checks then, but gradual changes would throw my results off as well.
Ideally I'd be like to access the camera/EXIF data for the stream and see what it is registering the unfiltered brightness as, is there a way to do this?
(I am working for devices iOS 5 and above)
Thank you
Available in iOS 4.0 and above. It's possible to get EXIF information from CMSampleBufferRef.
//Import ImageIO & include framework in your project.
#import <ImageIO/CGImageProperties.h>
In your sample buffer delegate toll-free bridging will get a NSDictionary of results from CoreMedia's CMGetAttachment.
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
NSDictionary* dict = (NSDictionary*)CMGetAttachment(sampleBuffer, kCGImagePropertyExifDictionary, NULL);
Complete code, as used in my own app:
- (void)setupAVCapture {
//-- Setup Capture Session.
_session = [[AVCaptureSession alloc] init];
[_session beginConfiguration];
//-- Set preset session size.
[_session setSessionPreset:AVCaptureSessionPreset1920x1080];
//-- Creata a video device and input from that Device. Add the input to the capture session.
AVCaptureDevice * videoDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
if(videoDevice == nil)
assert(0);
//-- Add the device to the session.
NSError *error;
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:videoDevice error:&error];
if(error)
assert(0);
[_session addInput:input];
//-- Create the output for the capture session.
AVCaptureVideoDataOutput * dataOutput = [[AVCaptureVideoDataOutput alloc] init];
[dataOutput setAlwaysDiscardsLateVideoFrames:YES]; // Probably want to set this to NO when recording
//-- Set to YUV420.
[dataOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]
forKey:(id)kCVPixelBufferPixelFormatTypeKey]]; // Necessary for manual preview
// Set dispatch to be on the main thread so OpenGL can do things with the data
[dataOutput setSampleBufferDelegate:self queue:dispatch_get_main_queue()];
[_session addOutput:dataOutput];
[_session commitConfiguration];
[_session startRunning];
}
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
CFDictionaryRef metadataDict = CMCopyDictionaryOfAttachments(NULL,
sampleBuffer, kCMAttachmentMode_ShouldPropagate);
NSDictionary *metadata = [[NSMutableDictionary alloc]
initWithDictionary:(__bridge NSDictionary*)metadataDict];
CFRelease(metadataDict);
NSDictionary *exifMetadata = [[metadata
objectForKey:(NSString *)kCGImagePropertyExifDictionary] mutableCopy];
self.autoBrightness = [[exifMetadata
objectForKey:(NSString *)kCGImagePropertyExifBrightnessValue] floatValue];
float oldMin = -4.639957; // dark
float oldMax = 4.639957; // light
if (self.autoBrightness > oldMax) oldMax = self.autoBrightness; // adjust oldMax if brighter than expected oldMax
self.lumaThreshold = ((self.autoBrightness - oldMin) * ((3.0 - 1.0) / (oldMax - oldMin))) + 1.0;
NSLog(#"brightnessValue %f", self.autoBrightness);
NSLog(#"lumaThreshold %f", self.lumaThreshold);
}
The lumaThreshold variable is sent as a uniform variable to my fragment shader, which multiplies the Y sampler texture to find the ideal luminosity based on the brightness of the environment. Right now, it uses the back camera; I'll probably switch to the front camera, since I'm only changing the "brightness" of the screen to adjust for indoor/outdoor viewing, and the user's eyes are on the front of the camera (and not the back).

QTCaptureDecompressedVideoOutput with FireWire

I'm trying to create a custom patch for Quartz Composer that captures from a FireWire input. I can open the device and add the input successfully. I even disable the audio inputs as per the QTKit documentation. I suspect my problem is either adding the output (although it adds without error). Every time the frame runs it crashes when it appears the imageBuffer is empty. It has no dimensions or anything. I'm using the standard captureOutput:didOutputVideoFrame:withSampleBuffer:fromConnection from the documentation.
[mCaptureDecompressedVideoOutput release];
mCaptureDecompressedVideoOutput = [[QTCaptureDecompressedVideoOutput alloc] init];
NSLog(#"allocated mCaptureDecompressedVideoOutput");
[mCaptureDecompressedVideoOutput setPixelBufferAttributes:
[NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferOpenGLCompatibilityKey,
[NSNumber numberWithLong:k32ARGBPixelFormat], kCVPixelBufferPixelFormatTypeKey, nil]];
[mCaptureDecompressedVideoOutput setDelegate:self];
success = [mCaptureSession addOutput:mCaptureDecompressedVideoOutput error:&error];
if (!success) {
NSLog(#"Failed to add output");
self.outputImage = nil;
if (mCaptureSession) {
[mCaptureSession release];
mCaptureSession= nil;
}
if (mCaptureDeviceInput) {
[mCaptureDeviceInput release];
mCaptureDeviceInput= nil;
}
if (mCaptureDecompressedVideoOutput) {
[mCaptureDecompressedVideoOutput release];
mCaptureDecompressedVideoOutput= nil;
}
return YES;
}
[mCaptureSession startRunning];
_currentDevice= self.inputDevice;
}
CVImageBufferRef imageBuffer = CVBufferRetain(mCurrentImageBuffer);
if (imageBuffer) {
CVPixelBufferLockBaseAddress(imageBuffer, 0);
NSLog(#"ColorSpace: %#", CVImageBufferGetColorSpace(imageBuffer));
id provider= [context outputImageProviderFromBufferWithPixelFormat:QCPlugInPixelFormatARGB8
pixelsWide:CVPixelBufferGetWidth(imageBuffer)
pixelsHigh:CVPixelBufferGetHeight(imageBuffer)
baseAddress:CVPixelBufferGetBaseAddress(imageBuffer)
bytesPerRow:CVPixelBufferGetBytesPerRow(imageBuffer)
releaseCallback:_BufferReleaseCallback
releaseContext:imageBuffer
colorSpace:CVImageBufferGetColorSpace(imageBuffer)
shouldColorMatch:YES];
What could I be doing wrong? This code works great for video only inputs, but not for FireWire (muxed) inputs.
I managed to get this patch to work with both Video and Muxed inputs by removing the kCVPixelBufferOpenGLCompatibilityKey from mCaptureDecompressedVideoOutput. While that allows the patch to work perfectly inside Quartz Composer, my intent is to run this patch in a composition that is used inside CamTwist, which appears not to need OpenGL support. Right now, it just displays a black screen with wither Video or Muxed inputs, where it was working with Video inputs before. So, I'm going to convert my CVImageBufferRef to an OpenGL texture and see if I can get that to work with
outputImageProviderFromTextureWithPixelFormat:pixelsWide:pixelsHigh:name:flipped:releaseCallback:releaseContext:colorSpace:shouldColorMatch

Reading samples via AVAssetReader

How do you read samples via AVAssetReader? I've found examples of duplicating or mixing using AVAssetReader, but those loops are always controlled by the AVAssetWriter loop. Is it possible just to create an AVAssetReader and read through it, getting each sample?
Thanks.
It's unclear from your question whether you are talking about video samples or audio samples. To read video samples, you need to do the following:
Construct an AVAssetReader:
asset_reader = [[AVAssetReader alloc] initWithAsset:asset error:&error];
(error checking goes here)
Get the video track(s) from your asset:
NSArray* video_tracks = [asset tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack* video_track = [video_tracks objectAtIndex:0];
Set the desired video frame format:
NSMutableDictionary* dictionary = [[NSMutableDictionary alloc] init];
[dictionary setObject:[NSNumber numberWithInt:<format code from CVPixelBuffer.h>] forKey:(NSString*)kCVPixelBufferPixelFormatTypeKey];
Note that certain video formats just will not work, and if you're doing something real-time, certain video formats perform better than others (BGRA is faster than ARGB, for instance).
Construct the actual track output and add it to the asset reader:
AVAssetReaderTrackOutput* asset_reader_output = [[AVAssetReaderTrackOutput alloc] initWithTrack:video_track outputSettings:dictionary];
[asset_reader addOutput:asset_reader_output];
Kick off the asset reader:
[asset_reader startReading];
Read off the samples:
CMSampleBufferRef buffer;
while ( [asset_reader status]==AVAssetReaderStatusReading )
buffer = [asset_reader_output copyNextSampleBuffer];
damian's answer worked for me with video, and one minor modification: In step 3, I think the dictionary needs to be mutable:
NSMutableDictionary* dictionary = [[NSMutableDictionary alloc] init];