Extracting RGB data from bitmapData (NSBitmapImageRep) Cocoa - objective-c

I am working on an application which needs to compare two images, in order to see how different they are and the application does this repeatedly for different images. So the way I currently do this is by having both the images as NSBitmapImageRep, then using the colorAtX: y: function in order to get a NSColor object, and then examining the RGB components. But this approach is extremely slow. So researching around the internet I found posts saying that a better way would be to get the bitmap data, using the function bitmapData, which returns an unsigned char. Unfortunately for me I don't know how to progress further from here, and none of the posts I've found show you how to actually get the RGB components for each pixel from this bitmapData. So currently I have :
NSBitmapImageRep* imageRep = [self screenShot]; //Takes a screenshot of the content displayed in the nswindow
unsigned char *data = [imageRep bitmapData]; //Get the bitmap data
//What do I do here in order to get the RGB components?
Thanks

The pointer you get back from -bitmapData points to the RGB pixel data. You have to query the image rep to see what format it's in. You can use the -bitmapFormat method which will tell you whether the data is alpha first or last (RGBA or ARGB), and whether the pixels are ints or floats. You need to check how many samples per pixel, etc. Here are the docs. If you have more specific questions about the data format, post those questions and we can try to help answer them.
Usually the data will be non-planar, which means it's just interleaved RGBA (or ARGB) data. You can loop over it like this (assuming 8-bit per channel, 4 channels of data) :
int width = [imageRep pixelsWide];
int height = [imageRep pixelsHight];
int rowBytes = [imageRep bytesPerRow];
char* pixels = [imageRep bitmapData];
int row, col;
for (row = 0; row < height; row++)
{
unsigned char* rowStart = (unsigned char*)(pixels + (row * rowBytes));
unsigned char* nextChannel = rowStart;
for (col = 0; col < width; col++)
{
unsigned char red, green, blue, alpha;
red = *nextChannel;
nextChannel++;
green = *nextChannel;
nextChannel++;
// ...etc...
}
}

Related

Changing gain of audio data in objective C results in messed up waveform

So I'm working on processing audio with Objective C, and am attempting to write a gain change function. I have limited the accepted audio formats to 16-bit AIFF files only for now. The process I am using is straightforward: I grab the audio data from my AIFF object, I skip to the point in the audio where I want to process (if x1: 10 and x2: 20 the goal is to change the amplitude of the samples from 10 seconds into the audio to 20 seconds in), and then step through the samples applying the gain change through multiplication. The problem is after I write the processed samples to a new NSMutableData, and then write a new AIFF file using the sound data, the processed samples are completely messed up, and the audio is basically just noise.
-(NSMutableData *)normalizeAIFF:(AIFFAudio *)audio x1:(int)x1 x2:(int)x2{
// obtain audio data bytes from AIFF object
SInt16 * bytes = (SInt16 *)[audio.ssndData bytes];
NSUInteger length = [audio.ssndData length] / sizeof(SInt16);
NSMutableData *newAudio = [[NSMutableData alloc] init];
int loudestSample = [self findLoudestSample:audio.ssndData];
// skip offset and blocksize in SSND data and proceed to user selected point
// For 16 bit, 44.1 audio, each second of sound data holds 88.2 thousand samples
int skipTo = 4 + (x1 * 88200);
int processChunk = ((x2 - x1) * 88200) + skipTo;
for(int i = skipTo; i < processChunk; i++){
// convert to float format for processing
Float32 sampleFloat = (Float32)bytes[i];
sampleFloat = sampleFloat / 32768.0;
// This is where I would change the amplitude of the sample
// sampleFloat = sampleFloat + (sampleFloat * 0.5);
// make sure not clipping
if (sampleFloat > 1.0){
sampleFloat = 1.0;
} else if (sampleFloat < -1.0){
sampleFloat = -1.0;
}
// convert back to SInt16
sampleFloat = sampleFloat * 32768.0;
if (sampleFloat > 32767.0){
sampleFloat = 32767.0;
} else if (sampleFloat < -32768.0){
sampleFloat = -32768.0;
}
bytes[i] = (SInt16)sampleFloat;
}
[newAudio appendBytes:bytes length:length];
return newAudio;
}
Where in this process could I be going wrong? Is it converting the sample from SInt16 -> float -> SInt16? Printing the data before during and after this conversion seems to show that there is nothing going wrong there. It seems to be after I pack it back into an NSMutableData object, but I'm not too sure.
Any help is appreciated.
EDIT: I also want to mention when I send audio through this function and set the change gain factor to 0 such that the resulting waveform is identical to the input, there are no issues. The waveform comes out looking and sounding exactly the same. It is only when the change gain factor is set to a value that actually changes the samples.
EDIT2: I changed the code to use a pointer and a type cast rather than memcpy(). I still am getting weird results when multiplying the floating point representation of the sample by any number. When I multiply the sample as an SInt16 by an integer I get the proper result, though. This leads me to believe my problem lies in the way I am going about floating point arithmetic. Is there anything anyone sees with the floating point equation I commented out that could be leading to errors?
The problem turned out to be an endianness issue as Zaph alluded to. I thought I was handling the conversion of big-endian to little-endian correctly when I was not. Now the code looks like:
-(NSMutableData *)normalizeAIFF:(AIFFAudio *)audio x1:(int)x1 x2:(int)x2{
// obtain audio data bytes from AIFF object
SInt16 * bytes = (SInt16 *)[audio.ssndData bytes];
NSUInteger length = [audio.ssndData length];
NSMutableData *newAudio = [[NSMutableData alloc] init];
// skip offset and blocksize in SSND data and proceed to user selected point
// For 16 bit, 44.1 audio, each second of sound data holds 88.2 thousand samples
int skipTo = 4 + (x1 * 88200);
int processChunk = ((x2 - x1) * 88200) + skipTo;
for(int i = skipTo; i < processChunk; i++){
SInt16 sample = CFSwapInt16BigToHost(bytes[i]);
bytes[i] = CFSwapInt16HostToBig(sample * 0.5);
}
[newAudio appendBytes:bytes length:length];
return newAudio;
}
The gain change factor of 0.5 will change, and I still have to actually normalize the data in relation to the sample with the greatest amplitude in the selection, but the issue I had is solved. When writing the new waveform out to a file it sounds and looks as expected.

Core Graphics pointillize effect on CGImage

So I have been writing a lot of image processing code lately using only core graphics and i have made quite a few working filters that manipulate the colors, apply blends, blurs and stuff like that. But I'm having trouble writing a filter to apply a pointillize effect to an image like this:
what I'm trying to do is get the color of a pixel and fill an ellipse with that color, looping through the image and doing this every few pixels here is the code:
EDIT: here is my new code this time its just drawing a few little circles in the bottom of the image am I doing it right like you said?
-(UIImage*)applyFilterWithAmount:(double)amount {
CGImageRef inImage = self.CGImage;
CFDataRef m_dataRef = CGDataProviderCopyData(CGImageGetDataProvider(inImage));
UInt8* m_pixelBuf = (UInt8*)CFDataGetBytePtr(m_dataRef);
int length = CFDataGetLength(m_dataRef);
CGContextRef ctx = CGBitmapContextCreate(m_pixelBuf,
CGImageGetWidth(inImage),
CGImageGetHeight(inImage),
CGImageGetBitsPerComponent(inImage),
CGImageGetBytesPerRow(inImage),
CGImageGetColorSpace(inImage),
CGImageGetBitmapInfo(inImage));
int row = 0;
int imageWidth = self.size.width;
if ((row%imageWidth)==0) {
row++;
}
int col = row%imageWidth;
for (int i = 0; i<length; i+=4) {
//filterPointillize(m_pixelBuf, i, context);
int r = i;
int g = i+1;
int b = i+2;
int red = m_pixelBuf[r];
int green = m_pixelBuf[g];
int blue = m_pixelBuf[b];
CGContextSetRGBFillColor(ctx, red/255, green/255, blue/255, 1.0);
CGContextFillEllipseInRect(ctx, CGRectMake(col, row, amount, amount));
}
CGImageRef imageRef = CGBitmapContextCreateImage(ctx);
CGContextRelease(ctx);
UIImage* finalImage = [UIImage imageWithCGImage:imageRef];
CGImageRelease(imageRef);
CFRelease(m_dataRef);
return finalImage;
}
One problem I see right off the bat is you are using the raster cell number for both your X and Y origin. A raster in this configuration is just a single dimension line. It is up to you to calculate the second dimension based on the raster image's width. That could explain why you got a line.
Another thing: seems like you are reading every pixel of the image. Didn't you want to skip pixels that are the width of the the ellipses you are trying to draw?
Next thing that looks suspicious is I think you should create the context you are drawing in before drawing. In addition, you should not be calling:
CGContextRef contextRef = UIGraphicsGetCurrentContext();
CGContextSaveGState(contextRef);
and
CGContextRestoreGState(contextRef);
inside the loop.
EDIT:
One further observation: your read RGB values are 0-255, and the CGContextSetRGBFillColor function expects values to be between 0.f - 1.f. This would explain why you got white. So you need to divide by 255:
CGContextSetRGBFillColor(contextRef, red / 255, green / 255, blue / 255, 1.0);
If you have any further questions, please don't hesitate to ask!
EDIT 2:
To calculate the row, first declare a row counter outside the loop:
int row = 0; //declare before the loop
int imageWidth = self.size.width; //get the image width
if ((i % imageWidth) == 0) { //we divide the cell number and if the remainder is 0
//then we want to increment the row counter
row++;
}
We can also use mod to calculate the current column:
int col = i % imageWidth; //divide i by the image width. the remainder is the col num
EDIT 3:
You have to put this inside the for loop:
if ((row%imageWidth)==0) {
row++;
}
int col = row%imageWidth;
Also, I forgot to mention before, to make the column and row 0 based (which is what you want) you will need to subtract 1 from the image size:
int imageWidth = self.size.width - 1;

Average Color of Mac Screen

I'm trying to find out a way to calculate the average color of the screen using objective-c.
So far I use this code to get a screen shot, which works great:
CGImageRef image1 = CGDisplayCreateImage(kCGDirectMainDisplay);
NSBitmapImageRep *bitmapRep = [[NSBitmapImageRep alloc] initWithCGImage:image1];
// Create an NSImage and add the bitmap rep to it...
NSImage *image = [[NSImage alloc] init];
[image addRepresentation:bitmapRep];
Now my problem is to calculate the average RGB color of this image.
I've found one solution, but the R G and B color components were always calculated to be the same (equal):
NSInteger i = 0;
NSInteger components[3] = {0,0,0};
unsigned char *data = [bitmapRep bitmapData];
NSInteger pixels = ([bitmapRep size].width *[bitmapRep size].height);
do {
components[0] += *data++;
components[1] += *data++;
components[2] += *data++;
} while (++i < pixels);
int red = (CGFloat)components[0] / pixels;
int green = (CGFloat)components[1] / pixels;
int blue = (CGFloat)components[2] / pixels;
A short analysis of bitmapRep shows that each pixel has 32 bits (4 bytes) where the first byte is unused, it is a padding byte, in other words the format is XRGB and X is not used. (There are no padding bytes at the end of a pixel row).
Another remark: for counting the number of pixels you use the method -(NSSize)size.
You should never do this! size has nothing to do with pixels. It only says how big the image should be depicted (expressed in inch or cm or mm) on the screen or the printer. For counting (or using otherwise) the pixels you should use -(NSInteger)pixelsWide and -(NSInteger)pixelsHigh. But the (wrong) using of -size works if and only if the resolution of the imageRep is 72 dots per inch.
Finally: there is a similar question at Average Color of Mac Screen
Your data is probably aligned as 4 bytes per pixel (and not 3 bytes, like you assume). That would (statistically) explain the near-equal values that you get.

Manipulating raw png data

I want to read a PNG file such that I can:
a) Access the raw bitmap data of the file, with no color space adjustment or alpha premultiply.
b) Based on that bitmap, display bit slices (any single bit of R, G, B, or A, across the whole image) in an image in the window. If I have the bitmap I can find the right bits, but what can I stuff them into to get them onscreen?
c) After some modification of the bitplanes, write a new PNG file, again with no adjustments.
This is only for certain specific images. The PNG is not expected to have any data other than simply RGBA-32.
From reading some similar questions here, I'm suspecting NSBitmapImageRep for the file read/write, and drawing in an NSView for the onscreen part. Does this sound right?
1.) You can use NSBitmapImageRep's -bitmapData to get the raw pixel data. Unfortunately, CG (NSBitmapImageRep's backend) does not support native unpremultiplication so you would have to unpremultiply yourself. The colorspace used in this will be the same as present in the file. Here is how to unpremultiply the image data:
NSBitmapImageRep *imageRep = [NSBitmapImageRep imageRepWithData:data];
NSInteger width = [imageRep pixelsWide];
NSInteger height = [imageRep pixelsHigh];
unsigned char *bytes = [imageRep bitmapData];
for (NSUInteger y = 0; y < width * height * 4; y += 4) { // bgra little endian + alpha first
uint8_t a, r, g, b;
if (imageRep.bitmapFormat & NSAlphaFirstBitmapFormat) {
a = bytes[y];
r = bytes[y+1];
g = bytes[y+2];
b = bytes[y+3];
} else {
r = bytes[y+0];
g = bytes[y+1];
b = bytes[y+2];
a = bytes[y+3];
}
// unpremultiply alpha if there is any
if (a > 0) {
if (!(imageRep.bitmapFormat & NSAlphaNonpremultipliedBitmapFormat)) {
float factor = 255.0f/a;
b *= factor;
g *= factor;
r *= factor;
}
} else {
b = 0;
g = 0;
r = 0;
}
bytes[y]=a; // for argb
bytes[y+1]=r;
bytes[y+2]=g;
bytes[y+3]=b;
}
2.) I couldn't think of a simple way to do this. You could make your own image drawing method that loops through the raw image data and generates a new image based on the values. Refer above to see how to start doing it.
3.) Here is a method to get a CGImage from raw data places (you can write the png to a file using native CG functions or convert it to NSBitmapImageRep if CG makes you uncomfortable)
static CGImageRef cgImageFrom(NSData *data, uint16_t width, uint16_t height) {
CGDataProviderRef provider = CGDataProviderCreateWithCFData((CFDataRef)data);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGImageAlphaFirst;
CGImageRef cgImage = CGImageCreate(width, height, 8, 32, 4 * width, colorSpace, bitmapInfo, provider, NULL, NO, kCGRenderingIntentDefault);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return cgImage;
}
You can create the NSData object out of the raw data object with +dataWithBytes:length:
I haven't ever worked in this area, but you may be able to use Image IO for this.

iOS OpenGL using parameters for glTexImage2D to make a UIImage?

I am working through some existing code for a project i am assigned to.
I have a successful call to glTexImage2D like this:
glTexImage2D(GL_TEXTURE_2D, 0, texture->format, texture->widthTexture, texture->heightTexture, 0, texture->format, texture->type, texture->data);
I would like create an image (preferably a CGImage or UIImage) using the variables passed to glTexImage2D, but don't know if it's possible.
I need to create many sequential images(many of them per second) from an OpenGL view and save them for later use.
Should i be able to create a CGImage or UIImage using the variables i use in glTexImage2D?
If i should be able to, how should i do it?
If not, why can't i and what do you suggest for my task of saving/capturing the contents of my opengl view many times per second?
edit: i have already successfully captured images using some techniques provided by apple with glReadPixels, etc etc. i want something faster so i can get more images per second.
edit: after reviewing and adding the code from Thomson, here is the resulting image:
the image very slightly resembles what the image should look like, except duplicated ~5 times horizontally and with some random black space underneath.
note: the video(each frame) data is coming over an ad-hoc network connection to the iPhone. i believe the camera is shooting over each frame with the YCbCr color space
edit: further reviewing Thomson's code
I have copied your new code into my project and got a different image as result:
width: 320
height: 240
i am not sure how to find the number of bytes in texture-> data. it is a void pointer.
edit: format and type
texture.type = GL_UNSIGNED_SHORT_5_6_5
texture.format = GL_RGB
Hey binnyb, here's the solution to creating a UIImage using the data stored in texture->data. v01d is certainly right that you're not going to get the UIImage as it appears in your GL framebuffer, but it'll get you an image from the data before it has passed through the framebuffer.
Turns out your texture data is in 16 bit format, 5 bits for red, 6 bits for green, and 5 bits for blue. I've added code for converting the 16 bit RGB values into 32 bit RGBA values before creating a UIImage. I'm looking forward to hearing how this turns out.
float width = 512;
float height = 512;
int channels = 4;
// create a buffer for our image after converting it from 565 rgb to 8888rgba
u_int8_t* rawData = (u_int8_t*)malloc(width*height*channels);
// unpack the 5,6,5 pixel data into 24 bit RGB
for (int i=0; i<width*height; ++i)
{
// append two adjacent bytes in texture->data into a 16 bit int
u_int16_t pixel16 = (texture->data[i*2] << 8) + texture->data[i*2+1];
// mask and shift each pixel into a single 8 bit unsigned, then normalize by 5/6 bit
// max to 8 bit integer max. Alpha set to 0.
rawData[channels*i] = ((pixel16 & 63488) >> 11) / 31.0 * 255;
rawData[channels*i+1] = ((pixel16 & 2016) << 5 >> 10) / 63.0 * 255;
rawData[channels*i+2] = ((pixel16 & 31) << 11 >> 11) / 31.0 * 255;
rawData[channels*4+3] = 0;
}
// same as before
int bitsPerComponent = 8;
int bitsPerPixel = channels*bitsPerComponent;
int bytesPerRow = channels*width;
CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault;
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
CGDataProviderRef provider = CGDataProviderCreateWithData(NULL,
rawData,
channels*width*height,
NULL);
free( rawData );
CGImageRef imageRef = CGImageCreate(width,
height,
bitsPerComponent,
bitsPerPixel,
bytesPerRow,
colorSpaceRef,
bitmapInfo,
provider,NULL,NO,renderingIntent);
UIImage *newImage = [UIImage imageWithCGImage:imageRef];
The code for creating a new image comes from Creating UIImage from raw RGBA data thanks to Rohit. I've tested this with our original 320x240 image dimension, having converted a 24 bit RGB image into 5,6,5 format and then up to 32 bit. I haven't tested it on a 512x512 image but I don't expect any problems.
You could make an image from the data you are sending to GL, but I doubt that's really what you want to achieve.
My guess is you want the output of the Frame Buffer. To do that you need glReadPixels(). Bare in mind for a large buffer (say 1024x768) it will take seconds to read the pixels back from GL, you wont get more than 1 per second.
You should be able to use the UIImage initializer imageWithData for this. All you need is to ensure that the data in texture->data is in a structured format that is recognizable to the UIImage constructor.
NSData* imageData = [NSData dataWithBytes:texture->data length:(3*texture->widthTexture*texture->heightTexture)];
UIImage* theImage = [UIImage imageWithData:imageData];
The types that imageWithData: supports are not well documented, but you can create NSData from .png, .jpg, .gif, and I presume .ppm files without any difficulty. If texture->data is in one of those binary formats I suspect you can get this running with a little experimentation.