I'm currently working a lot with CoreGraphics on OSX.
I've run Time Profiler over my code and found the biggest hang-up is in CGContextDrawImage. It's part of a loop that gets called many times per second.
I don't have any way of optimizing this code per se (since it's in the Apple libraries) - but I am wondering if there's a speedier alternative or way to improve the speed.
I'm using CGContextDraw image after some blend-mode code such as: CGContextSetBlendMode(context, kCGBlendModeDifference); so alternative implementations would need to be able to support blending.
Time profiler results:
3658.0ms 15.0% 0.0 CGContextDrawImage
3658.0ms 15.0% 0.0 ripc_DrawImage
3539.0ms 14.5% 0.0 ripc_AcquireImage
3539.0ms 14.5% 0.0 CGSImageDataLock
3539.0ms 14.5% 1.0 img_data_lock
3465.0ms 14.2% 0.0 img_interpolate_read
2308.0ms 9.4% 7.0 resample_band
1932.0ms 7.9% 1932.0 resample_byte_h_3cpp_vector
369.0ms 1.5% 369.0 resample_byte_v_Ncpp_vector
1157.0ms 4.7% 2.0 img_decode_read
1150.0ms 4.7% 8.0 decode_data
863.0ms 3.5% 863.0 decode_swap
267.0ms 1.0% 267.0 decode_byte_8bpc_3
Update:
The actual source is something along the lines of the following:
/////////////////////////////////////////////////////////////////////////////////////////
- (CGImageRef)createBlendedImage:(CGImageRef)image
secondImage:(CGImageRef)secondImage
blendMode:(CGBlendMode)blendMode
{
// Get the image width and height
size_t width = CGImageGetWidth(image);
size_t height = CGImageGetHeight(image);
// Set the frame
CGRect frame = CGRectMake(0, 0, width, height);
// Create context with alpha channel
CGContextRef context = CGBitmapContextCreate(NULL,
width,
height,
CGImageGetBitsPerComponent(image),
CGImageGetBytesPerRow(image),
CGImageGetColorSpace(image),
kCGImageAlphaPremultipliedLast);
if (!context) {
return nil;
}
// Draw the image inside the context
CGContextSetBlendMode(context, kCGBlendModeCopy);
CGContextDrawImage(context, frame, image);
// Set the blend mode and draw the second image
CGContextSetBlendMode(context, blendMode);
CGContextDrawImage(context, frame, secondImage);
// Get the masked image from the context
CGImageRef blendedImage = CGBitmapContextCreateImage(context);
CGContextRelease(context);
return blendedImage;
}
/////////////////////////////////////////////////////////////////////////////////////////
- (CGImageRef)createImageTick
{
// `self.image` and `self.previousImage` are two instance properties (CGImageRefs)
// Create blended image (stage one)
CGImageRef stageOne = [self createBlendedImage:self.image
secondImage:self.previousImage
blendMode:kCGBlendModeXOR];
// Create blended image (stage two) if stage one image is 50% red
CGImageRef stageTwo = nil;
if ([self isImageRed:stageOne]) {
stageTwo = [self createBlendedImage:self.image
secondImage:stageOne
blendMode:kCGBlendModeSourceAtop];
}
// Release intermediate image
CGImageRelease(stageOne);
return stageTwo;
}
#JeremyRoman et al: Thank you so much for your comments. I am drawing the same image a couple of times per loop, onto different contexts with different filters, and combining with new images. Does resampling include switching from RGB to RGBA? What could I try to speed up or eliminate resampling? – Chris Nolet
This is what Core Image is for. See the Core Image Programming Guide for details. CGContext is designed for rendering final images to the screen, which it sounds like is not your goal with every image you're creating.
Related
I have an app that renders into a UIView's CGContext in drawRect. I also export those renderings using a background renderer. It uses the same rendering logic to render (in faster than real time) into a CGBitmapContext (which I subsequently transform into an mp4 file).
I have noticed that the output video has a number of weird glitches. Such as the image being rotated, weird duplications of the rendered images, random noise, and the timing is also odd.
I'm looking for ways to debug this. For the timing issue, I thought I'd render a string that tells me which frame I'm currently viewing, only to find rendering text into CGContext's not very well documented. In fact, the documentations around much of core graphics is quite unforgiving to some one of my experience.
So specifically, I'd like to know how to render text into a context. If its Core Text, must it inter-operate some how with the core graphics context? And in general, I'd appreciate any tips and advice on doing bitmap rendering and debugging the results.
according another question:
How to convert Text to Image in Cocoa Objective-C
we can use the CTLineDraw to draw the text in a CGBitmapContext
sample code:
NSString* string = #"terry.wang";
CGFloat fontSize = 10.0f;
// Create an attributed string with string and font information
CTFontRef font = CTFontCreateWithName(CFSTR("Helvetica Light"), fontSize, nil);
NSDictionary* attributes = [NSDictionary dictionaryWithObjectsAndKeys:
(id)font, kCTFontAttributeName,
nil];
NSAttributedString* as = [[NSAttributedString alloc] initWithString:string attributes:attributes];
CFRelease(font);
// Figure out how big an image we need
CTLineRef line = CTLineCreateWithAttributedString((CFAttributedStringRef)as);
CGFloat ascent, descent, leading;
double fWidth = CTLineGetTypographicBounds(line, &ascent, &descent, &leading);
// On iOS 4.0 and Mac OS X v10.6 you can pass null for data
size_t width = (size_t)ceilf(fWidth);
size_t height = (size_t)ceilf(ascent + descent);
void* data = malloc(width*height*4);
// Create the context and fill it with white background
CGColorSpaceRef space = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGImageAlphaPremultipliedLast;
CGContextRef ctx = CGBitmapContextCreate(data, width, height, 8, width*4, space, bitmapInfo);
CGColorSpaceRelease(space);
CGContextSetRGBFillColor(ctx, 1.0, 1.0, 1.0, 1.0); // white background
CGContextFillRect(ctx, CGRectMake(0.0, 0.0, width, height));
// Draw the text
CGFloat x = 0.0;
CGFloat y = descent;
CGContextSetTextPosition(ctx, x, y);
CTLineDraw(line, ctx);
CFRelease(line);
I am making a PDF annotator and when you switch pages it has to redraw all of the previously drawn OpenGL content (which was saved to file in JSON format). The problem is that it takes longer the more content there is to draw. I have a UIImage saved to disk for each page so I was hoping to speed up this process by drawing that UIImage onto EAGLContext in one big stroke.
I want to know how to take an UIImage (or JPEG/PNG file) and draw it directly on to the screen. The reason why it has to be on the EAGLView is because it needs to support the eraser, and using the regular UIKit way wouldn't work with that.
I assume there's some way to set a brush as the whole image and just stamp the screen with it once. Any suggestions?
As a pedantic note, there is no standard class named EAGLView, but I assume you're referring to one of Apple's sample UIView subclasses that host OpenGL ES content.
The first step in doing this would be to load the UIImage into a texture. The following is some code that I've used for this in my image processing framework (newImageSource is the input UIImage):
CGSize pointSizeOfImage = [newImageSource size];
CGFloat scaleOfImage = [newImageSource scale];
pixelSizeOfImage = CGSizeMake(scaleOfImage * pointSizeOfImage.width, scaleOfImage * pointSizeOfImage.height);
CGSize pixelSizeToUseForTexture = pixelSizeOfImage;
BOOL shouldRedrawUsingCoreGraphics = YES;
// For now, deal with images larger than the maximum texture size by resizing to be within that limit
CGSize scaledImageSizeToFitOnGPU = [GPUImageOpenGLESContext sizeThatFitsWithinATextureForSize:pixelSizeOfImage];
if (!CGSizeEqualToSize(scaledImageSizeToFitOnGPU, pixelSizeOfImage))
{
pixelSizeOfImage = scaledImageSizeToFitOnGPU;
pixelSizeToUseForTexture = pixelSizeOfImage;
shouldRedrawUsingCoreGraphics = YES;
}
if (self.shouldSmoothlyScaleOutput)
{
// In order to use mipmaps, you need to provide power-of-two textures, so convert to the next largest power of two and stretch to fill
CGFloat powerClosestToWidth = ceil(log2(pixelSizeOfImage.width));
CGFloat powerClosestToHeight = ceil(log2(pixelSizeOfImage.height));
pixelSizeToUseForTexture = CGSizeMake(pow(2.0, powerClosestToWidth), pow(2.0, powerClosestToHeight));
shouldRedrawUsingCoreGraphics = YES;
}
GLubyte *imageData = NULL;
CFDataRef dataFromImageDataProvider;
if (shouldRedrawUsingCoreGraphics)
{
// For resized image, redraw
imageData = (GLubyte *) calloc(1, (int)pixelSizeToUseForTexture.width * (int)pixelSizeToUseForTexture.height * 4);
CGColorSpaceRef genericRGBColorspace = CGColorSpaceCreateDeviceRGB();
CGContextRef imageContext = CGBitmapContextCreate(imageData, (int)pixelSizeToUseForTexture.width, (int)pixelSizeToUseForTexture.height, 8, (int)pixelSizeToUseForTexture.width * 4, genericRGBColorspace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGContextDrawImage(imageContext, CGRectMake(0.0, 0.0, pixelSizeToUseForTexture.width, pixelSizeToUseForTexture.height), [newImageSource CGImage]);
CGContextRelease(imageContext);
CGColorSpaceRelease(genericRGBColorspace);
}
else
{
// Access the raw image bytes directly
dataFromImageDataProvider = CGDataProviderCopyData(CGImageGetDataProvider([newImageSource CGImage]));
imageData = (GLubyte *)CFDataGetBytePtr(dataFromImageDataProvider);
}
glBindTexture(GL_TEXTURE_2D, outputTexture);
if (self.shouldSmoothlyScaleOutput)
{
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR);
}
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, (int)pixelSizeToUseForTexture.width, (int)pixelSizeToUseForTexture.height, 0, GL_BGRA, GL_UNSIGNED_BYTE, imageData);
if (self.shouldSmoothlyScaleOutput)
{
glGenerateMipmap(GL_TEXTURE_2D);
}
if (shouldRedrawUsingCoreGraphics)
{
free(imageData);
}
else
{
CFRelease(dataFromImageDataProvider);
}
As you can see, this has some functions for resizing images that exceed the maximum texture size of the device (the class method in the above code merely queries the max texture size), as well as a boolean flag for whether or not to generate mipmaps for the texture for smoother downsampling. These can be removed if you don't care about those cases. This is also OpenGL ES 2.0 code, so there might be an OES suffix or two that you'd need to add to some of the functions above in order for them to work with 1.1.
Once you have the UIImage in a texture, you can draw it to the screen by using a textured quad (two triangles that make up a rectangle, with appropriate texture coordinates for the corners). How you do this will differ between OpenGL ES 1.1 and 2.0. For 2.0, you use a passthrough shader program that just reads the color from that location in the texture and draws that to the screen and for 1.1, you just set up the texture coordinates for your geometry and draw the two triangles.
I have some OpenGL ES 2.0 code for this in this answer.
Currently i load a texture in iOS using Image I/O and I extract its image data with Core Graphics. Then i can send the image data to OpenGL like this :
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, texture->width, texture->height, 0, GL_RGBA, GL_UNSIGNED_BYTE, texture->imageData);
The problem is that the Core Graphics part is really slow, i need to setup and draw with Core Graphics just to extract the image data...i don't want to show it on screen. There must be a more efficient way to extract image data in iOS?...
Here is my code :
...
myTexRef = CGImageSourceCreateWithURL((__bridge CFURLRef)url, myOptions);
...
MyTexture2D* texture;
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
void *imageData = malloc( tileSize.width * tileSize.height * 4 );
CGContextRef imgContext = CGBitmapContextCreate( imageData, tileSize.width, tileSize.height, 8, 4 * tileSize.width, colorSpace, kCGImageAlphaNoneSkipLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease( colorSpace );
CGContextClearRect( imgContext, CGRectMake( 0, 0, tileSize.width, tileSize.height ) );
CGContextTranslateCTM( imgContext, 0, 0 );
...
CGImageRef tiledImage = CGImageCreateWithImageInRect (imageRef, tileArea);
CGRect drawRect = CGRectMake(0, 0, tileSize.width, tileSize.height);
// *** THIS CALL IS REALLY EXPENSIVE!
CGContextDrawImage(imgContext, drawRect, tiledImage);
CGImageRelease(tiledImage);
// TamTexture2D takes the ownership of imageData and will be responsible to free it
texture = new MyTexture2D(tileSize.width, tileSize.height, imageData);
CGContextRelease(imgContext);
If you are developing for iOS 5 and above, GLKTextureLoader is what you're looking for:
GLKTextureLoader is orders of magnitude slower than using CG.
I have logged this as a bug with Apple DTS, and got it thrown back as "yes, we know, don't care, not going to fix it. You should be using CoreGraphics instead if you want your textures to be loaded fast" (almost that wording)
glTexImage2D is immensely fast if you give it the raw buffer that CGContext* methods create for you / allow you to pass in. Assuming you get the RGBA/ARGB/etc colour-spaces correct, of course.
cgcontextDrawImage is also super fast. My guess is that it's taking time to load your data over the web...
I've finally got my main app release (Tap Play MMO - check it out ;-) ) and I'm now working on expanding it.
To do this I need to have a circle that has four seperate buttons in it, these buttons will essentially be quarters. I've come to the conclusion that the circlular image will need to be constructed of four images, one for each quarter, but due to the necessity of rectangular image shapes I'm going to end up with some overlap, although the overlap will be transparent.
What's the best way of getting this to work? I need something really simple really, I've looked at this
http://iphonedevelopment.blogspot.com/2010/03/irregularly-shaped-uibuttons.html
Before but not yet succeeded in getting it to work. Anyone able to offer some advice?
In case it makes any difference I'll be deploying to a iOS 3.X framework (will be 4.2 down the line when 4.2 comes out for iPad)
Skip the buttons and simply respond to touches in your view that contains the circle.
Create a CGPath for each area that you want to capture touches, when your UIview receives a touch, check for membership inside the paths.
[Edited answer to show skeleton implementation details -- TomH]
Here's how I would approach the problem: (I haven't tested this code and the syntax may not be quite right, but this is the general idea)
1) Using PS or your favorite image creation application, create one png of the quarter circles. Add it to your XCode project.
2) Add a UIView to the UI. Set the UIView's layer's contents to the png.
self.myView = [[UIView alloc] initWithRect:CGRectMake(10.0, 10.0, 100.0, 100,0)];
[myView.layer setContents:(id)[UIImage loadImageNamed:#"my.png"]];
3) Create CGPaths that describe the region in the UIView that you are interested in.
self.quadrantOnePath = CGPathCreateMutable();
CGPathMoveToPoint(self.quadrantOnePath, NULL, 50.0, 50.0);
CGPathAddLineToPoint(self.quadrantOnePath, NULL, 100.0, 50.0);
CGPathAddArc(self.quadrantOnePath, NULL, 50.0, 50.0, 50.0, 0.0, M_PI2, 1);
CGPathCloseSubpath(self.quadrantOnePath);
// create paths for the other 3 circle quadrants too!
4) Add a UIGestureRecognizer and listen/observe for taps in the view
UITapGestureRecognizer *tapRecognizer = [[UITapGestureRecognizer alloc] initWithTarget:self action:#selector(handleGesture:)];
[tapRecognizer setNumberOfTapsRequired:2]; // default is 1
5) When tapRecognizer invokes its target selector
- (void)handleGesture:(UIGestureRecognizer *) recognizer {
CGPoint touchPoint = [recognizer locationOfTouch:0 inView:self.myView];
bool processTouch = CGPathContainsPoint(self.quadrantOnePath, NULL, touchPoint, true);
if(processTouch) {
// call your method to process the touch
}
}
Don't forget to release everything when appropriate -- use CGPathRelease to release paths.
Another thought: If the graphic that you are using to represent your circle quadrants is simply a filled color (i.e. no fancy graphics, layer effects, etc.), you could also use the paths you created in the UIView's drawRect method to draw the quadrants too. This would address one of the failings of the approach above: there isn't a tight integration between the graphic and the paths used to check for the touches. That is, if you swap out the graphic for something different, change the size of the graphic, etc., your paths used to check for touches will be out of sync. Potentially a high maintenance piece of code.
I can't see, why overlapping is needed.
Just create 4 buttons and give each one a slice of your image.
edit after comment
see this great project. One example is exactly what you want to do.
It works by incorporating the alpha-value of a pixel in the overwritten
pointInside:withEvent: and a category on UIImage, that adds this method
- (UIColor *)colorAtPixel:(CGPoint)point {
// Cancel if point is outside image coordinates
if (!CGRectContainsPoint(CGRectMake(0.0f, 0.0f, self.size.width, self.size.height), point)) {
return nil;
}
// Create a 1x1 pixel byte array and bitmap context to draw the pixel into.
// Reference: http://stackoverflow.com/questions/1042830/retrieving-a-pixel-alpha-value-for-a-uiimage
NSInteger pointX = trunc(point.x);
NSInteger pointY = trunc(point.y);
CGImageRef cgImage = self.CGImage;
NSUInteger width = self.size.width;
NSUInteger height = self.size.height;
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
int bytesPerPixel = 4;
int bytesPerRow = bytesPerPixel * 1;
NSUInteger bitsPerComponent = 8;
unsigned char pixelData[4] = { 0, 0, 0, 0 };
CGContextRef context = CGBitmapContextCreate(pixelData,
1,
1,
bitsPerComponent,
bytesPerRow,
colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);
CGContextSetBlendMode(context, kCGBlendModeCopy);
// Draw the pixel we are interested in onto the bitmap context
CGContextTranslateCTM(context, -pointX, pointY-(CGFloat)height);
CGContextDrawImage(context, CGRectMake(0.0f, 0.0f, (CGFloat)width, (CGFloat)height), cgImage);
CGContextRelease(context);
// Convert color values [0..255] to floats [0.0..1.0]
CGFloat red = (CGFloat)pixelData[0] / 255.0f;
CGFloat green = (CGFloat)pixelData[1] / 255.0f;
CGFloat blue = (CGFloat)pixelData[2] / 255.0f;
CGFloat alpha = (CGFloat)pixelData[3] / 255.0f;
return [UIColor colorWithRed:red green:green blue:blue alpha:alpha];
}
Here's an awesome project that solves the problem of irregular shaped buttons so easily:
http://christinemorris.com/2011/06/ios-irregular-shaped-buttons/
When I load textures from images normally, they are upside down because of OpenGL's coordinate system. What would be the best way to flip them?
glScalef(1.0f, -1.0f, 1.0f);
mapping the y coordinates of the textures in reverse
vertically flipping the image files manually (in Photoshop)
flipping them programatically after loading them (I don't know how)
This is the method I'm using to load png textures, in my Utilities.m file (Objective-C):
+ (TextureImageRef)loadPngTexture:(NSString *)name {
CFURLRef textureURL = CFBundleCopyResourceURL(
CFBundleGetMainBundle(),
(CFStringRef)name,
CFSTR("png"),
CFSTR("Textures"));
NSAssert(textureURL, #"Texture name invalid");
CGImageSourceRef imageSource = CGImageSourceCreateWithURL(textureURL, NULL);
NSAssert(imageSource, #"Invalid Image Path.");
NSAssert((CGImageSourceGetCount(imageSource) > 0), #"No Image in Image Source.");
CFRelease(textureURL);
CGImageRef image = CGImageSourceCreateImageAtIndex(imageSource, 0, NULL);
NSAssert(image, #"Image not created.");
CFRelease(imageSource);
GLuint width = CGImageGetWidth(image);
GLuint height = CGImageGetHeight(image);
void *data = malloc(width * height * 4);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
NSAssert(colorSpace, #"Colorspace not created.");
CGContextRef context = CGBitmapContextCreate(
data,
width,
height,
8,
width * 4,
colorSpace,
kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Host);
NSAssert(context, #"Context not created.");
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(context, CGRectMake(0, 0, width, height), image);
CGImageRelease(image);
CGContextRelease(context);
return TextureImageCreate(width, height, data);
}
Where TextureImage is a struct that has a height, width and void *data.
Right now I'm just playing around with OpenGL, but later I want to try making a simple 2d game. I'm using Cocoa for all the windowing and Objective-C as the language.
Also, another thing I was wondering about: If I made a simple game, with pixels mapped to units, would it be alright to set it up so that the origin is in the top-left corner (personal preference), or would I run in to problems with other things (e.g. text rendering)?
Thanks.
Any of those:
Flip texture during the texture load,
OR flip model texture coordinates during model load
OR set texture matrix to flip y (glMatrixMode(GL_TEXTURE)) during render.
Also, another thing I was wondering about: If I made a simple game, with pixels mapped to units, would it be alright to set it up so that the origin is in the top-left corner (personal preference), or would I run in to problems with other things (e.g. text rendering)?
Depends on how you are going to render text.
Jordan Lewis pointed out CGContextDrawImage draws image upside down when passed UIImage.CGImage. There I found a quick and easy solution: Before calling CGContextDrawImage,
CGContextTranslateCTM(context, 0, height);
CGContextScaleCTM(context, 1.0f, -1.0f);
Does the job perfectly well.