Drawing a CIImage to a NSOpenGLView (without hitting main memory) - objective-c

Question

I feel like I must be missing something here. I've subclassed NSOpenGLView, and I'm attempting to draw a CIImage in the drawRect: call.
override func drawRect(dirtyRect: NSRect) {
super.drawRect(dirtyRect)
let start = CFAbsoluteTimeGetCurrent()
openGLContext!.makeCurrentContext()
let cglContext = openGLContext!.CGLContextObj
let pixelFormat = openGLContext!.pixelFormat.CGLPixelFormatObj
let colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceSRGB)!
let options: [String: AnyObject] = [kCIContextOutputColorSpace: colorSpace]
let context = CIContext(CGLContext: cglContext, pixelFormat: pixelFormat, colorSpace: colorSpace, options: options)
context.drawImage(inputImage, inRect: self.bounds, fromRect: input.extent)
openGLContext!.flushBuffer()
let end = CFAbsoluteTimeGetCurrent()
Swift.print("updated view in \(end - start)")
}
I'm obviously under the mistaken impression that the NSOpenGLContext (and it's underlying CGLContext) can be wrapped in a CIContext, and that rendering into that will produce an image in the view. But while work is being done in the above code, I have no idea where the actual pixels are ending up (because the view ends up blank).
If I just grab the current NSGraphicsContext and render into that, then I get an image in the NSOpenGLView, but the rendering seems to take about 10x as long (i.e. by changing the CIContext declaration to this):
// Works, but is slow
let context = NSGraphicsContext.currentContext()!.CIContext!
Also tried this, which is both slow AND doesn't actually display an image, making it a double fail:
// Doesn't work. AND it's slow.
input.drawInRect(bounds, fromRect: input.extent, operation: .CompositeDestinationAtop, fraction: 1.0)
The simple solution is just to render out to a CGImage, and then pop that onto the screen (via CGImage -> NSImage and then an NSImageView, or a backing CALayer). But that's not performant enough for my case. In my app, I'm looking to render a couple dozen thumbnail-sized images, each with their own different chain of CIFilters, and refresh them in realtime as their underlying base image changes. While they each render in a few milliseconds each (with my current CGImage-bracketed pathway), the view updates still are on the order of a few frames per second.
I currently have this working with a path that looks something like CGImageRef -> CIImage -> Bunch of CIFilters -> CGImage -> assign to CALayer for display. But it appears that having CGImages at both ends of the rendering chain is killing performance.
After profiling, it appears that MOST of the time is being spent just copying memory around, which I suppose is expected, but not very efficient. The backing CGImage needs to be shuttled to the GPU, then filtered, then goes back to main memory as a CGImage, then (presumably) goes right back to the GPU to be scaled and displayed by the CALayer. Ideally, the root images (before filtering) would just sit on the GPU, and results would be rendered directly to video memory, but I have no idea how to accomplish this. My current rendering pathway does the pixel smashing on the GPU (that's fantastic!), but is swamped by shuttling memory around to actually display the darned thing.
So can anyone enlighten me on how to do Core Image filtering with a pathway that keeps things on the GPU end-to-end? (or at least only has to swap that data in once?) If I have, say, a IOSurface-backed CIImage, how do I draw that directly to the UI without hitting main memory? Any hints?

Answer 1

After a day of banging my head against the wall on this, still no dice with NSOpenGLView... BUT, I think I'm able to do what I want via CAOpenGLLayer instead, which is just fine with me:
class GLLayer: CAOpenGLLayer {
var image: CIImage?
override func drawInCGLContext(ctx: CGLContextObj, pixelFormat pf: CGLPixelFormatObj, forLayerTime t: CFTimeInterval, displayTime ts: UnsafePointer<CVTimeStamp>) {
if image == nil {
let url = NSURL(fileURLWithPath: "/Users/doug/Desktop/test.jpg")
image = CIImage(contentsOfURL: url)!
}
let colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceSRGB)!
let options: [String: AnyObject] = [kCIContextOutputColorSpace: colorSpace]
let context = CIContext(CGLContext: ctx, pixelFormat: pf, colorSpace: colorSpace, options: options)
let targetRect = CGRectMake(-1, -1, 2, 2)
context.drawImage(image!, inRect: targetRect, fromRect: image!.extent)
}
}
One thing of note - the coordinate system for the CAOpenGLView is different. The view is 2.0x2.0 units, with the origin at 0,0 (took a while to figure that out). Other than that, this is basically the same as the non-working code in my original question (except, you know, it actually works here). Perhaps the NSOpenGLView class isn't returning the proper context. Who knows. Still interested in why, but at least I can move on now.

Answer 2

The code should work with following additions/modifications:
var ciContext: CIContext!
override func drawRect(dirtyRect: NSRect) {
super.drawRect(dirtyRect)
let start = CFAbsoluteTimeGetCurrent()
openGLContext!.makeCurrentContext()
let cglContext = openGLContext!.CGLContextObj
let attr = [
NSOpenGLPixelFormatAttribute(NSOpenGLPFAAccelerated),
NSOpenGLPixelFormatAttribute(NSOpenGLPFANoRecovery),
NSOpenGLPixelFormatAttribute(NSOpenGLPFAColorSize), 32,
0
]
let pixelFormat = NSOpenGLPixelFormat(attributes: attr)
let colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceSRGB)!
let options: [String: AnyObject] = [kCIContextOutputColorSpace:colorSpace, kCIContextWorkingColorSpace:colorSpace]
ciContext = CIContext(CGLContext: cglContext, pixelFormat: pixelFormat!.CGLPixelFormatObj, colorSpace: colorSpace, options: options)
glClearColor(0.0, 0.0, 0.0, 0.0)
glClear(GLbitfield(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT))
// setup viewing projection
glMatrixMode(GLenum(GL_PROJECTION))
// start with identity matrix
glLoadIdentity();
// Create an image
let image = NSImage(contentsOfFile: "/Users/kuestere/Pictures/DSCI0004.JPG")
let imageData = image?.TIFFRepresentation!
let ciImage = CIImage(data: imageData!)
if (ciImage == nil) {
Swift.print("ciImage loading error")
}
let imageFrame = ciImage!.extent
let viewFrame = self.visibleRect
// setup a viewing world left, right, bottom, top, zNear, zFar
glOrtho(0.0, Double(viewFrame.width), 0.0, Double(viewFrame.height), -1.0, 1.0);
ciContext.drawImage(ciImage!, inRect: self.visibleRect, fromRect: imageFrame)
glFlush();
let end = CFAbsoluteTimeGetCurrent()
Swift.print("updated view in \(end - start)")
}