Using NSImageView for video streaming - objective-c

Question

I am making an voip application which also has video support,
for video, i am getting data as a YUV format and to decode i am using libvpx,
and then i will RGB data,
Now to display i am using NSImageView where i will change the NSImage, please refer the code below
-(void)gotNewRGBBuffer:(void *)imageData Height:(int)height Width:(int)width Scan:(int)scan VideoId:(const char *)pId{
#autoreleasepool {
// display the image (on the UI thread)
NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc]
initWithBitmapDataPlanes:(unsigned char **)&imageData
pixelsWide:width pixelsHigh:height
bitsPerSample:8
samplesPerPixel:3 // or 4 with alpha
hasAlpha:NO
isPlanar:NO
colorSpaceName:NSDeviceRGBColorSpace
bitmapFormat:0
bytesPerRow:scan // 0 == determine automatically
bitsPerPixel:24]; // 0 == determine automatically
NSImage *image = [[NSImage alloc] initWithSize:NSMakeSize(width, height)];
[image addRepresentation:bitmap];
if ( ![NSThread isMainThread]){
[self performSelectorOnMainThread:#selector(updateImage:) withObject:image waitUntilDone:NO];
}else{
[self updateImage:image];
}
}
}
and function to update the image here it goes,
-(void)updateImage:(NSImage *)pNewLocalImage{
[self pImageView].image = pNewLocalImage;
[[self pImageView] setNeedsDisplay:YES];
NSLog(#" Updating the image ");
}
this is working, but taking some considerable amount of memory , so i my question, is there any other way to optimize it!

Answer 1

Thanks for looking at this,
I am now moved to OpenGL NSOpenGLView, which is rendering fast and consuming less cpu and memory,
Refer the code :
GLEssentials
In the given example, it will draw the PNG images , but i need to have YUV buffer, so you need to tweak the vertex and shaders like this,
- (BOOL)setupProgram {
NSAssert(!_program, #"program already set up");
GLuint vertexShader = CreateShader(GL_VERTEX_SHADER, kVertexShaderSource);
NSAssert(vertexShader, #"failed to create vertex shader");
GLuint fragmentShader =
CreateShader(GL_FRAGMENT_SHADER, kFragmentShaderSource);
NSAssert(fragmentShader, #"failed to create fragment shader");
_program = CreateProgram(vertexShader, fragmentShader);
// Shaders are created only to generate program.
if (vertexShader) {
glDeleteShader(vertexShader);
}
if (fragmentShader) {
glDeleteShader(fragmentShader);
}
if (!_program) {
return NO;
}
_position = glGetAttribLocation(_program, "position");
_texcoord = glGetAttribLocation(_program, "texcoord");
_ySampler = glGetUniformLocation(_program, "s_textureY");
_uSampler = glGetUniformLocation(_program, "s_textureU");
_vSampler = glGetUniformLocation(_program, "s_textureV");
if (_position < 0 || _texcoord < 0 || _ySampler < 0 || _uSampler < 0 ||
_vSampler < 0) {
return NO;
}
return YES;
}
Vertex and shader sources,
// Vertex shader doesn't do anything except pass coordinates through.
static const char kVertexShaderSource[] =
SHADER_VERSION
VERTEX_SHADER_IN " vec2 position;\n"
VERTEX_SHADER_IN " vec2 texcoord;\n"
VERTEX_SHADER_OUT " vec2 v_texcoord;\n"
"void main() {\n"
" gl_Position = vec4(position.x, position.y, 0.0, 1.0);\n"
" v_texcoord = texcoord;\n"
"}\n";
// Fragment shader converts YUV values from input textures into a final RGB
// pixel. The conversion formula is from http://www.fourcc.org/fccyvrgb.php.
static const char kFragmentShaderSource[] =
SHADER_VERSION
"precision highp float;"
FRAGMENT_SHADER_IN " vec2 v_texcoord;\n"
"uniform lowp sampler2D s_textureY;\n"
"uniform lowp sampler2D s_textureU;\n"
"uniform lowp sampler2D s_textureV;\n"
FRAGMENT_SHADER_OUT
"void main() {\n"
" float y, u, v, r, g, b;\n"
" y = " FRAGMENT_SHADER_TEXTURE "(s_textureY, v_texcoord).r;\n"
" u = " FRAGMENT_SHADER_TEXTURE "(s_textureU, v_texcoord).r;\n"
" v = " FRAGMENT_SHADER_TEXTURE "(s_textureV, v_texcoord).r;\n"
" u = u - 0.5;\n"
" v = v - 0.5;\n"
" r = y + 1.403 * v;\n"
" g = y - 0.344 * u - 0.714 * v;\n"
" b = y + 1.770 * u;\n"
" " FRAGMENT_SHADER_COLOR " = vec4(r, g, b, 1.0);\n"
" }\n";