I'm currently trying to create a 16 bit/channel or 48bpp image as an array in Objective-C, then put it in a NSBitmapImageRep. For a start, I want to just fill it with noise to see if it works, so I'm using a C for loop, and for some reason it creates a Segmentation Fault 11, and I can't see how that's possible. I've tried this EXACT code snippet in a simple .c file, and it works absolutely fine. By the way this is happening in the main function of the app, I want it to be done from a method on the click of a button, but don't know how I'd access the array from there as it is not global(could do with a couple of tips about that too).
Also i've tried using uint8_t instead of 16 bit pixels, doesn't make a difference.
int sizeElements = 1880 * 1056 * 3;
int sizeBytes = sizeElements * sizeof(uint16_t);
uint16_t * imagearray = (uint16_t *)malloc(sizeBytes);
for (int i = 1; i < sizeElements; i++)
{
imagearray[i] = rand() % 65536;
}
NSBitmapImageRep * theBitmap = [
[NSBitmapImageRep alloc]
initWithBitmapDataPlanes: (unsigned char * _Nullable * _Nullable)imagearray
pixelsWide: 1880
pixelsHigh: 1056
bitsPerSample: 16
samplesPerPixel: 3
hasAlpha: NO
isPlanar: NO
colorSpaceName: #"NSDeviceRGBColorSpace"
bitmapFormat: 0
bytesPerRow: 1880 * 3 * 2
bitsPerPixel: 48
];
Thanks to anyone who knows, this has been hurting me for hours :(
edit: initialising with NULL as bitmap data planes gets rid of the fault, but so does removing the for loop. Why is this and how can i get it working?
EDIT: adding & like this: &imagearray helped, thanks #everyoneWhoAnswered
The array of planes is not the pixel data, it's an array of pointers to pixel data. You need to put another array in between.
In your case, since your image is not planar, that's an array with the pixel data pointer in its first element, and the others set to NULL.
I have a terrain in OpenGL. I want to dynamicly change the space between points.
But when the vertex data is send to the vertex buffer object, i cannot modify anything.
The only thing i can do is delete the VBO and create a replacement VBO with new positions of each point.
Is there a best way to do this ?
As mentioned in the comments, it sounds like you want glBufferSubData.
If you planned to modify the data often, first setup your VBO's initial state:
float[] positions = { 0, 0, 0, 0, 0, 0 };
int numberOfPositions = 6;
int vbo = glGenBuffers();
glBindBuffer(GL_ARRAY_BUFFER, vbo);
glBufferData(GL_ARRAY_BUFFER, sizeof(float) * numberOfPositions, positions, GL_DYNAMIC_DRAW);
Then later say you want to change the last two values to 1, you would do this:
float[] update = { 1, 1 };
int offset = 4
glBindBuffer(GL_ARRAY_BUFFER, vbo);
glBufferSubData(GL_ARRAY_BUFFER, sizeof(float) * offset, update);
Check out the docs.gl page on glBufferSubData for more information.
It seems like this should be simpler than I'm finding it to be.
I have an AVFoundation frame coming back in the standard delegate method:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
where I would like to convert the frame to greyscale using the Accelerate.Framework.
There is a family of conversion methods in the framework, including vImageConvert_RGBA8888toPlanar8(), which looks like it might be what I would like to see, however, I can't find any examples of how to use them!
So far, I have the code:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
#autoreleasepool {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
/*Lock the image buffer*/
CVPixelBufferLockBaseAddress(imageBuffer,0);
/*Get information about the image*/
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
size_t stride = CVPixelBufferGetBytesPerRow(imageBuffer);
// vImage In
Pixel_8 *bitmap = (Pixel_8 *)malloc(width * height * sizeof(Pixel_8));
const vImage_Buffer inImage = { bitmap, height, width, stride };
//How can I take this inImage and convert it to greyscale?????
//vImageConvert_RGBA8888toPlanar8()??? Is the correct starting format here??
}
}
So I have two questions:
(1) In the code above, is RBGA8888 the correct starting format?
(2) How can I actually make the Accelerate.Framework call to convert to greyscale?
There is an easier option here. If you change the camera acquire format to YUV, then you already have a greyscale frame that you can use as you like. When setting up your data output, use something like:
dataOutput.videoSettings = #{ (id)kCVPixelBufferPixelFormatTypeKey : #(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) };
You can then access the Y plane in your capture callback using:
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
uint8_t *yPlane = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
... do stuff with your greyscale camera image ...
CVPixelBufferUnlockBaseAddress(pixelBuffer);
The vImage method is to use vImageMatrixMultiply_Planar8 and a 1x3 matrix.
vImageConvert_RGBA8888toPlanar8 is the function you use to convert a RGBA8888 buffer into 4 planar buffers. These are used by vImageMatrixMultiply_Planar8. vImageMatrixMultiply_ARGB8888 will do it too in one pass, but your gray channel will be interleaved with three other channels in the result. vImageConvert_RGBA8888toPlanar8 itself doesn't do any math. All it does is separate your interleaved image into separate image planes.
If you need to adjust the gamma as well, then probably vImageConvert_AnyToAny() is the easy choice. It will do the fully color managed conversion from your RGB format to a grayscale colorspace. See vImage_Utilities.h.
I like Tarks answer better though. It just leaves you in a position of having to color manage the Luminance manually (if you care).
Convert BGRA Image to Grayscale with Accelerate vImage
This method is meant to illustrate getting Accelerate's vImage use in converting BGR images to grayscale. Your image may very well be in RGBA format and you'll need to adjust the matrix accordingly, but the camera outputs BGRA so I'm using it here. The values in the matrix are the same values used in OpenCV for cvtColor, there are other values you might play with like luminosity. I assume you malloc the appropriate amount of memory for the result. In the case of grayscale it is only 1-channel or 1/4 the memory used for BGRA. If anyone finds issues with this code please leave a comment.
Performance note
Converting to grayscale in this way may NOT be the fastest. You should check the performance of any method in your environment. Brad Larson's GPUImage might be faster, or even OpenCV's cvtColor. In any case you will want to remove the calls to malloc and free for the intermediate buffers and manage them for the app lifecycle. Otherwise, the function call will be dominated by the malloc and free. Apple's docs recommend reusing the whole vImage_Buffer when possible.
You can also read about solving the same problem with NEON intrinsics.
Finally, the fastest method is not converting at all. If you're getting image data from the device camera the device camera is natively in the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format. Meaning, grabbing the first plane's data (Y-Channel, luma) is the fastest way to get grayscale.
BGRA to Grayscale
- (void)convertBGRAFrame:(const CLPBasicVideoFrame &)bgraFrame toGrayscale:(CLPBasicVideoFrame &)grayscaleFrame
{
vImage_Buffer bgraImageBuffer = {
.width = bgraFrame.width,
.height = bgraFrame.height,
.rowBytes = bgraFrame.bytesPerRow,
.data = bgraFrame.rawPixelData
};
void *intermediateBuffer = malloc(bgraFrame.totalBytes);
vImage_Buffer intermediateImageBuffer = {
.width = bgraFrame.width,
.height = bgraFrame.height,
.rowBytes = bgraFrame.bytesPerRow,
.data = intermediateBuffer
};
int32_t divisor = 256;
// int16_t a = (int16_t)roundf(1.0f * divisor);
int16_t r = (int16_t)roundf(0.299f * divisor);
int16_t g = (int16_t)roundf(0.587f * divisor);
int16_t b = (int16_t)roundf(0.114f * divisor);
const int16_t bgrToGray[4 * 4] = { b, 0, 0, 0,
g, 0, 0, 0,
r, 0, 0, 0,
0, 0, 0, 0 };
vImage_Error error;
error = vImageMatrixMultiply_ARGB8888(&bgraImageBuffer, &intermediateImageBuffer, bgrToGray, divisor, NULL, NULL, kvImageNoFlags);
if (error != kvImageNoError) {
NSLog(#"%s, vImage error %zd", __PRETTY_FUNCTION__, error);
}
vImage_Buffer grayscaleImageBuffer = {
.width = grayscaleFrame.width,
.height = grayscaleFrame.height,
.rowBytes = grayscaleFrame.bytesPerRow,
.data = grayscaleFrame.rawPixelData
};
void *scratchBuffer = malloc(grayscaleFrame.totalBytes);
vImage_Buffer scratchImageBuffer = {
.width = grayscaleFrame.width,
.height = grayscaleFrame.height,
.rowBytes = grayscaleFrame.bytesPerRow,
.data = scratchBuffer
};
error = vImageConvert_ARGB8888toPlanar8(&intermediateImageBuffer, &grayscaleImageBuffer, &scratchImageBuffer, &scratchImageBuffer, &scratchImageBuffer, kvImageNoFlags);
if (error != kvImageNoError) {
NSLog(#"%s, vImage error %zd", __PRETTY_FUNCTION__, error);
}
free(intermediateBuffer);
free(scratchBuffer);
}
CLPBasicVideoFrame.h - For reference
typedef struct
{
size_t width;
size_t height;
size_t bytesPerRow;
size_t totalBytes;
unsigned long pixelFormat;
void *rawPixelData;
} CLPBasicVideoFrame;
I got through the grayscale conversion, but was having trouble with the quality when I found this book on the web called Instant OpenCV for iOS. I personally picked up a copy and it has a number of gems, although the code is bit of a mess. On the bright-side it is a very reasonably priced eBook.
I'm very curious about that matrix. I toyed around with it for hours trying to figure out what the arrangement should be. I would have thought the values should be on the diagonal, but the Instant OpenCV guys put it as above.
if you need to use BGRA vide streams - you can use this excellent conversion
here
This is the function you'll need to take:
void neon_convert (uint8_t * __restrict dest, uint8_t * __restrict src, int numPixels)
{
int i;
uint8x8_t rfac = vdup_n_u8 (77);
uint8x8_t gfac = vdup_n_u8 (151);
uint8x8_t bfac = vdup_n_u8 (28);
int n = numPixels / 8;
// Convert per eight pixels
for (i=0; i < n; ++i)
{
uint16x8_t temp;
uint8x8x4_t rgb = vld4_u8 (src);
uint8x8_t result;
temp = vmull_u8 (rgb.val[0], bfac);
temp = vmlal_u8 (temp,rgb.val[1], gfac);
temp = vmlal_u8 (temp,rgb.val[2], rfac);
result = vshrn_n_u16 (temp, 8);
vst1_u8 (dest, result);
src += 8*4;
dest += 8;
}
}
more optimisations (using assembly) are in the link
(1) My experience with the iOS camera framework has been with images in the kCMPixelFormat_32BGRA format, which is compatible with the ARGB8888 family of functions. (It may be possible to use other formats as well.)
(2) The simplest way to convert from BGR to grayscale on iOS is to use vImageMatrixMultiply_ARGB8888ToPlanar8():
https://developer.apple.com/documentation/accelerate/1546979-vimagematrixmultiply_argb8888top
Here is a fairly complete example written in Swift. I'm assuming the Objective-C code would be similar.
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
// TODO: report error
return
}
// Lock the image buffer
if (kCVReturnSuccess != CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)) {
// TODO: report error
return
}
defer {
CVPixelBufferUnlockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)
}
// Create input vImage_Buffer
let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
let stride = CVPixelBufferGetBytesPerRow(imageBuffer)
var inImage = vImage_Buffer(data: baseAddress, height: UInt(height), width: UInt(width), rowBytes: stride)
// Create output vImage_Buffer
let bitmap = malloc(width * height)
var outImage = vImage_Buffer(data: bitmap, height: UInt(height), width: UInt(width), rowBytes: width)
defer {
// Make sure to free unless the caller is responsible for this
free(bitmap)
}
// Arbitrary divisor to scale coefficients to integer values
let divisor: Int32 = 0x1000
let fDivisor = Float(divisor)
// Rec.709 coefficients
var coefficientsMatrix = [
Int16(0.0722 * fDivisor), // blue
Int16(0.7152 * fDivisor), // green
Int16(0.2126 * fDivisor), // red
0 // alpha
]
// Convert to greyscale
if (kvImageNoError != vImageMatrixMultiply_ARGB8888ToPlanar8(
&inImage, &outImage, &coefficientsMatrix, divisor, nil, 0, vImage_Flags(kvImageNoFlags))) {
// TODO: report error
return
}
The code above was inspired by a tutorial from Apple on grayscale conversion, which can be found at the following link. It also includes conversion to a CGImage if that is needed. Note that they assume RGB order instead of BGR, and they only provide a 3 coefficients instead of 4 (mistake?)
https://developer.apple.com/documentation/accelerate/vimage/converting_color_images_to_grayscale
So basically I was messing about with LWJGL for a while now, and I came to a sudden stop with with annoyances surrounding glReadPixels().
And why it will only read from left-bottom -> top-right.
So I am here to answer my own question since I figured all this stuff out, And I am hoping my discoveries might be of some use to someone else.
As a side-note I am using:
glOrtho(0, WIDTH, 0 , HEIGHT, 1, -1);
So here it is my screen-capture code which can be implemented in any LWJGL application C:
//=========================getScreenImage==================================//
private void screenShot(){
//Creating an rbg array of total pixels
int[] pixels = new int[WIDTH * HEIGHT];
int bindex;
// allocate space for RBG pixels
ByteBuffer fb = ByteBuffer.allocateDirect(WIDTH * HEIGHT * 3);
// grab a copy of the current frame contents as RGB
glReadPixels(0, 0, WIDTH, HEIGHT, GL_RGB, GL_UNSIGNED_BYTE, fb);
BufferedImage imageIn = new BufferedImage(WIDTH, HEIGHT,BufferedImage.TYPE_INT_RGB);
// convert RGB data in ByteBuffer to integer array
for (int i=0; i < pixels.length; i++) {
bindex = i * 3;
pixels[i] =
((fb.get(bindex) << 16)) +
((fb.get(bindex+1) << 8)) +
((fb.get(bindex+2) << 0));
}
//Allocate colored pixel to buffered Image
imageIn.setRGB(0, 0, WIDTH, HEIGHT, pixels, 0 , WIDTH);
//Creating the transformation direction (horizontal)
AffineTransform at = AffineTransform.getScaleInstance(1, -1);
at.translate(0, -imageIn.getHeight(null));
//Applying transformation
AffineTransformOp opRotated = new AffineTransformOp(at, AffineTransformOp.TYPE_BILINEAR);
BufferedImage imageOut = opRotated.filter(imageIn, null);
try {//Try to screate image, else show exception.
ImageIO.write(imageOut, format , fileLoc);
}
catch (Exception e) {
System.out.println("ScreenShot() exception: " +e);
}
}
I hope this has been useful.
For any questions or comments on the code, ask/suggest as you like. C:
Hugs,
Rose.
sorry for the late reply but this is for anybody still looking for a solution.
public static void saveScreenshot() throws Exception {
System.out.println("Saving screenshot!");
Rectangle screenRect = new Rectangle(Display.getX(), Display.getY(), Display.getWidth(), Display.getHeight());
BufferedImage capture = new Robot().createScreenCapture(screenRect);
ImageIO.write(capture, "png", new File("doc/saved/screenshot.png"));
}
I have raw data of an image captured from my phone, it's resolution is 480*800, format is RGBA. Then I want to save it into a jpeg image with wxImage function. The cod is listed below:
wxFile m_bufferfile = wxFile(wxT("out.raw"));
if(!m_bufferfile->IsOpened())
{
wxLogMessage( _T("Fail to open the config file.") );
}
m_count = m_bufferfile->Length();
unsigned char* rawdata = new unsigned char[m_count];
for(unsigned int i = 0; i < m_count; i=i+4)
{
m_bufferfile->Seek(1,wxFromCurrent);
m_bufferfile->Read(rawdata, 3);
}
wxImage *image = new wxImage(480, 800, rawdata, false);
image->SaveFile(wxT("raw.jpg"),wxBITMAP_TYPE_JPEG);
When I opened the raw.jpg, it turns out black.
Is there wrong?
You have a bug here:
m_bufferfile->Read(rawdata, 3);
You only ever read into the beginning of your rawdata buffer.