I was using CGImageCreate with CGColorSpaceCreateDeviceGray to convert a buffer (CVPixelBufferRef) to grayscale image. It was very fast and did work well until iOS 12... now the returned image is empty.
The code look like this:
bitmapInfo = kCGBitmapByteOrder32Little | kCGImageAlphaNoneSkipFirst;
CGDataProviderRef provider = CGDataProviderCreateWithData((void *)i_PixelBuffer,
sourceBaseAddr,
sourceRowBytes * height,
ReleaseCVPixelBuffer);
retImage = CGImageCreate(width,
height,
8,
32,
sourceRowBytes,
CGColorSpaceCreateDeviceGray(),
bitmapInfo,
provider,
NULL,
true,
kCGRenderingIntentDefault);
CGDataProviderRelease(provider);
This is a known bug in iOS 12? If device gray is no supported anymore in this function, can you suggest me another way to do it?
Note that conversion should take less than 0.1 seconds for a 4K image.
Thanks in advance!
According to the list of Supported Pixel Formats in the Quartz 2D Programming Guide, iOS doesn't support 32 bits per pixel with gray color spaces. And even on macOS, 32 bpp gray requires the use of kCGBitmapFloatComponents (and float data).
Is your data really 32 bpp? If so, is it float? What are you using for bitmapInfo?
I would not expect CGImageCreate() to "convert" a buffer, including to grayscale. The parameters you're supplying are telling it how to interpret the data. If you're not using floating-point components, I suspect it was just taking one of the color channels and interpreting that as the gray level and ignoring the other components. So, it wasn't a proper grayscale conversion.
Apple's advice is to create an image that properly represents the image; create a bitmap context with the colorspace, pixel layout, and bitmap info you desire; draw the former into the latter; and create the final image from the context.
I finally found a workaround for my purpose. Note that the CVPixelBuffer is coming from the video camera.
Changed camera output pixel format to
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
(AVCaptureVideoDataOutput)
Extract the Y plane from YpCbCr
Build a CGImage with the Y plane
Code:
// some code
colorSpace = CGColorSpaceCreateDeviceGray();
sourceRowBytes = CVPixelBufferGetBytesPerRowOfPlane(i_PixelBuffer, 0);
sourceBaseAddr = (unsigned char*)CVPixelBufferGetBaseAddressOfPlane(i_PixelBuffer,0);
bitmapInfo = kCGImageByteOrderDefault;
// some code
CGContextRef context = CGBitmapContextCreate(sourceBaseAddr,
width,
height,
8,
sourceRowBytes,
colorSpace,
bitmapInfo);
retImage = CGBitmapContextCreateImage(context);
// some code
You can also look at this related post:
420YpCbCr8BiPlanarVideoRange To YUV420 ?/How to copy Y and Cbcr plane to Single plane?
Related
Curently based on Sascha Willems examples I've cerated samplerCube texture for fragment shader.
It has same JPG image copied to all 6 layers (faces).
I use stbi image library for image loading, it works okay if I use it for regular 2D texture, but if it's mapped on cube mesh it creates distorted image:
int width = 0, height = 0, channel = 0;
float* pixels = stbi_loadf("textures/test.hdr", &width, &height, &channel, STBI_rgb_alpha);
if(!pixels) throw std::runtime_error("failed to load texture image!");
this->texture_image.create_image(width, height, VK_FORMAT_R32G32B32A32_SFLOAT, VK_IMAGE_USAGE_TRANSFER_DST_BIT | VK_IMAGE_USAGE_SAMPLED_BIT);
this->texture_image.fill_memory(width, height, 4*sizeof(float), pixels);
this->texture_image.create_image_view(VK_FORMAT_R32G32B32A32_SFLOAT, VK_IMAGE_ASPECT_COLOR_BIT);
stbi_image_free(pixels);
Found how to do it here: https://learnopengl.com/PBR/IBL/Diffuse-irradiance
Even if it's in OpenGL the concenpt is the same.
I have an app written with RXSwift which processes 500+ days of HealthKit data to draw a chart for the user.
The chart image is drawn incrementally using the code below. Starting with a black screen, previous image is drawn in the graphics context, then a new segment is drawn over this image with certain offset. The combined image is saved and the process repeats around 70+ times. Each time the image is saved, so the user sees the update. The result is a single chart image which the user can export from the app.
Even with autorelease pool, I see spikes of memory usage up to 1Gb, which prevents me from doing other resource intensive processing.
How can I optimize incremental drawing of very large (1440 × 5000 pixels) image?
When image is displayed or saved at 3x scale, it is actually 4320 × 15360.
Is there a better way than trying to draw over an image?
autoreleasepool {
//activeEnergyCanvas is custom data processing class
let newActiveEnergySegment = activeEnergyCanvas.draw(in: CGRect(x: 0, y: 0, width: 1440, height: days * 10), with: energyPalette)
let size = CGSize(width: 1440, height: height)
UIGraphicsBeginImageContextWithOptions(size, false, 0.0)
//draw existing image
self.activeEnergyImage.draw(in: CGRect(origin: CGPoint(x: 0, y: 0),
size: size))
//calculate where to draw smaller image over larger one
let offsetRect = CGRect(origin: CGPoint(x: 0, y: offset * 10),
size: newActiveEnergySegment.size)
newActiveEnergySegment.draw(in: offsetRect)
//get the combined image
let newImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
//assign combined image to be displayed
if let unwrappedImage = newImage {
self.activeEnergyImage = unwrappedImage
}
}
Turns out my mistake was in passing invalid drawing scale (0.0) when creating graphics context, which defaulted to drawing at the device's native screen scale.
In case of iPhone 8 it was 3.0 The result is needing extreme amounts of memory to draw, zoom and export these images. Even if all debug logging prints that image is 1440 pixels wide, the actual canvas ends up being 1440 * 3.0 = 4320.
Passing 1.0 as the drawing scale makes the image more fuzzy, but reduces memory usage to less than 200mb.
// UIGraphicsBeginImageContext() <- also uses #3x scale, even when all display size printouts show
let drawingScale: CGFloat = 1.0
UIGraphicsBeginImageContextWithOptions(size, true, drawingScale)
In the fragment function of a Metal Shader file, is there a way to redefine the "bounds" of the texture with respect to what the sample will consider it's normalized coordinates to be?
By default, a value of 0,0 for the sample is the top-left "pixel" and 1,1 is the bottom right "pixel" of the texture. However, I'm re-using textures for drawing and at any given render pass there's only a portion of the texture that contains the relevant data.
For example, in a texture of width: 500 and height: 500, I might have only copied data into the region of 0,0,250,250. In my fragment function, I'd like the sampler to interpret a normalized coordinate of 1.0 to be 250 and not 500. Is that possible?
I realize I can just change the sampler to use pixel addressing, but that comes with a few restrictions as noted in the Metal Shader Specification.
No, but if you know the region you want to sample from, it's quite easy to do a little math in the shader to fix up your sampling coordinates. This is used often with texture atlases.
Suppose you have an image that's 500x500 and you want to sample the bottom-right 125x125 region (just to make things more interesting). You could pass this sampling region in as a float4, storing the bounds as (left, top, width, height) in the xyzw components. In this case, the bounds would be (375, 375, 125, 125). Your incoming texture coordinates are "normalized" with respect to this square. The shader simply scales and biases these coordinates into texel coordinates, then normalizes them to the dimensions of the whole texture:
fragment float4 fragment_main(FragmentParams in [[stage_in]],
texture2d<float, access::sample> tex2d [[texture(0)]],
sampler sampler2d [[sampler(0)]],
// ...
constant float4 &spriteBounds [[buffer(0)]])
{
// original coordinates, normalized with respect to subimage
float2 texCoords = in.texCoords;
// texture dimensions
float2 texSize = float2(tex2d.get_width(), tex2d.get_height());
// adjusted texture coordinates, normalized with respect to full texture
texCoords = (texCoords * spriteBounds.zw + spriteBounds.xy) / texSize;
// sample color at modified coordinates
float4 color = tex2d.sample(sampler2d, texCoords);
// ...
}
I am trying to read a 12-bit grayscale (DICOM:MONOCHROME2) image. I can read DICOM RGB files fine. When I attempt to load a grayscale image into NSBitmapImageRep, I get the following error message:
Inconsistent set of values to create NSBitmapImageRep
I have the following code fragment:
NSBitmapImageRep *rep = [[NSBitmapImageRep alloc]
initWithBitmapDataPlanes : nil
pixelsWide : width
pixelsHigh : height
bitsPerSample : bitsStored
samplesPerPixel : 1
hasAlpha : NO
isPlanar : NO
colorSpaceName : NSCalibratedWhiteColorSpace
bytesPerRow : width * bitsAllocated / 8
bitsPerPixel : bitsAllocated];
With these values:
width = 256
height = 256
bitsStored = 12
bitsAllocated = 16
Nothing seems inconsistent to me. I have verified that the image is: width*height*2 in length. So I am pretty sure that it is in a 2-byte grayscale format. I have tried many variations of the parameters, but nothing works. If I change "bitsPerSample" to 16, the error message goes away, but I get a solid black image. The closest success that I have been able to achieve, is to set "bitsPerPixel" to zero. When I do this, I successfully produce an image but it is clearly incorrectly rendered (you can barely make out the original image). Please some suggestions!! I have tried a long time to get this to work and have checked the Stack overflow and the web (many times). Thanks very much for any help!
SOLUTION:
After the very helpful suggestions from LEADTOOLS Support, I was able to solve my problem. Here is the code fragment that works (assuming a MONOCHROME2 DICOM image):
// If, and only if, MONOCHROME2:
NSBitmapImageRep *imageRep = [[NSBitmapImageRep alloc]
initWithBitmapDataPlanes : &pixelData
pixelsWide : width
pixelsHigh : height
bitsPerSample : bitsAllocated /*bitsStored-this will not work*/
samplesPerPixel : samplesPerPixel
hasAlpha : NO
isPlanar : NO
colorSpaceName : NSCalibratedWhiteColorSpace
bytesPerRow : width * bitsAllocated / 8
bitsPerPixel : bitsAllocated];
int scale = USHRT_MAX / largestImagePixelValue;
uint16_t *ptr = (uint16_t *)imageRep.bitmapData;
for (int i = 0; i < width * height; i++) *ptr++ *= scale;
It is important to know about the Transfer Syntax (0002:0010) and Number of frames in the dataset. Also, try to get the value length and VR for Pixel Data (7FE0:0010) element. Using value length of the pixel data element you will be able to validate your calculation for uncompressed image.
As for displaying the image, you will also need the value for High Bit (0028:0102) and Pixel Representation (0028:0103). An image could be 16-bit allocated, 12-bit stored, high bit set to 15 and have one sample per pixel. That means 4 lest significant bits of each word do not contain pixel data. Pixel Representation when set to 1 means sign bit is the high bit in pixel sample.
In addition, you many need to apply modality LUT transformation (rescale slope and rescale intercept for linear transformation) when present in the dataset to prepare the data for display. At the end, you apply the VOI LUT transformation (Window center and Window Width) to display the image.
Can you explain to me difference between kCGImageAlphaPremultipliedFirst and kCGImageAlphaFirst?
What's mean Premultiplied in practice?
In short, premultiplied means that the alpha value will also affect the color component values of the pixels when a pixel which is not opaque is represented.
From the Quartz 2D drawing guide:
For bitmaps that have an alpha component, whether the color components
are already multiplied by the alpha value. Premultiplied alpha
describes a source color whose components are already multiplied by an
alpha value. Premultiplying speeds up the rendering of an image by
eliminating an extra multiplication operation per color component. For
example, in an RGB color space, rendering an image with premultiplied
alpha eliminates three multiplication operations (red times alpha,
green times alpha, and blue times alpha) for each pixel in the image.
BTW, Pre-Multiplied is likely what the APIs will force you to use because that is Quartz's preference. Fortunately, the conversions aren't terrible (lossy OTOH…).
the shortest way to explain this is in float components, using the range [0...1].
If our RGBA input representation is:
typedef struct t_rgba { float r,g,b,a; } t_rgba;
const t_rgba rgba = { 0.5, 0.5, 0.5, 0.5 };
Then to pre-multiply it:
t_rgba rgba_PreMul = rgba;
rgba_PreMul.r *= rgba_PreMul.a;
rgba_PreMul.g *= rgba_PreMul.a;
rgba_PreMul.b *= rgba_PreMul.a;
Then to de-pre-multiply it:
t_rgba rgba_DePreMul = rgba_PreMul;
if (0.0 < rgba_DePreMul.a && 1.0 > rgba_DePreMul.a) {
const float ialpha = 1.0/rgba_DePreMul.a;
rgba_DePreMul.r *= ialpha;
rgba_DePreMul.g *= ialpha;
rgba_DePreMul.b *= ialpha;
}
You might want some saturation in there, too.
Now that's the basic form, which can be repurposed to other numeric representations. Note that these conversions are lossy. As well, be careful not to pass premultiplied bitmaps where regular bitmaps are expected, and vice versa.