Can you change the bounds of a Sampler in a Metal Shader? - objective-c

In the fragment function of a Metal Shader file, is there a way to redefine the "bounds" of the texture with respect to what the sample will consider it's normalized coordinates to be?
By default, a value of 0,0 for the sample is the top-left "pixel" and 1,1 is the bottom right "pixel" of the texture. However, I'm re-using textures for drawing and at any given render pass there's only a portion of the texture that contains the relevant data.
For example, in a texture of width: 500 and height: 500, I might have only copied data into the region of 0,0,250,250. In my fragment function, I'd like the sampler to interpret a normalized coordinate of 1.0 to be 250 and not 500. Is that possible?
I realize I can just change the sampler to use pixel addressing, but that comes with a few restrictions as noted in the Metal Shader Specification.

No, but if you know the region you want to sample from, it's quite easy to do a little math in the shader to fix up your sampling coordinates. This is used often with texture atlases.
Suppose you have an image that's 500x500 and you want to sample the bottom-right 125x125 region (just to make things more interesting). You could pass this sampling region in as a float4, storing the bounds as (left, top, width, height) in the xyzw components. In this case, the bounds would be (375, 375, 125, 125). Your incoming texture coordinates are "normalized" with respect to this square. The shader simply scales and biases these coordinates into texel coordinates, then normalizes them to the dimensions of the whole texture:
fragment float4 fragment_main(FragmentParams in [[stage_in]],
texture2d<float, access::sample> tex2d [[texture(0)]],
sampler sampler2d [[sampler(0)]],
// ...
constant float4 &spriteBounds [[buffer(0)]])
{
// original coordinates, normalized with respect to subimage
float2 texCoords = in.texCoords;
// texture dimensions
float2 texSize = float2(tex2d.get_width(), tex2d.get_height());
// adjusted texture coordinates, normalized with respect to full texture
texCoords = (texCoords * spriteBounds.zw + spriteBounds.xy) / texSize;
// sample color at modified coordinates
float4 color = tex2d.sample(sampler2d, texCoords);
// ...
}

Related

How to map hdr file image onto cubemap with Vulkan?

Curently based on Sascha Willems examples I've cerated samplerCube texture for fragment shader.
It has same JPG image copied to all 6 layers (faces).
I use stbi image library for image loading, it works okay if I use it for regular 2D texture, but if it's mapped on cube mesh it creates distorted image:
int width = 0, height = 0, channel = 0;
float* pixels = stbi_loadf("textures/test.hdr", &width, &height, &channel, STBI_rgb_alpha);
if(!pixels) throw std::runtime_error("failed to load texture image!");
this->texture_image.create_image(width, height, VK_FORMAT_R32G32B32A32_SFLOAT, VK_IMAGE_USAGE_TRANSFER_DST_BIT | VK_IMAGE_USAGE_SAMPLED_BIT);
this->texture_image.fill_memory(width, height, 4*sizeof(float), pixels);
this->texture_image.create_image_view(VK_FORMAT_R32G32B32A32_SFLOAT, VK_IMAGE_ASPECT_COLOR_BIT);
stbi_image_free(pixels);
Found how to do it here: https://learnopengl.com/PBR/IBL/Diffuse-irradiance
Even if it's in OpenGL the concenpt is the same.

CGImageCreate with CGColorSpaceCreateDeviceGray on iOS12

I was using CGImageCreate with CGColorSpaceCreateDeviceGray to convert a buffer (CVPixelBufferRef) to grayscale image. It was very fast and did work well until iOS 12... now the returned image is empty.
The code look like this:
bitmapInfo = kCGBitmapByteOrder32Little | kCGImageAlphaNoneSkipFirst;
CGDataProviderRef provider = CGDataProviderCreateWithData((void *)i_PixelBuffer,
sourceBaseAddr,
sourceRowBytes * height,
ReleaseCVPixelBuffer);
retImage = CGImageCreate(width,
height,
8,
32,
sourceRowBytes,
CGColorSpaceCreateDeviceGray(),
bitmapInfo,
provider,
NULL,
true,
kCGRenderingIntentDefault);
CGDataProviderRelease(provider);
This is a known bug in iOS 12? If device gray is no supported anymore in this function, can you suggest me another way to do it?
Note that conversion should take less than 0.1 seconds for a 4K image.
Thanks in advance!
According to the list of Supported Pixel Formats in the Quartz 2D Programming Guide, iOS doesn't support 32 bits per pixel with gray color spaces. And even on macOS, 32 bpp gray requires the use of kCGBitmapFloatComponents (and float data).
Is your data really 32 bpp? If so, is it float? What are you using for bitmapInfo?
I would not expect CGImageCreate() to "convert" a buffer, including to grayscale. The parameters you're supplying are telling it how to interpret the data. If you're not using floating-point components, I suspect it was just taking one of the color channels and interpreting that as the gray level and ignoring the other components. So, it wasn't a proper grayscale conversion.
Apple's advice is to create an image that properly represents the image; create a bitmap context with the colorspace, pixel layout, and bitmap info you desire; draw the former into the latter; and create the final image from the context.
I finally found a workaround for my purpose. Note that the CVPixelBuffer is coming from the video camera.
Changed camera output pixel format to
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
(AVCaptureVideoDataOutput)
Extract the Y plane from YpCbCr
Build a CGImage with the Y plane
Code:
// some code
colorSpace = CGColorSpaceCreateDeviceGray();
sourceRowBytes = CVPixelBufferGetBytesPerRowOfPlane(i_PixelBuffer, 0);
sourceBaseAddr = (unsigned char*)CVPixelBufferGetBaseAddressOfPlane(i_PixelBuffer,0);
bitmapInfo = kCGImageByteOrderDefault;
// some code
CGContextRef context = CGBitmapContextCreate(sourceBaseAddr,
width,
height,
8,
sourceRowBytes,
colorSpace,
bitmapInfo);
retImage = CGBitmapContextCreateImage(context);
// some code
You can also look at this related post:
420YpCbCr8BiPlanarVideoRange To YUV420 ?/How to copy Y and Cbcr plane to Single plane?

how to plot rgb color histogram of image with objective c

i want to show image RGB colour histogram in cocoa application. Please suggest possible way to do it with objective c or any third party library available to achieve this.
well this is a problem as RGB colors are 3D space so their histogram would lead to 4D plot which is something we do not really comprehend.
So the solution to this is to convert the 4D plot to 3D plot somehow. This can be done by sorting the colors by something that has some meaning. I will not speculate and describe what I am using. I use HSV color space and ignore the V value. This way I lose a lot of color shade info but it is still enough to describe colors for my purposes. This is how it looks like:
You can also use more plots with different V to cover more colors. For more info see:
HSV histogram
Anyway you can use any gradient sorting or any shape of your plot that is completely on you.
If you want pure RGB then you could adapt this and use RGB cube surface or map it on sphere and ignore the length from (0,0,0) (use unit vectors) something like this:
So if you R,G,B are in <0,1> you convert that to <-1,+1> then compute the spherical coordinates (ignoring radius) and you got your 2 variables instead of 3 which you can use as a plot (either as 2D globe base or 3D sphere ...).
Here C++ code how to do this (made from the HSV histogram):
picture pic0,pic1,pic2,zed;
const int na=360,nb=180,nb2=nb>>1; // size of histogram table
int his[na][nb];
DWORD w;
int a,b,r,g,x,y,z,l,i,n;
double aa,bb,da,db,dx,dy,dz,rr;
color c;
pic2=pic0; // copy input image pic0 to pic2
for (a=0;a<na;a++) // clear histogram
for (b=0;b<nb;b++)
his[a][b]=0;
for (y=0;y<pic2.ys;y++) // compute it
for (x=0;x<pic2.xs;x++)
{
c=pic2.p[y][x];
r=c.db[picture::_r]-128;
g=c.db[picture::_g]-128;
b=c.db[picture::_b]-128;
l=sqrt(r*r+g*g+b*b); // convert RGB -> spherical a,b angles
if (!l) { a=0; b=0; }
else{
a=double(double(na)*acos(double(b)/double(l))/(2.0*M_PI));
if (!r) b=0; else b=double(double(nb)*atan(double(g)/double(r))/(M_PI)); b+=nb2;
while (a<0) a+=na; while (a>=na) a-=na;
if (b<0) b=0; if (b>=nb) b=nb-1;
}
his[a][b]++; // update color usage count ...
}
for (n=0,a=0;a<na;a++) // max probability
for (b=0;b<nb;b++)
if (n<his[a][b]) n=his[a][b];
// draw the colored RGB sphere and histogram
zed =pic1; zed .clear(9999); // zed buffer for 3D
pic1.clear(0); // image of histogram
da=2.0*M_PI/double(na);
db=M_PI/double(nb);
for (aa=0.0,a=0;a<na;a++,aa+=da)
for (bb=-M_PI,b=0;b<nb;b++,bb+=db)
{
// normal
dx=cos(bb)*cos(aa);
dy=cos(bb)*sin(aa);
dz=sin(bb);
// color of surface (darker)
rr=75.0;
c.db[picture::_r]=double(rr*dx)+128;
c.db[picture::_g]=double(rr*dy)+128;
c.db[picture::_b]=double(rr*dz)+128;
c.db[picture::_a]=0;
// histogram center
x=pic1.xs>>1;
y=pic1.ys>>1;
// surface position
rr=64.0;
z=rr;
x+=double(rr*dx);
y+=double(rr*dy);
z+=double(rr*dz);
if (zed.p[y][x].dd>=z){ pic1.p[y][x]=c; zed.p[y][x].dd=z; }
// ignore lines if zero color count
if (!his[a][b]) continue;
// color of lines (bright)
rr=125.0;
c.db[picture::_r]=double(rr*dx)+128;
c.db[picture::_g]=double(rr*dy)+128;
c.db[picture::_b]=double(rr*dz)+128;
c.db[picture::_a]=0;
// line length
l=(xs*his[a][b])/(n*3);
for (double xx=x,yy=y,zz=z;l>=0;l--)
{
if (zed.p[y][x].dd>=z){ pic1.p[y][x]=c; zed.p[y][x].dd=z; }
xx+=dx; yy+=dy; zz+=dz; x=xx; y=yy; z=zz;
if (x<0) break; if (x>=xs) break;
if (y<0) break; if (y>=ys) break;
}
}
input image is pic0, output image is pic1 (histogram graph)
pic2 is copy of pic0 (remnant of old code)
zed is the Zed buffer for 3D display avoiding Z sorting ...
I use my own picture class for images so some members are:
xs,ys size of image in pixels
p[y][x].dd is pixel at (x,y) position as 32 bit integer type
clear(color) - clears entire image
resize(xs,ys) - resizes image to new resolution
As the sphere is a 3D object you should add rotation to it so all the surface is visible in time (or rotate with mouse or whatever) ...

kCGImageAlphaPremultipliedFirst and kCGImageAlphaFirst

Can you explain to me difference between kCGImageAlphaPremultipliedFirst and kCGImageAlphaFirst?
What's mean Premultiplied in practice?
In short, premultiplied means that the alpha value will also affect the color component values of the pixels when a pixel which is not opaque is represented.
From the Quartz 2D drawing guide:
For bitmaps that have an alpha component, whether the color components
are already multiplied by the alpha value. Premultiplied alpha
describes a source color whose components are already multiplied by an
alpha value. Premultiplying speeds up the rendering of an image by
eliminating an extra multiplication operation per color component. For
example, in an RGB color space, rendering an image with premultiplied
alpha eliminates three multiplication operations (red times alpha,
green times alpha, and blue times alpha) for each pixel in the image.
BTW, Pre-Multiplied is likely what the APIs will force you to use because that is Quartz's preference. Fortunately, the conversions aren't terrible (lossy OTOH…).
the shortest way to explain this is in float components, using the range [0...1].
If our RGBA input representation is:
typedef struct t_rgba { float r,g,b,a; } t_rgba;
const t_rgba rgba = { 0.5, 0.5, 0.5, 0.5 };
Then to pre-multiply it:
t_rgba rgba_PreMul = rgba;
rgba_PreMul.r *= rgba_PreMul.a;
rgba_PreMul.g *= rgba_PreMul.a;
rgba_PreMul.b *= rgba_PreMul.a;
Then to de-pre-multiply it:
t_rgba rgba_DePreMul = rgba_PreMul;
if (0.0 < rgba_DePreMul.a && 1.0 > rgba_DePreMul.a) {
const float ialpha = 1.0/rgba_DePreMul.a;
rgba_DePreMul.r *= ialpha;
rgba_DePreMul.g *= ialpha;
rgba_DePreMul.b *= ialpha;
}
You might want some saturation in there, too.
Now that's the basic form, which can be repurposed to other numeric representations. Note that these conversions are lossy. As well, be careful not to pass premultiplied bitmaps where regular bitmaps are expected, and vice versa.

Is there a way to set alpha channel color for sampler2D texture?

I am working on an app that uses OpenGL-ES to blend two textures. For overlay image, I want to specify a certain color as the alpha channel e.g.) green. How could this be accomplished? I tried using glBlendFunc without much success. Any help would be greatly appreciated!
there is no such feature in OpenGL itself, but maybe you could achieve that in shader:
uniform vec3 colorMask; // eg. green
varying vec2 texCoord;
uniform sampler2D overlay;
void main()
{
vec4 texture = texture2D(overlay, texCoord);
// get RGB of the texture
vec3 absdiff = abs(texture.rgb - colorMask);
float alpha = all(lessThan(absdiff, vec3(0.1)))? 1.0 : 0.0;
// calculate alpha using vector relational functions
texture.a = alpha; // use alpha
gl_FragColor = texture;
// write final color, can use blending
}
This works by calculating absolute difference of texture color and masking color, and comparing it to 0.1. It is pretty simple, but it might be slow (i'm writing it from memory, you have to test it).
Or you can use a different way of calculating alpha:
uniform vec3 colorMask; // eg. green
varying vec2 texCoord;
uniform sampler2D overlay;
void main()
{
vec4 texture = texture2D(overlay, texCoord);
// get RGB of the texture
vec3 diff = texture.rgb - colorMask;
float alpha = step(dot(diff, diff), 0.1);
// calculate alpha using difference squared and a step function
texture.a = alpha; // use alpha
gl_FragColor = texture;
// write final color, can use blending
}
This is using the square error metric to calculate alpha. It calculates distance of colors in RGB space, calculates it's square, and compares that to 0.1. This might be a little bit harder to tweak the threshold (0.1) but enables you to use soft threshold. If you were to have color gradient and wanted some colors to be more transparent than other, you can throw away step function and use smoothstep instead.