I didn't find in Vulkan spec anything to support or against:
Is it possible to render to a YUV420p format render target texture ?
And if so, does vulkan convert from RGB given in the fragment shader output to YUV of the render target?
Related
In a Vulkan program, fragment shaders generally output single-precision floating-point colors in the range 0.0 to 1.0 to each red/blue/green channel, and these are then written to (blended into) the swapchain image that is then presented to screen. The floating point values are encoded into bits according to the format of the swapchain image (specified when the swapchain is created).
When I change my swapchain format from VK_FORMAT_B8G8R8A8_UNORM to VK_FORMAT_B8G8R8A8_SRGB I observe that the overall brightness of the frames is greatly increased, and also there are some minor color shifts.
My understanding of the SRGB format was that it was a lot like the UNORM format just having a different mapping of floating point values to 8-bit integers, such that it had higher color resolution in some areas and less in others, but the actually meaning of the "pre-encoded" RGB floating-point values remained unchanged.
So I'm a little suprised about the brightness increase. Is my understanding of SRGB encoding wrong? and/or is such a brightness increase is expected vs UNORM?
or maybe I have a bug and a brightness increase is not expected?
Update:
I've observed that if I use SRGB swapchain images and also load my images/textures in VK_FORMAT_B8G8R8A8_SRGB format rather than VK_FORMAT_B8G8R8A8_UNORM then the extra brightness goes away. It looks the same as if I use VK_FORMAT_B8G8R8A8_UNORM swapchain images and load my images/textures in VK_FORMAT_B8G8R8A8_UNORM format.
Also, if I put the swapchain image into VK_FORMAT_B8G8R8A8_UNORM format and then load the images/textures with VK_FORMAT_B8G8R8A8_SRGB, the frames look extra dark / almost black.
Some clarity about what is going on would be helpful.
This is a colorspace and display issue.
Fragment shaders are assumed to be writing values in a linear RGB colorspace. As such, if you are rendering to an image that has a linear RGB colorspace (UNORM), the values your FS produces are interpreted directly. When you render to an image which has an sRGB colorspace, you are writing values from one space (linear) into another space (sRGB). As such, these values are automatically converted into the sRGB colorspace. It's no different from transforming a position from model space to world space or whatever.
What is different is the fact that you've been looking at your scene incorrectly. See, odds are very good that your swapchain's VkSurfaceFormat::colorSpace value is VK_COLOR_SPACE_SRGB_NONLINEAR_KHR.
VkSurfaceFormat::colorspace tells you how the display engine will interpret the pixel data in swapchain images you present. This is not a setting you provide; this is the display engine telling you something about how it is going to interpret the values you send it.
I say "odds are very good" that it is sRGB because, outside of extensions, this is the only possible value. You are rendering to an sRGB display device whether you like it or not.
So if you write to a UNORM image, the display device will read the actual bits of data and interpret them as if they are in the sRGB colorspace. This operation only makes sense if the data your fragment shader wrote itself is in the sRGB colorspace.
However, that's generally not how FS's generate data. The lighting computations you compute only make sense on color values in a linear RGB colorspace. So unless you wrote your FS to deliberately do sRGB conversion after computing the final color value, odds are good that all of your results have been in a linear RGB colorspace. And that's what you've been writing to your framebuffer.
And then the display engine mangles it.
By using an sRGB image as your destination, you force a colorspace conversion from linear RGB to sRGB, which will then be interpreted by the display engine as sRGB values. This means that your lighting equations are finally producing the correct results.
Failure to do gamma-correct rendering properly (including the source texture images which are almost certainly also in the sRGB colorspace, as this is the default colorspace for most image editors. The exceptions would be for things like gloss-maps, normal maps, or other images that aren't storing "colors".) leads to cases where linear light attention appears more correct than quadratic attenuation, even though quadratic is how reality works.
This is gamma correction.
Using a swapchain with VK_FORMAT_B8G8R8A8_SRGB leverages the ability to to apply gamma correction as the final step in your render pipeline. This happens for you automatically behind the scenes.
That is the only place you want gamma correction to happen. Make sure your shaders are not applying gamma correction. You might see it as:
color = pow(color, vec3(1.0/2.2));
If your swapchain does the gamma correction, you do not need todo it in your shaders.
Most images are SRGB (pictures, color textures, etc). Linear images are for specific data, like a blue noise texture or heightmap.
Load SRGB images w/ VK_FORMAT_R8G8B8A8_SRGB
Load LINEAR images w/ VK_FORMAT_R8G8B8A8_UNORM
No shader conversion is required if the rules outlined above are followed.
I have some legacy DX11 code that renders to multiple 3d render targets. Destination target is passed via SV_TARGETxx and the slice is set via SV_RenderTargetArrayIndex in GS. Is there any way to do the same in Vulkan?
My plan is to create individual view for each slice of each 3d target and pass them all together as attachments to a single frame buffer, then in GS I can have something like gl_Layer = sliceNo + targetOffsets[xx]. Is there any better solution?
In Vulkan, the GS SV_RenderTargetArrayIndex is called Layer in SPIR-V or gl_Layer in GLSL. It behaves the same as in D3D. You create one view per 3D target, and attach that to the framebuffer. The Layer output from the GS will say which layer (of all the targets) the output primitive is drawn to.
In Vulkan there's no "true" 3D framebuffer attachments, in the sense that after projection to screen space coordinates everything exists in a 2D plane. So attachment image views can have 2D_ARRAY dimensionality, but not 3D. The Image and image view parameter compatibility requirements table says that given a 3D image, you can create a 2D_ARRAY image view with layerCount >= 1. Note that you have to create the image with the VK_IMAGE_CREATE_2D_ARRAY_COMPATIBLE_BIT flag.
So if you want to have N 3D render target images:
Create your N 3D images, with the VK_IMAGE_CREATE_2D_ARRAY_COMPATIBLE_BIT flag.
Create one image view for each image, with VK_IMAGE_VIEW_TYPE_2D_ARRAY and layerCount equal to the number of slices you want to be able to render to.
Create a VkRenderPass with one VkAttachmentDescription per 3D render target, plus whatever others you need for depth/stencil, resolve target, etc.
Create a VkFrameBuffer based on that VkRenderPass, and pass your image views in the VkFrameBufferCreateInfo::pAttachments array. Set VkFramebufferCreateInfo::layerCount to the number of layers/slices you want to be able to render to.
[Edit: Below paragraph can be ignored based on first comment. Leaving it for transparency.]
I'm confused what you're trying to do with SV_Target[n]. In both D3D and Vulkan, if you've got multiple render targets / color attachments, the fragment shader will write to all of them -- if your fragment shader doesn't provide a value for a bound target, the value written is undefined. So SV_Target[n] is used to tell which shader output variables go to which target, but they don't let you write to some without writing to others. Vulkan works similarly, using output variables gl_FragData[n] in GLSL.
If you're talking about having 1 draw call rendered from multiple points of view (but otherwise using the same pipeline) then you want VK_KHR_multiview. This is an extension in Vulkan 1.0, but core in 1.1.
There's an example of it's usage here and the corresponding shader functionality is here. It functions similar to what you seem to describe. You attach multiple images from a texture array to a single framebuffer ("rendertarget" in D3D) and then in the vertex shader you can determine which layer you're rendering to via the gl_ViewIndex variable. There's no need for a geometry shader with this approach.
From OpenGL ES 2.0 specification section 4.4.5:
"Formats not listed in table 4.5, including compressed internal formats. are not
color-, depth-, or stencil-renderable, no matter which components they contain."
Then there are extensions that extend this table such as:
OES_depth24
OES_depth32
OES_rgb8_rgba8
ARM_rgba8
If I understood the specification correctly, table 4.5 affects both texture and renderbuffer formats. And in that case, for example, RGB and RGBA textures with 8-bits per component are not color-renderable unless the extension OES_rgb8_rgba8 (or ARM_rgba8 for RGBA) is supported.
On a test device that supports OES_rgb8_rgba8 the following texture formats are valid, i.e., framebuffer complete when attached to FBO as the color attachment:
RGB 565
RGB 888
RGBA 4444
RGBA 5551
RGBA 8888
And these were not:
Alpha 8
Luminance 8
LuminanceAlpha 88
The results match my assumptions (at least on 1 device) but I would like to know if I understood the specification correctly or is this working by accident?
Yes, your assumptions are correct.
The list of renderable formats in the official specification are just what an OpenGL ES 2.0 implementation is required to support. Most support many more than what is listed there.
However, no implementation of OpenGL / OpenGL ES supports alpha or luminance textures as color-renderable. You can replicate the behavior with texture swizzle extensions and/or GLSL vector swizzling. The extension: EXT_texture_rg adds Red and Red/Green image formats that are useable by textures and renderbuffers. These two formats are very useful when you want to draw into a one or two-channel image format using an FBO (since GL_LUMINANCE, GL_ALPHA and GL_LUMINANCE_ALPHA are not color-renderable formats).
Generally speaking, the set of renderbuffer image formats is a subset of texture image formats. You can use compressed image formats (optional in ES 2.0), luminance and alpha image formats for texturing and pixel transfer operations but they are not supported by renderbuffers. This means that while you can draw using these textures, you cannot draw into them by attaching them to an FBO. To draw into a texture, it must have an image format that is both a renderbuffer format and texture format.
On a historical note, there are renderbuffer formats that are/were not usable as textures. Multisample formats are one example, multisample texturing was added after the initial FBO specification was created.
I have RGBA values for a frame in a buffer. How do I draw the frame on the screen using opengl? Which function do I have to use? I'm using opengl-es 2.0.
gldrawpixels() is not available in iOS glKit.
Use texture. glTexImage2D is probably available.
Then use this texture on full-screen quad.
That will involve some shader coding, when you could additionally process your colour data.
Using WebGL (which is constrained to the OpenGL ES 2 API), I am successfully rendering to texture and then displaying that texture onscreen. Because it is a texture, it is not being antialiased. If I were rendering to an RBO and then displaying that onscreen, I would be able to take advantage of AA.
My render target setup looks like this:
Create FBO
Bind FBO
Create texture (to be rendered to)
Create and bind depth buffer as RBO
Attach texture and RBO to FBO
And my rendering update loop looks like this:
Render the scene to the FBO created in step #2 above
Render a screen aligned quad with the texture created in step #3 above
With desktop OpenGL, I would call glBlitFramebuffer() instead of drawing the screen aligned quad.
How do I render my scene with antialiasing? Do I need to replace the texture with an RBO? If so, what calls do I use to bind the RBO to draw a screen-aligned quad?
You cannot blit the contents of an RBO to screen in WebGL unless you perform a readback and re-upload to texture to blit, which is rather slow.
WebGL has no support for MSAA on FBOs in any form (neither as RBO nor as RTT).
You can implement your own antialiasing in a variety of ways.
Render at 2:2 size and scale down (google maps with webgl does this)
Render at 1:1 size, run a sobel or laplace edge detection on color and depth, and run a bilateral gaussian blur using edge strength as weight (I've used this technique in some of my demos, it works well, http://codeflow.org/entries/2011/apr/11/advanced-webgl-part-1/ )
Use the morphological antialiasing recipe from GPU Pro 2 (I've yet to try that)