Passing a large array from vertex to fragment shader in glsl v100 - opengl-es-2.0

(OpenGL ES 2.0) (glsl v100)
I am trying to pass a large kernel to the fragment shader for a convolution operation. I started by creating the shader dynamically and hardcoding all the kernel colors, but for some reason if I read the array past index 31 the shader does not compile and no error is given. Below is some code i ended up with while trying to figure where the problem is.
vertex shader
attribute vec4 position;
attribute vec4 inputTextureCoordinate;
varying float kernelColors[33];
void main()
{
gl_Position = position;
kernelColors[0] = 0.000000;
kernelColors[1] = 1.000000;
kernelColors[2] = 1.000000;
kernelColors[3] = 1.000000;
kernelColors[4] = 1.000000;
kernelColors[5] = 1.000000;
kernelColors[6] = 1.000000;
kernelColors[7] = 1.000000;
kernelColors[8] = 1.000000;
kernelColors[9] = 1.000000;
kernelColors[10] = 1.000000;
kernelColors[11] = 1.000000;
kernelColors[12] = 1.000000;
kernelColors[13] = 1.000000;
kernelColors[14] = 1.000000;
kernelColors[15] = 1.000000;
kernelColors[16] = 1.000000;
kernelColors[17] = 1.000000;
kernelColors[18] = 1.000000;
kernelColors[19] = 1.000000;
kernelColors[20] = 1.000000;
kernelColors[21] = 1.000000;
kernelColors[22] = 1.000000;
kernelColors[23] = 1.000000;
kernelColors[24] = 1.000000;
kernelColors[25] = 1.000000;
kernelColors[26] = 1.000000;
kernelColors[27] = 1.000000;
kernelColors[28] = 1.000000;
kernelColors[29] = 1.000000;
kernelColors[30] = 1.000000;
kernelColors[31] = 1.000000;
kernelColors[32] = 1.000000;
}
fragment shader
precision highp float;
uniform sampler2D inputImageTexture;
varying float kernelColors[33];
void main()
{
kernelColors[0];
kernelColors[1];
kernelColors[2];
kernelColors[3];
kernelColors[4];
kernelColors[5];
kernelColors[6];
kernelColors[7];
kernelColors[8];
kernelColors[9];
kernelColors[10];
kernelColors[11];
kernelColors[12];
kernelColors[13];
kernelColors[14];
kernelColors[15];
kernelColors[16];
kernelColors[17];
kernelColors[18];
kernelColors[19];
kernelColors[20];
kernelColors[21];
kernelColors[22];
kernelColors[23];
kernelColors[24];
kernelColors[25];
kernelColors[26];
kernelColors[27];
kernelColors[28];
kernelColors[29];
kernelColors[30];
kernelColors[31];
kernelColors[32]; //if i comment this line out it works
gl_FragColor = vec4(vec3(.3,.5,.1), 1.0);
}
There is probably a better way to do this, but i still would like to know why the code above does not compile.
Regarding other methods - the 'flat' qualifier is not available in glsl 100, so I can't apply that if that would even change/optimize anything. Also if I load in the kernel data as a texture how can I iterate over the pixels if I can't have a dynamic index?
I need to convolve each pixel with the kernel, the kernel does not change from frame to frame but I may need to swap it out for another kernel in some cases.

There is a limit on the amount of varying you can use depending on the device you run on (search about MAX_VARYING_VECTORS and OpenGLES), if you exceed it shader will not compile.
But using varyings to store a kernel which is the same for each pixel is a complete waste of resources. Remember that varyings are set for vertices in vertex shader and then GPU interpolates between them for each pixel, so here you are telling the GPU to interpolate values which are the same for all fragments, this is a totally useless operation.
Better solutions :
Use uniforms in the fragment shader to store your kernel (like that you can also change it from your app code ...)
Store your kernel in a texture which you sample in the fragment shader, in this case make sure that the texture uses nearest filtering (same, you can change the texture from app code. whenever you need...)

Related

Skeletal animation bug with Assimp in DirectX 12

I am using Assimp to load an FBX model with animation (created in Blender) into my DirectX 12 game, but I'm experiencing a very frustrating bug with the animation rendered by the game application.
The test model is a simple 'flagpole' containing four bones like so:
Bone0 -> Bone1 -> Bone2 -> Bone3
The model renders correctly in its rest pose when the keyframe animation is bypassed.
The model also renders and animates properly when the animation rotates the model only by the root bone (Bone0).
However, when importing a model that rotates at the first joint (i.e. at Bone1), the vertices clustered around each joint seem 'stuck' in their original positions, while the vertices surrounding the 'bones' proper appear to follow through with the correct animation.
The result is a crappy zigzag of stretched geometry like so:
Instead the model should resemble an 'allen-key' shape at the end of its animation pose, as shown by the same model rendered in the AssimpViewer utility tool:
Since the model is rendering correctly in AssimpViewer, it's reasonable to assume there are no issues with the FBX file exported by Blender. I then checked and confirmed that the vertices 'stuck' around the joints did indeed have their vertex weights correctly assigned by the game loading code.
The C++ model loading and animation code is based on the popular OGLDev tutorial: https://ogldev.org/www/tutorial38/tutorial38.html
Now the infuriating thing is, since the AssimpViewer tool was correctly rendering the model animation, I also copied in the SceneAnimator and AnimEvaluator classes from that tool to generate the final bone transforms via that code branch as well... only to end up with exactly the same zigzag bug in the game!
I'm reasonably confident there aren't any issues with finding the bone hierarchy structure at initialization, so here are the key functions that traverse the hierarchy and interpolate key frames each frame.
VOID Mesh::ReadNodeHeirarchy(FLOAT animationTime, CONST aiNode* pNode, CONST aiAnimation* pAnim, CONST aiMatrix4x4 parentTransform)
{
std::string nodeName(pNode->mName.data);
// nodeTransform is a relative transform to parent node space
aiMatrix4x4 nodeTransform = pNode->mTransformation;
CONST aiNodeAnim* pNodeAnim = FindNodeAnim(pAnim, nodeName);
if (pNodeAnim)
{
// Interpolate scaling and generate scaling transformation matrix
aiVector3D scaling(1.f, 1.f, 1.f);
CalcInterpolatedScaling(scaling, animationTime, pNodeAnim);
// Interpolate rotation and generate rotation transformation matrix
aiQuaternion rotationQ (1.f, 0.f, 0.f, 0.f);
CalcInterpolatedRotation(rotationQ, animationTime, pNodeAnim);
// Interpolate translation and generate translation transformation matrix
aiVector3D translat(0.f, 0.f, 0.f);
CalcInterpolatedPosition(translat, animationTime, pNodeAnim);
// build the SRT transform matrix
nodeTransform = aiMatrix4x4(rotationQ.GetMatrix());
nodeTransform.a1 *= scaling.x; nodeTransform.b1 *= scaling.x; nodeTransform.c1 *= scaling.x;
nodeTransform.a2 *= scaling.y; nodeTransform.b2 *= scaling.y; nodeTransform.c2 *= scaling.y;
nodeTransform.a3 *= scaling.z; nodeTransform.b3 *= scaling.z; nodeTransform.c3 *= scaling.z;
nodeTransform.a4 = translat.x; nodeTransform.b4 = translat.y; nodeTransform.c4 = translat.z;
}
aiMatrix4x4 globalTransform = parentTransform * nodeTransform;
if (m_boneMapping.find(nodeName) != m_boneMapping.end())
{
UINT boneIndex = m_boneMapping[nodeName];
// the global inverse transform returns us to mesh space!!!
m_boneInfo[boneIndex].FinalTransform = m_globalInverseTransform * globalTransform * m_boneInfo[boneIndex].BoneOffset;
//m_boneInfo[boneIndex].FinalTransform = m_boneInfo[boneIndex].BoneOffset * globalTransform * m_globalInverseTransform;
m_shaderTransforms[boneIndex] = aiMatrixToSimpleMatrix(m_boneInfo[boneIndex].FinalTransform);
}
for (UINT i = 0u; i < pNode->mNumChildren; i++)
{
ReadNodeHeirarchy(animationTime, pNode->mChildren[i], pAnim, globalTransform);
}
}
VOID Mesh::CalcInterpolatedRotation(aiQuaternion& out, FLOAT animationTime, CONST aiNodeAnim* pNodeAnim)
{
UINT rotationKeys = pNodeAnim->mNumRotationKeys;
// we need at least two values to interpolate...
if (rotationKeys == 1u)
{
CONST aiQuaternion& key = pNodeAnim->mRotationKeys[0u].mValue;
out = key;
return;
}
UINT rotationIndex = FindRotation(animationTime, pNodeAnim);
UINT nextRotationIndex = (rotationIndex + 1u) % rotationKeys;
assert(nextRotationIndex < rotationKeys);
CONST aiQuatKey& key = pNodeAnim->mRotationKeys[rotationIndex];
CONST aiQuatKey& nextKey = pNodeAnim->mRotationKeys[nextRotationIndex];
FLOAT deltaTime = FLOAT(nextKey.mTime) - FLOAT(key.mTime);
FLOAT factor = (animationTime - FLOAT(key.mTime)) / deltaTime;
assert(factor >= 0.f && factor <= 1.f);
aiQuaternion::Interpolate(out, key.mValue, nextKey.mValue, factor);
}
I've just included the rotation interpolation here, since the scaling and translation functions are identical. For those unaware, Assimp's aiMatrix4x4 type follows a column-vector math convention, so I haven't messed with original matrix multiplication order.
About the only deviation between my code and the two Assimp-based code branches I've adopted is the requirement to convert the final transforms from aiMatrix4x4 types into a DirectXTK SimpleMath Matrix (really an XMMATRIX) with this conversion function:
Matrix Mesh::aiMatrixToSimpleMatrix(CONST aiMatrix4x4 m)
{
return Matrix
(m.a1, m.a2, m.a3, m.a4,
m.b1, m.b2, m.b3, m.b4,
m.c1, m.c2, m.c3, m.c4,
m.d1, m.d2, m.d3, m.d4);
}
Because of the column-vector orientation of aiMatrix4x4 Assimp matrices, the final bone transforms are not transposed for HLSL consumption. The array of final bone transforms are passed to the skinning vertex shader constant buffer as follows.
commandList->SetPipelineState(m_psoForwardSkinned.Get()); // set PSO
// Update vertex shader with current bone transforms
CONST std::vector<Matrix> transforms = m_assimpModel.GetShaderTransforms();
VSBonePassConstants vsBoneConstants{};
for (UINT i = 0; i < m_assimpModel.GetNumBones(); i++)
{
// We do not transpose bone matrices for HLSL because the original
// Assimp matrices are column-vector matrices.
vsBoneConstants.boneTransforms[i] = transforms[i];
//vsBoneConstants.boneTransforms[i] = transforms[i].Transpose();
//vsBoneConstants.boneTransforms[i] = Matrix::Identity;
}
GraphicsResource vsBoneCB = m_graphicsMemory->AllocateConstant(vsBoneConstants);
vsPerObjects.gWorld = m_assimp_world.Transpose(); // vertex shader per object constant
vsPerObjectCB = m_graphicsMemory->AllocateConstant(vsPerObjects);
commandList->SetGraphicsRootConstantBufferView(RootParameterIndex::VSBoneConstantBuffer, vsBoneCB.GpuAddress());
commandList->SetGraphicsRootConstantBufferView(RootParameterIndex::VSPerObjConstBuffer, vsPerObjectCB.GpuAddress());
//commandList->SetGraphicsRootDescriptorTable(RootParameterIndex::ObjectSRV, m_shaderTextureHeap->GetGpuHandle(ShaderTexDescriptors::SuzanneDiffuse));
commandList->SetGraphicsRootDescriptorTable(RootParameterIndex::ObjectSRV, m_shaderTextureHeap->GetGpuHandle(ShaderTexDescriptors::DefaultDiffuse));
for (UINT i = 0; i < m_assimpModel.GetMeshSize(); i++)
{
commandList->IASetVertexBuffers(0u, 1u, &m_assimpModel.meshEntries[i].GetVertexBufferView());
commandList->IASetIndexBuffer(&m_assimpModel.meshEntries[i].GetIndexBufferView());
commandList->IASetPrimitiveTopology(D3D_PRIMITIVE_TOPOLOGY_TRIANGLELIST);
commandList->DrawIndexedInstanced(m_assimpModel.meshEntries[i].GetIndexCount(), 1u, 0u, 0u, 0u);
}
Please note I am using the Graphics Resource memory management helper object found in the DirectXTK12 library in the code above. Finally, here's the skinning vertex shader I'm using.
// Luna (2016) lighting model adapted from Moller
#define MAX_BONES 4
// vertex shader constant data that varies per object
cbuffer cbVSPerObject : register(b3)
{
float4x4 gWorld;
//float4x4 gTexTransform;
}
// vertex shader constant data that varies per frame
cbuffer cbVSPerFrame : register(b5)
{
float4x4 gViewProj;
float4x4 gShadowTransform;
}
// bone matrix constant data that varies per object
cbuffer cbVSBonesPerObject : register(b9)
{
float4x4 gBoneTransforms[MAX_BONES];
}
struct VertexIn
{
float3 posL : SV_POSITION;
float3 normalL : NORMAL;
float2 texCoord : TEXCOORD0;
float3 tangentU : TANGENT;
float4 boneWeights : BONEWEIGHT;
uint4 boneIndices : BONEINDEX;
};
struct VertexOut
{
float4 posH : SV_POSITION;
//float3 posW : POSITION;
float4 shadowPosH : POSITION0;
float3 posW : POSITION1;
float3 normalW : NORMAL;
float2 texCoord : TEXCOORD0;
float3 tangentW : TANGENT;
};
VertexOut VS_main(VertexIn vin)
{
VertexOut vout = (VertexOut)0.f;
// Perform vertex skinning.
// Ignore BoneWeights.w and instead calculate the last weight value
// to ensure all bone weights sum to unity.
float4 weights = vin.boneWeights;
//weights.w = 1.f - dot(weights.xyz, float3(1.f, 1.f, 1.f));
//float4 weights = { 0.f, 0.f, 0.f, 0.f };
//weights.x = vin.boneWeights.x;
//weights.y = vin.boneWeights.y;
//weights.z = vin.boneWeights.z;
weights.w = 1.f - (weights.x + weights.y + weights.z);
float4 localPos = float4(vin.posL, 1.f);
float3 localNrm = vin.normalL;
float3 localTan = vin.tangentU;
float3 objPos = mul(localPos, (float4x3)gBoneTransforms[vin.boneIndices.x]).xyz * weights.x;
objPos += mul(localPos, (float4x3)gBoneTransforms[vin.boneIndices.y]).xyz * weights.y;
objPos += mul(localPos, (float4x3)gBoneTransforms[vin.boneIndices.z]).xyz * weights.z;
objPos += mul(localPos, (float4x3)gBoneTransforms[vin.boneIndices.w]).xyz * weights.w;
float3 objNrm = mul(localNrm, (float3x3)gBoneTransforms[vin.boneIndices.x]) * weights.x;
objNrm += mul(localNrm, (float3x3)gBoneTransforms[vin.boneIndices.y]) * weights.y;
objNrm += mul(localNrm, (float3x3)gBoneTransforms[vin.boneIndices.z]) * weights.z;
objNrm += mul(localNrm, (float3x3)gBoneTransforms[vin.boneIndices.w]) * weights.w;
float3 objTan = mul(localTan, (float3x3)gBoneTransforms[vin.boneIndices.x]) * weights.x;
objTan += mul(localTan, (float3x3)gBoneTransforms[vin.boneIndices.y]) * weights.y;
objTan += mul(localTan, (float3x3)gBoneTransforms[vin.boneIndices.z]) * weights.z;
objTan += mul(localTan, (float3x3)gBoneTransforms[vin.boneIndices.w]) * weights.w;
vin.posL = objPos;
vin.normalL = objNrm;
vin.tangentU.xyz = objTan;
//vin.posL = posL;
//vin.normalL = normalL;
//vin.tangentU.xyz = tangentL;
// End vertex skinning
// transform to world space
float4 posW = mul(float4(vin.posL, 1.f), gWorld);
vout.posW = posW.xyz;
// assumes nonuniform scaling, otherwise needs inverse-transpose of world matrix
vout.normalW = mul(vin.normalL, (float3x3)gWorld);
vout.tangentW = mul(vin.tangentU, (float3x3)gWorld);
// transform to homogenous clip space
vout.posH = mul(posW, gViewProj);
// pass texcoords to pixel shader
vout.texCoord = vin.texCoord;
//float4 texC = mul(float4(vin.TexC, 0.0f, 1.0f), gTexTransform);
//vout.TexC = mul(texC, gMatTransform).xy;
// generate projective tex-coords to project shadow map onto scene
vout.shadowPosH = mul(posW, gShadowTransform);
return vout;
}
Some last tests I tried before posting:
I tested the code with a Collada (DAE) model exported from Blender, only to observe the same distorted zigzagging in the Win32 desktop application.
I also confirmed the aiScene object for the loaded model returns an identity matrix for the global root transform (also verified in AssimpViewer).
I have stared at this code for about a week and am going out of my mind! Really hoping someone can spot what I have missed. If you need more code or info, please ask!
This seems to be a bug with the published code in the tutorials / documentation. It would be great if you could open an issue-report here: Assimp-Projectpage on GitHub .
It's taken almost another two weeks of pain, but I finally found the bug. It was in my own code, and it was self-inflicted. Before I show the solution, I should explain the further troubleshooting I did to get there.
After losing faith with Assimp (even though the AssimpViewer tool was animating my model correctly), I turned to the FBX SDK. The FBX ViewScene command line utility tool that's available as part of the SDK was also showing and animating my model properly, so I had hope...
So after a few days reviewing the FBX SDK tutorials, and taking another week to write an FBX importer for my Windows desktop game, I loaded my model and... saw exactly the same zig-zag animation anomaly as the version loaded by Assimp!
This frustrating outcome meant I could at least eliminate Assimp and the FBX SDK as the source of the problem, and focus again on the vertex shader. The shader I'm using for vertex skinning was adopted from the 'Character Animation' chapter of Frank Luna's text. It was identical in every way, which led me to recheck the C++ vertex structure declared on the application side...
Here's the C++ vertex declaration for skinned vertices:
struct Vertex
{
// added constructors
Vertex() = default;
Vertex(FLOAT x, FLOAT y, FLOAT z,
FLOAT nx, FLOAT ny, FLOAT nz,
FLOAT u, FLOAT v,
FLOAT tx, FLOAT ty, FLOAT tz) :
Pos(x, y, z),
Normal(nx, ny, nz),
TexC(u, v),
Tangent(tx, ty, tz) {}
Vertex(DirectX::SimpleMath::Vector3 pos,
DirectX::SimpleMath::Vector3 normal,
DirectX::SimpleMath::Vector2 texC,
DirectX::SimpleMath::Vector3 tangent) :
Pos(pos), Normal(normal), TexC(texC), Tangent(tangent) {}
DirectX::SimpleMath::Vector3 Pos;
DirectX::SimpleMath::Vector3 Normal;
DirectX::SimpleMath::Vector2 TexC;
DirectX::SimpleMath::Vector3 Tangent;
FLOAT BoneWeights[4];
BYTE BoneIndices[4];
//UINT BoneIndices[4]; <--- YOU HAVE CAUSED ME A MONTH OF PAIN
};
Quite early on, being confused by Luna's use of BYTE to store the array of bone indices, I changed this structure element to UINT, figuring this still matched the input declaration shown here:
static CONST D3D12_INPUT_ELEMENT_DESC inputElementDescSkinned[] =
{
{ "SV_POSITION", 0u, DXGI_FORMAT_R32G32B32_FLOAT, 0u, D3D12_APPEND_ALIGNED_ELEMENT, D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0u },
{ "NORMAL", 0u, DXGI_FORMAT_R32G32B32_FLOAT, 0u, D3D12_APPEND_ALIGNED_ELEMENT, D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0u },
{ "TEXCOORD", 0u, DXGI_FORMAT_R32G32_FLOAT, 0u, D3D12_APPEND_ALIGNED_ELEMENT, D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0u },
{ "TANGENT", 0u, DXGI_FORMAT_R32G32B32_FLOAT, 0u, D3D12_APPEND_ALIGNED_ELEMENT, D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0u },
//{ "BINORMAL", 0, DXGI_FORMAT_R32G32B32_FLOAT, 0, D3D12_APPEND_ALIGNED_ELEMENT, D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0 },
{ "BONEWEIGHT", 0u, DXGI_FORMAT_R32G32B32A32_FLOAT, 0u, D3D12_APPEND_ALIGNED_ELEMENT, D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0u },
{ "BONEINDEX", 0u, DXGI_FORMAT_R8G8B8A8_UINT, 0u, D3D12_APPEND_ALIGNED_ELEMENT, D3D12_INPUT_CLASSIFICATION_PER_VERTEX_DATA, 0u },
};
Here was the bug. By declaring UINT in the vertex structure for bone indices, four bytes were being assigned to store each bone index. But in the vertex input declaration, the DXGI_FORMAT_R8G8B8A8_UINT format specified for the "BONEINDEX" was assigning one byte per index. I suspect this data type and format size mismatch was resulting in only one valid bone index being able to fit in the BONEINDEX element, and so only one index value was passed to the vertex shader each frame, instead of the whole array of four indices for correct bone transform lookups.
So now I've learned... the hard way... why Luna had declared an array of BYTE for bone indices in the original C++ vertex structure.
I hope this experience will be of value to someone else, and always be careful changing code from your original learning sources.

In Vulkan, the output color of vertices in vertex shader is different than what I am getting in fragment shader

In Vulkan, I had written a simple program to draw lines with fixed color, with simple vertex shader and fragement shader. But the colors input to fragment shaders are different than what is set in vertices. I checked with RenderDoc, and the colors passed to the vertex shader are correct (1,1,1,1) for both vertices of a line and also checked its output, its also same. But in Fragment shader, the colors I am getting are (1,1,0,1). Dont understand why this is happening. Irrespetive of what colors vertex shader emit, the input in fragment shader is always yellow.
Vertex shader:
layout(location = 0) in vec4 position;
layout(location = 1) in vec4 color;
layout(location = 2) in vec2 texcoord;
out vec4 io_color;
out vec2 io_uv;
out vec4 io_position2;
layout(std140, binding = 0) uniform UniformBlock_uTransform
{
mat4 uTransform;
};
layout(std140, binding = 1) uniform UniformBlock_uTransform2
{
mat4 uTransform2;
};
void main ()
{
io_uv = texcoord;
io_color = vec4(1,1,1,1); //Just to debug it
gl_Position = uTransform * position;
io_position2 = uTransform2 * position;
}
//Fragement :
in vec4 io_color;
layout(location = 0) out vec4 result;
void main ()
{
result = io_color;
}
Try adding output and input layout qualifiers to the values you pass from one shader to the other to ensure that they actually point to the same location:
VS:
layout (location = 0) out vec4 io_color;
FS:
layout (location = 0) in vec4 io_color;
I recommend always using that syntax to connect shader out- and inputs.
Check if color write mask is not disabled for blue channel.

Fish Eye Wide-angle with a Scene Kit Camera: Possible?

How do I get a distortion like what a fisheye lens does to a view with a SCNCamera in Scene Kit?
Something like this kind of "bowing" of the imagery:
// as Rickster pointed out, this kind of distortion is known as "Barrel Distortion".
From the docs, this is the part that got me intrigued by the possibility of doing this kind of distortion with the camera:
If you compute your own projection transform matrix, you can use this
method to set it directly, overriding the transformation synthesized
from the camera’s geometric properties.
Unfortunately I know nothing about the powers and possibilities of computing ones own projection transform matrix. I'm hoping it's possible to do this kind of distortion via it... but dunno, hence the question.
Any other means via a camera is ideal. Too. Wanting to avoid post processing trickery and get the more "organic" look of this kind of distortion when the camera rotates and moves through the scene.
See any skateboarding video for how this looks in real life.
What you are looking for is called Barrel Distrortion.
There are a few ways of doing this, all of them using GLSL shaders.
You can either use classic OpenGL code, such as this example for the Occulus Rift (you will need to change the shader a little bit), or my personal favorite: SCNTechnique.
Create a technique containing a Barrel Fragment Shader (.fsh), and set its draw parameter to DRAW_QUAD. Then, simply apply the technique to your camera.
You can find an example of Barrel Distortion shader here : http://www.geeks3d.com/20140213/glsl-shader-library-fish-eye-and-dome-and-barrel-distortion-post-processing-filters/2/
EDIT: here's a sample code:
barrel.json (this should go in your scnassets bundle)
{
"passes" : {
"barrel" : {
"outputs" : {
"color" : "COLOR"
},
"inputs" : {
"colorSampler" : "COLOR",
"noiseSampler" : "noiseSymbol",
"a_position" : "a_position-symbol"
},
"program" : "art.scnassets/barrel",
"draw" : "DRAW_QUAD"
}
},
"sequence" : [
"barrel"
],
"symbols" : {
"a_position-symbol" : {
"semantic" : "vertex"
},
"noiseSymbol" : {
"image" : "noise.png",
"type" : "sampler2D"
},
"barrelPower" : {
"type" : "float"
}
}
}
barrel.vsh
attribute vec4 a_position;
varying vec2 uv;
void main() {
gl_Position = a_position;
uv = a_position.xy;
}
barrel.fsh
// Adapted from :
// http://www.geeks3d.com/20140213/glsl-shader-library-fish-eye-and-dome-and-barrel-distortion-post-processing-filters/2/
uniform sampler2D colorSampler;
const float PI = 3.1415926535;
uniform float barrelPower;
varying vec2 uv;
vec2 Distort(vec2 p)
{
float theta = atan(p.y, p.x);
float radius = length(p);
radius = pow(radius, barrelPower);
p.x = radius * cos(theta);
p.y = radius * sin(theta);
return 0.5 * (p + 1.0);
}
void main() {
vec2 rg = 2.0 * uv.xy - 1.0;
vec2 uv2;
float d = length(xy);
if (d < 1.0){
uv2 = Distort(xy);
}else{
uv2 = uv.xy;
}
gl_FragColor = texture2D(colorSampler, uv2);
}
something.m
NSURL *url = [[NSBundle mainBundle] URLForResource:#"art.scnassets/barrel" withExtension:#"json"];
NSDictionary *tecDic = [NSJSONSerialization JSONObjectWithData:[NSData dataWithContentsOfURL: url] options:nil error:nil];
SCNTechnique* technique = [SCNTechnique techniqueWithDictionary:tecDic];
[technique setValue: [NSNumber numberWithFloat:0.5] forKey:#"barrelPower"];
cameraNode.technique = technique;

Switching between textures in OpenGL ES 2

I'm trying to figure out how to put different textures into different texture units and choose which texture to draw with. I have the following code in my onDrawFrame() method
int[] texture = new int[7];
texture[0] =TextureHelper.loadTexture(mActivityContext,R.drawable.texture1);
texture[1] =TextureHelper.loadTexture(mActivityContext,R.drawable.texture2);
texture[2] =TextureHelper.loadTexture(mActivityContext,R.drawable.texture3);
texture[3] =TextureHelper.loadTexture(mActivityContext,R.drawable.texture4);
texture[4] =TextureHelper.loadTexture(mActivityContext,R.drawable.texture5);
texture[5] =TextureHelper.loadTexture(mActivityContext,R.drawable.texture6);
texture[6] =TextureHelper.loadTexture(mActivityContext,R.drawable.texture7);
for (int i = 0; i < 7; i ++) {
GLES20.glActiveTexture(GLES20.GL_TEXTURE0 + i);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texture[i]);
GLES20.glUniform1i(mTextureUniformHandle, i);
Matrix.setIdentityM(mModelMatrix, 0);
Matrix.translateM(mModelMatrix, 0, -0.60f + 0.2f * i, 0.0f, 0.0f);
draw();
}
What this is supposed to do is load seven different textures into separate texture units and draw cubes, each cube with a different texture. However, what ends up happening is that all of the cubes end up being drawn with the first texture.
It works correctly if I change GLES20.glActiveTexture(GLES20.GL_TEXTURE0 + i) to GLES20.glActiveTexture(GLES20.GL_TEXTURE0) and GLES20.glUniform1i(mTextureUniformHandle, i) to GLES20.glUniform1i(mTextureUniformHandle, 0), but that just uses a single texture unit and replaces the texture in that unit every time, which is not what I want to do.
What am I doing wrong?
Thanks in advance.
EDIT:
Vertex shader:
"uniform mat4 u_MVPMatrix;" + // A constant representing the
// combined
// model/view/projection matrix.
"uniform mat4 u_MVMatrix;" + // A constant representing the
// combined model/view matrix.
"attribute vec4 a_Position;" + // Per-vertex position
// information we will pass in.
"attribute vec4 a_Color;" + // Per-vertex color information we
// will pass in.
"attribute vec2 a_TexCoordinate;" + // Per-vertex texture
// coordinate information we
// will pass in.
"varying vec3 v_Position;" + // This will be passed into the
// fragment shader.
"varying vec4 v_Color;" + // This will be passed into the
// fragment shader.
"varying vec2 v_TexCoordinate;" + // This will be passed into
// the fragment shader.
// The entry point for our vertex shader.
"void main()" + "{" +
// Transform the vertex into eye space.
"v_Position = vec3(u_MVMatrix * a_Position);" +
// Pass through the color.
"v_Color = a_Color;" +
// Pass through the texture coordinate.
"v_TexCoordinate = a_TexCoordinate;" +
// gl_Position is a special variable used to store the final
// position.
// Multiply the vertex by the matrix to get the final point in
// normalized screen coordinates.
"gl_Position = u_MVPMatrix * a_Position;" + "} ";
Fragment shader:
"precision mediump float;" + // Set the default precision to medium. We don't need as high of a
// precision in the fragment shader.
"uniform sampler2D u_Texture;" + // The input texture.
"varying vec3 v_Position;" + // Interpolated position for this fragment.
"varying vec4 v_Color;" + // This is the color from the vertex shader interpolated across the
// triangle per fragment.
"varying vec2 v_TexCoordinate;" + // Interpolated texture coordinate per fragment.
// The entry point for our fragment shader.
"void main()" +
"{" +
// Multiply the color by the diffuse illumination level and texture value to get final output color.
"gl_FragColor = (v_Color * texture2D(u_Texture, v_TexCoordinate));" +
"}";
draw() method:
public void draw() {
// Pass in the position information
mCubePositions.position(0);
GLES20.glVertexAttribPointer(mPositionHandle, mPositionDataSize, GLES20.GL_FLOAT, false, 0, mCubePositions);
GLES20.glEnableVertexAttribArray(mPositionHandle);
// Pass in the color information
mCubeColors.position(0);
GLES20.glVertexAttribPointer(mColorHandle, mColorDataSize, GLES20.GL_FLOAT, false, 0, mCubeColors);
GLES20.glEnableVertexAttribArray(mColorHandle);
// Pass in the texture coordinate information
mCubeTextureCoordinates.position(0);
GLES20.glVertexAttribPointer(mTextureCoordinateHandle, mTextureCoordinateDataSize, GLES20.GL_FLOAT, false, 0, mCubeTextureCoordinates);
GLES20.glEnableVertexAttribArray(mTextureCoordinateHandle);
// This multiplies the view matrix by the model matrix, and stores the
// result in the MVP matrix
// (which currently contains model * view).
Matrix.multiplyMM(mMVPMatrix, 0, mViewMatrix, 0, mModelMatrix, 0);
// Pass in the modelview matrix.
GLES20.glUniformMatrix4fv(mMVMatrixHandle, 1, false, mMVPMatrix, 0);
// This multiplies the modelview matrix by the projection matrix, and
// stores the result in the MVP matrix
// (which now contains model * view * projection).
Matrix.multiplyMM(mMVPMatrix, 0, mProjectionMatrix, 0, mMVPMatrix, 0);
// Pass in the combined matrix.
GLES20.glUniformMatrix4fv(mMVPMatrixHandle, 1, false, mMVPMatrix, 0);
// Draw the cube.
GLES20.glDrawArrays(GLES20.GL_TRIANGLES, 0, 6);
}
Assigning mTextureUniformHandle :
mTextureUniformHandle = GLES20.glGetUniformLocation(mProgramHandle, "u_Texture");
Lately I've been searching for multiple textures in fragment shader and came across this Binding textures to samplers
from which I got the following to work:
In onSurfaceCreated or onSurfaceChanged:
Load shaders (attach and link) and get uniform locations for sampler2D (and other variables):
normalMapLoc = GLES20.glGetUniformLocation(shaderProgram, "normalMap");
shadowMapLoc = GLES20.glGetUniformLocation(shaderProgram, "shadowMap");
Load textures:
GLES20.glGenTextures(2, textures, 0);
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textures[0]);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_NEAREST);
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);
bitmap.recycle();
GLES20.glActiveTexture(GLES20.GL_TEXTURE1);
GLES20.glBindTexture(GL10.GL_TEXTURE_COORD_ARRAY, textures[1]);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_NEAREST);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, width, height, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, mColorBuffer);
GLES20.glUniform1i(normalMapLoc, 0); // Texture unit 0 is for normal images.
GLES20.glUniform1i(shadowMapLoc, 1); // Texture unit 1 is for shadow maps.
In onDrawFrame:
GLES20.glClearColor(0f, 0f, 0f, 0f);
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);
// pass variables to the fragment shader
...
// get handle to vertex shader's Position member, etcetera
int mPositionHandle = GLES20.glGetAttribLocation(shaderProgram, "vPosition");
GLES20.glEnableVertexAttribArray(mPositionHandle);
GLES20.glVertexAttribPointer(mPositionHandle, 3, GLES20.GL_FLOAT, false, 0, mVertexBuffer);
GLES20.glDrawElements(GLES20.GL_TRIANGLE_STRIP, 4, GLES20.GL_UNSIGNED_SHORT, mIndexBuffer);
and finally the fragment shader looks like this (only relevant portion of code):
uniform sampler2D normalMap, shadowMap;
varying vec2 pos;
void main() {
vec4 color = texture2D(normalMap, pos);
vec4 shadow = texture2D(shadowMap, pos);
// do stuff with the colors
...
gl_FragColor = ...;
}
This way i was finally able to access both textures !
Hope this helps.

GLSL shader generation of normals

Hi I am writing 3D modeling app and I want to speed up rendering in OpenGL. Currently I use glBegin/glEnd which is really slow and deprecated way. I need to draw very fast flat shaded models. I generate normals on CPU every single frame. This is very slow. I tried to use glDrawElements with indexed geometry, but there is problem in normal generation, because normals are specified at vertex not at triangle level.
Another idea was to use GLSL to generate normals on GPU in geometry shader. I written this code for normal generation:
#version 120
#extension GL_EXT_geometry_shader4 : enable
vec3 NormalFromTriangleVertices(vec3 triangleVertices[3])
{
// now is same as RedBook (OpenGL Programming Guide)
vec3 u = triangleVertices[0] - triangleVertices[1];
vec3 v = triangleVertices[1] - triangleVertices[2];
return cross(v, u);
}
void main()
{
// no change of position
// computes normal from input triangle and front color for that triangle
vec3 triangleVertices[3];
vec3 computedNormal;
vec3 normal, lightDir;
vec4 diffuse;
float NdotL;
vec4 finalColor;
for(int i = 0; i < gl_VerticesIn; i += 3)
{
for (int j = 0; j < 3; j++)
{
triangleVertices[j] = gl_PositionIn[i + j].xyz;
}
computedNormal = NormalFromTriangleVertices(triangleVertices);
normal = normalize(gl_NormalMatrix * computedNormal);
// hardcoded light direction
vec4 light = gl_ModelViewMatrix * vec4(0.0, 0.0, 1.0, 0.0);
lightDir = normalize(light.xyz);
NdotL = max(dot(normal, lightDir), 0.0);
// hardcoded
diffuse = vec4(0.5, 0.5, 0.9, 1.0);
finalColor = NdotL * diffuse;
finalColor.a = 1.0; // final color ignores everything, except lighting
for (int j = 0; j < 3; j++)
{
gl_FrontColor = finalColor;
gl_Position = gl_PositionIn[i + j];
EmitVertex();
}
}
EndPrimitive();
}
When I integrated shaders to my application, no speed improvement occurred. It was worse than before. I am newbie in GLSL and shaders overall so I don't know what I done wrong.
I tried this code on MacBook with Geforce 9400M.
To be more clear, this is code I want to replace:
- (void)drawAsCommandsWithScale:(Vector3D)scale
{
float frontDiffuse[4] = { 0.4, 0.4, 0.4, 1 };
CGFloat components[4];
[color getComponents:components];
float backDiffuse[4];
float selectedDiffuse[4] = { 1.0f, 0.0f, 0.0f, 1 };
for (uint i = 0; i < 4; i++)
backDiffuse[i] = components[i];
glMaterialfv(GL_BACK, GL_DIFFUSE, backDiffuse);
glMaterialfv(GL_FRONT, GL_DIFFUSE, frontDiffuse);
Vector3D triangleVertices[3];
float *lastDiffuse = frontDiffuse;
BOOL flip = scale.x < 0.0f || scale.y < 0.0f || scale.z < 0.0f;
glBegin(GL_TRIANGLES);
for (uint i = 0; i < triangles->size(); i++)
{
if (selectionMode == MeshSelectionModeTriangles)
{
if (selected->at(i))
{
if (lastDiffuse == frontDiffuse)
{
glMaterialfv(GL_FRONT_AND_BACK, GL_DIFFUSE, selectedDiffuse);
lastDiffuse = selectedDiffuse;
}
}
else if (lastDiffuse == selectedDiffuse)
{
glMaterialfv(GL_BACK, GL_DIFFUSE, backDiffuse);
glMaterialfv(GL_FRONT, GL_DIFFUSE, frontDiffuse);
lastDiffuse = frontDiffuse;
}
}
Triangle currentTriangle = [self triangleAtIndex:i];
if (flip)
currentTriangle = FlipTriangle(currentTriangle);
[self getTriangleVertices:triangleVertices fromTriangle:currentTriangle];
for (uint j = 0; j < 3; j++)
{
for (uint k = 0; k < 3; k++)
{
triangleVertices[j][k] *= scale[k];
}
}
Vector3D n = NormalFromTriangleVertices(triangleVertices);
n.Normalize();
for (uint j = 0; j < 3; j++)
{
glNormal3f(n.x, n.y, n.z);
glVertex3f(triangleVertices[j].x, triangleVertices[j].y, triangleVertices[j].z);
}
}
glEnd();
}
As you can see it is very inefficient, but working.triangles is array of indexes into vertices array.
I tried to use this code for drawing, but I can't have only one index array not two (one for vertices and second for normals).
glEnableClientState(GL_VERTEX_ARRAY);
uint *trianglePtr = (uint *)(&(*triangles)[0]);
float *vertexPtr = (float *)(&(*vertices)[0]);
glVertexPointer(3, GL_FLOAT, 0, vertexPtr);
glDrawElements(GL_TRIANGLES, triangles->size() * 3, GL_UNSIGNED_INT, trianglePtr);
glDisableClientState(GL_VERTEX_ARRAY);
Now, how can I specify pointer to normals, when some vertices are shared by different triangles, so different normals for them?
So I finally managed to increase rendering speed. I recalculate normals on CPU, only when vertices or triangles changes, which occurs only when working in one mesh not in whole scene.
It is not solution that I wanted but in real world it is better than previous approaches.
I cache whole geometry into separate normal and vertex array, indexed drawing cannot be used because I want flat shading (similar problem to smoothing groups in 3ds max).
I use simple glDrawArrays and for lighting vertex shader, that is because I want in triangle mode different color for selected triangle and another one for unselected and there is no array of materials (I didn't found any one).
You wouldn't usually calculate the normals every frame, only when the geometry changes.
And to have one normal per triangle just set the same normal for each vertex in the triangle. That does mean you can't share vertices between adjacent triangles in your mesh but that's not unusual at all in this kind of thing.
Your question makes me remember this Normals without Normals blog post.