Custom normal map shader

GreatRash

Please help with normal map shder. Here is my code:

shader_type spatial;

uniform sampler2D _normal : hint_normal;

varying mat3 TBN; // from view to tangent space
varying mat3 INV_TBN; // from tangent to view space

void vertex() {
	TBN = mat3(
		(MODELVIEW_MATRIX * vec4(TANGENT, 1.0)).xyz, // from model to view space
		(MODELVIEW_MATRIX * vec4(BINORMAL, 1.0)).xyz,
		(MODELVIEW_MATRIX * vec4(NORMAL, 1.0)).xyz
	);
	INV_TBN = transpose(TBN);
}

void light() {
	vec3 N = normalize(texture(_normal, UV).xyz * 2.0 - 1.0); // already in tangent space
	vec3 L = TBN * LIGHT; // from view to tangent space
	
	float NdotL = max(dot(N, L), 0.0);
	
	DIFFUSE_LIGHT += NdotL * ATTENUATION * ALBEDO;
}

And here is what I get with it. For example here is standart Godot shader.

What is wrong with my shader?

cybereality

I think the math is right, but the lighting calculation is different. Godot is using a point light, and ndotl is for a single directional light. You should change the light in Godot to use only a single directional light. Also create an environment on the camera with just a solid clear color, as the Godot standard shader uses image based lighting from the sky to affect the lighting on objects.

xyz

I think light direction shouldn't be a problem as LIGHT builtin in the light shader is always the correct per-pixel light vector, regardless of the light type. Environment is part of the issue as it looks like there's no way to get its parameters into a custom light shader (someone correct me if I'm wrong here)

That being said, your main problem is light vector transformation. Since LIGHT is in camera space and TBN matrix transforms from tangent space to camera space - you should use its inverse. Also the light vector should be reversed in order for dot product to give the proper result. So your line 19 should be: vec3 L = INV_TBN * -LIGHT Alternatively you could transform the normal using TBN matrix. Which will give you the same end result.

There's one additional caveat with your code that may cause unwanted behavior down the line. TBN matrix calculation using MODELVIEW_MATRIX will be correct only for proportional scaling. Otherwise the resulting TBN matrix will not be orthonormal, resulting in incorrect light/normal transformation in the light shader. The proper matrix that ensures orthonormality is: mat4 m = transpose(inverse(MODELVIEW_MATRIX));

cybereality

Okay, good to know.

GreatRash

Thanks fot your reply. But shader still not work correctly. Also I don't understand why LIGHT vector must be inversed.

xyz

Try to remove environment lighting and put two shaders side by side. Easier to debug that way.

Light vector must be reversed because it points from light to the surface, For proper calculation of light incidence angle via dot product, this vector should point in the opposite direction - from surface to light.

GreatRash

Yeah, tried that, but I still don't understand what is going on.

In all tutorials about normal mapping light vector is used, not inversed light. Is it somehow special in Godot? Also here is example from Godot docs fbout diffuse lighting:

void light() {
   DIFFUSE_LIGHT += clamp(dot(NORMAL, LIGHT), 0.0, 1.0) * ATTENUATION * ALBEDO;
}

xyz

@GreatRash You're right. LIGHT vector doesn't have to be inverted. Godot already does it for you. The problem is then in TBN matrix basis vectors orientation. Godot flips the y axis of normal maps on import so TBN calculation should take that into account. Here's the complete shader:

shader_type spatial;
uniform sampler2D _normal : hint_normal;
varying mat3 TBN; // from view to tangent space
varying mat3 INV_TBN; // from tangent to view space

void vertex() {
	mat4 m = transpose(inverse(MODELVIEW_MATRIX));
    TBN = mat3(
        (m * vec4(TANGENT, 1.0)).xyz, // from model to view space
		(m * vec4(BINORMAL, 1.0)).xyz,
		-(m * vec4(NORMAL, 1.0)).xyz
    );
    INV_TBN = inverse(TBN);
}

void light() {
    vec3 N = normalize(texture(_normal, UV).xyz * 2.0 - 1.0); // already in tangent space
    vec3 L = INV_TBN * LIGHT; // from view to tangent space
    float NdotL = max(dot(N, L), 0.0);
    DIFFUSE_LIGHT += NdotL * ATTENUATION * ALBEDO;
}

And here's the result. On the left is standard spatial material, on the right is the above shader:

GreatRash

Yes, it worked. Thank you! Actually it worked even better than standard Godot NORMALMAP option.

Final question: can you explain transpose(inverse(MODELVIEW_MATRIX)) Or maybe you have some links where I can read about it.

xyz

@GreatRash said: Final question: can you explain transpose(inverse(MODELVIEW_MATRIX)) Or maybe you have some links where I can read about it.

View space TBN matrix is constructed from 3 basis vectors representing 3 axes. The basis must be orthonormal for normal mapping to work properly. Orthonormal means that 3 basis vectors must be perpendicular to each other and their length must be 1.

So, when transforming TANGENT, BINORMAL and NORMAL vectors into view space, we must take care that above criteria are met. In order to transform them using modelview matrix we need to eliminate scaling and translation from it, so that only rotation from that matrix is applied. Simplest way to do so is to calculate transposed inverse. Explained in more detail here.

This matrix is often called 'normal matrix' and it's typically used in vertex shaders to transform mesh normals into camera space to avoid normal skewing due non-proportional scaling. It's odd that Godot didn't include it as a built-in uniform. If you plan to use it in vertex-heavy meshes it's more optimal to calculate it on the cpu side and send it to shader as an uniform. Calculating it per vertex is superfluous as the result is always the same.

Note that you can still use plain modelview matrix to do the job (either for normals or basis vectors) but there must be no non-proportional scaling in the matrix and resulting vectors must be normalized (to nullify translation and proportional scalling)

GreatRash

Thank you so much for such a detailed explanation!