Surfaces become dark based on rotation when normal map is used

MirceaKitsune

I'm running into one bizarre issue I'll need to know how to resolve. I'm experimenting with creating roads using 3D tilesets mapped in 2D space, everything's working fine and was looking good till I got down to texturing: I downloaded a bunch of images and set them on a StandardMaterial3D for my obj surfaces. Then I noticed some surfaces are randomly dark and reflect light differently from other tiles near them using the same mesh!

I pinned the culprit to the normal map: Either disabling the normal feature or setting lighting to per-vertex instead of per-pixel makes it go away, obviously this is not an option for anything of quality. The only thing that differs between tiles is their rotation, each edge and corner is rotated at either 0 or 90 or 180 or 270... somehow rotating each tile scene differently causes massive changes in the interpretation of the normal map. I know surface normals are facing the right way as backface culling is enabled so I wouldn't be seeing the surfaces if that was an issue.

Is this a known problem with Godot 4.0 / Vulkan? What should I change to get rid of this effect? It looks horrible so it's one of the things I'll definitely need to resolve somehow.

cybereality

This happens if the normal is in the wrong format. Especially when you download textures, they could be DirectX or OpenGL format (which is easy to fix in GIMP) or the normals can just be incorrect altogether. I had this problem with a map I downloaded online, even though on visual inspection of the image it looked okay. The best way to fix this is that if you have a height map, you can regenerate the normal in GIMP with the normal map filter. Otherwise you can just flip or adjust the channels in GIMP, but this can be unreliable depend on how the normals are corrupted.

MirceaKitsune

The normal map shouldn't be broken, it looks okay to me and works fine on a sphere. The texture set can be found here if anyone else wishes to test, using the 1k package for Godot:

https://polyhaven.com/a/floor_tiles_09

Could it be because of the format I converted it to? I'm using dds exported with BC1 / DXT1 from GIMP: I need to as this is the only way to reduce the file size which is an issue with textures, other dds compression types are twice bigger and formats other than DDS are typically larger as well (png or tga). I'll test BC3 / DXT5 and other formats if the issue persists tomorrow, if it's that I may have to leave the normal map twice larger.

Megalomaniak

DXT1 is oof. DXT5 should be fine enough tho, and DXT1 still has it's niche uses but is going to be a very obvious degrade in quality.

Also when dealing with 8 bit normal maps it's a good idea to dither them.

cybereality

The texture I was having problems with was from that same website, and in the same series (floor tiles) so it probably has the same problem I found. You need to download the height/depth map and regenerate the normal map in GIMP. And for a normal map, even JPG looks fine. The quality difference is likely not noticeable to users.

MirceaKitsune

Played a bit more with this and got some odd results: I noticed that when switching from DXT1 to DXT5 or no compression the issue partly goes away, same when using another format like jpg. But it only changes somewhat: Instead of getting random variations of darkness for each tile, there's just two now with either a light or dark version, the issue itself is still there regardless.

I'm prolly gonna use a different texture here and leave that one for areas where rotation isn't randomized. Very strange problem though. Could there be a bug in how Godot 4.0 interprets normal maps? If this issue can't be reproduced in Blender it might mean the engine is doing something wrong.

MirceaKitsune

I think I caught the subtle issue that's going on. Indeed the normal textures appear to be compiled in a bad way for some images on that site: The top of the tiles aren't a dark blue like most normal maps, they're lighter in color... this means they're interpreted as being at an angle rather than flat, only accepting light from one direction. I'm surprised a place like Polyheaven would make such a mistake, might have to bring it to their attention so folks there know... till then I'll try other alternatives that don't have this problem.

Or perhaps I can fix the normal map in Gimp? It looks like simply adjusting its brightness might do the trick, but with normal maps that's tricky to get right. I don't feel like recompiling them altogether so if that fails I'll just use a different texture set. At a first glance this levels adjustment would appear to correct it, untested but I'm assuming based on the color that normally represents flatness.

Megalomaniak

If you can, share a minimal reproduction test project, so we can better help.

edit: we posted at around the same time.. There are some tools such as photoshop and gimp plugins but also standalone apps such as xnormal or crazybump that can be used to normalize your textures.

MirceaKitsune

Problem solved! Simply adjusting the color levels in Gimp with a value of 0.4 for the middle slider does the trick, no other traits of the normal map are broken. As a general rule the flat surface needs to be 0.5 0.5 1.0 (128 128 256) RGB. Light now shines just fine from all angles.

Megalomaniak

Yeah, you effectively normalized it manually.

cybereality

Well you could have just regenerated the normal from the height like I was saying, since I encountered the exact same issue with the same texture series and provided you with the solution. Just saying.