3D light mask with Godot 3.2.3?

GlyphTheWolf

Hello,

Is it possible to make 3D light mask in Godot 3.2.3?

I am trying to make one of my lights on 3D scene being player "visibility" to render only meshes that are affected by this special light (also taking into account shadows, to block seeing through walls or doors).

As I know there is 2D light mask support in Godot, but no 3D light mask, so custom shader is needed to achieve what I want.

I came up with some idea to draw in first pass "visibility mask" which is actually only black/white colors (and in between depending on light attenuation, I choosen to use "black light" as my visibility light) and in second pass I used created mask to actually render real materials only on white areas while black areas were left black. It was working well rendering invisible parts just as black, but it have one big (for me) limitation - it doesn't support any aplha blending. Since black/white mask is rendered in first pass then anything using alpha can't be rendered properly, because instead of blending with real color it will always blend with "visibility mask" instead. If I render visibility mask without writing to depth buffer then some background meshes may overwrite my mask with different colors and then other meshes on top won't be rendered properly.

I tried to update alpha value in light shader to maybe instead of writing visibility mask in first pass just calculate alpha based on my "black light" attenuation, but writing to alpha in light shader doesn't seems to do anything (even setting alpha to constant 0.0 won't make any difference, like if writting to alpha from light shader just doesn't work, but engine doesn't complain that I can't write to alpha from light shader).

One solution I see could be drawing light mask instead directly to buffer just to some texture instead, but how it may be done without need to duplicate whole scene? I know I could render scene to viewport to use it as buffer/texture, but I need to render the same scene twice, one time with my "visibility mask" shader and second one with proper shader using created mask, since shader is connected with material (even using the same shader with uniform will need different material) I need copy of whole scene with different material set to all meshes or change somehow shader of all meshes for viewport render.

Is there any good way to properly implement 3D light masking supporting also alpha in current Godot version? Maybe I am missing something or there is simplier way to achieve that?

Thank you in advance for any help.

Calinou

Godot 3.x supports light masking, but it's difficult to use and often broken. Godot 4.0 will have built-in CanvasItem masking and clipping, making this much easier.

GlyphTheWolf

@Calinou Thank you for your answer, however that wasn't good news for me :confused:

Buuuut, after loooong time of fighting with Godot and trying many different ideas I finally achieved something close to what I wanted. It wasn't easy, because with every idea I had there was some obstacles with Godot that made my ideas impossible to implement (at least with Godot 3.2.3 without modifying source code). To achieve my final result I needed even write my own simplified implementation of GridMap (because default GridMap doesn't support render layers - I know it should be fixed in Godot 3.2.4, but I don't know when it will be released and anyway I created my custom GridMap with similar interface to default one, so it will be easy to change it later after fix).

I can't say it is ideal solution, but it gives at least important part of mechanics I want to use in my project. Without visibility it wouldn't be fun to explore different corridors and rooms, because player could see everything before even entering the room.

I think it is not the best solution, but I hope it is enaugh for now. There are some pros and cons I see.

Pros: My final solution allow alpha blending to be used properly It is possible to use standard shaders for most objects Possible to use also other lights even with shadows (except directional one becasue of render layer issue - it always affect all layers!) Possible to achieve nice visual effects because primary scene is rendered almost normally (with custom shaders for light mask it could be impossible to achieve some effects without losing masking I guess or it would be at least more complicated)

Cons: Not sure about performance when more stuff will be added (but I hope it is possible to optimize it somehow) I don't like sharp shadows that are created for visibility mask making it look a bit worse than I would like, but still at least it fulfill its main task * Need to maintain second scene with copy of almost everything (but it may be simplified and also it gives control over what exactly will affect visibility mask, what is very useful)

Here is some demo of how it looks like (I hope it will be available to see): https://deuter.ddns.net/download/mask.mp4

So, how my final solution works? It is almost what I wrote as possible solution in my question. I created separate scene as my "visibility world" where I instance simplified objects and use that scene to render visibility mask to the viewport. Main scene is rendered as always to buffer with all default features available. At the end I use rendered mask from viewport in simple screen space shader to mask buffer output.

And at the end I would like to point out some of obstacles I faced, some of them as I know will be fixed in godot 4.0: 1. No possibility to pass value calculated in fragment shader to light shader 2. Light cull mask is not working as I would expect. One issue is with directional light where cull mask is not working at all. Another issue I see is that when I want light to affect only one layer objects from other layers (not affected by light) still cast shadows for that light. I would expect that if light doesn't affect object it shouldn't also cast shadow for that light (I know it is possible to disable shadows for mesh, but that will remove all shadows, what is not what I wanted) 3. GirdMap render layers doesn't work (actually there is even no layers to set) 4. Setting alpha in light shader works strange (maybe there are other calculations done later with that, but I don't know how it works. I tested it a lot and sometimes setting alpha works as expected, but sometimes it doesn't change alpha at all? I don't know why, I gave up with that idea, because it produced more issues than solutions)

To sum up, if someone want to use 3D light mask in Godot 3.2.3 it is possible, but not easy (at least without custom Godot build), so if you don't decided yet, better stick with project where camera position and depth buffer will be enaugh for visibility or prepare for fight (but maybe a bit easier with my above experience).