First, I recommend decreasing the lightmap texel density as you did. When you only bake indirect lighting (the default), you don't need a very high texel density to begin with, as indirect lighting is low-frequency data.
For future readers: If decreasing the texel density does not resolve this, try disabling denoising in the BakedLightmap inspector, as OpenImageDenoise can be very RAM-intensive. To combat noise in your "final" light bake, increase the bake quality before baking lightmaps.
The out of memory issue likely happens because Godot 3.x's CowData limits individual data structures to 2 GB each (32-bit integer). This may be fixed in 4.0, which also has a GPU lightmapper rewritten from scratch (it's much faster on a dedicated GPU).