This problem happens frequently but not all the time. When I set a cell to -1 on my tilemap sometimes my raycast stops detecting collisions for 1 frame. All collision masks are set properly and the raycast is set to enabled.
I'm not sure if its because the entire tilemap is being redrawn when a cell is set so there is a chance for a frame for it to not actually be there and detected by the raycast or what.
The blue line on top of the sign is the raycast and it extends down into the ground. I have a print statement that constantly prints out "ground vision not colliding" when its, obviously, not colliding with the tilemap anymore. As you can see, when the first cell is set to -1, this statement isn't triggered but when the second cell is set to -1, the statement fires once and then immediately begins detecting the collision again.
I made a sample project that replicates the error. Just right click on the ground tiles to make them disappear. It happens seemily randomly but a lot. https://1drv.ms/u/s!AikP2SQvB8segeMBl6pgEmImbEYX7g?e=fLDeXv
