You're already doing it on cpu, since according to your code, cpu already knows the visibility. You're effectively delegating something that is per-object decision to a needlessly repeated per-pixel decision. And in a very roundabout way that loads more work on both; cpu and gpu.
Yeah, check if there is a way to toggle voxel visibility via the engine, and just do that :)