I actually just managed to do it by adding 180 to each component of the angle (camera+pivot+180).
Turns out, always perpendicular isn't exactly what I need.
For dragging vertically I just need the plane to sit vertically and spin only around Y to have the normal opposite to the camera, and that works perfectly. And for that, all I need to do is use the camera's pivot rotation in Y and change the plane in the CollisionShape itself to face the opposite way.
Dragging horizontally is the tricky part. Having the plane face the camera seems no good, as it limits the gizmo to how much the mouse can move on the screen, and the result is you can't drag the gizmo that much far away. This is because, the X position of the mouse in the plane (the raycast hit) is what I'm assigning to the X position of the gizmo. The same goes for Z.
Having the plane simply sitting still horizontally kinda works, except when the camera is about leveled with the plane. But I also have some problems with the gizmo being dragged way too much, depending on where the mouse is and maybe also where the camera is facing. Having the plane vertically also has the same problems.
I'm thinking maybe I need to keep it leaning to some degree depending on the camera's perspective, but I'm not sure.
EDIT: indeed, from playing around in blender a bit, I can already tell that the plane's normal should always be pointing toward the camera's position, but stay flat along the X axis (if locking to X).