Welcome to the forums @Throckmorpheus!
I have some experience with 3D GUI, as I updated the GUI in 3D demo to work with Godot 3.0, back when it first came out. That said, I haven't touched the demo in over a year, so... yeah. I can try to help though with my limited experience.
What issues are you having currently?
A couple things to right off that might be causing issues:
Were you scaling the plane/quad the 3D GUI is attached to? If so, you may need to multiple quad_mesh_size
by the scale of the MeshInstance node, so it can convert the 3D position from the raycast to the 2D position on the Viewport. I'm not sure if the demo was tested to work with scaled MeshInstance nodes.
The code expects the Viewport and Quad/Plane to always be square, from what I remember. If you have a non-square Viewport or Quad/Plane, I would suggest making it square and just adding padding to the Viewport so it appears rectangular.
* Are you using a Billboard material? Unfortunately I don't think we ever found a way to get billboard materials working in the demo, so you may need to rotate the MeshInstance manually through code if you need it to always face the player.
Also, if any of the issues you are having can be reproduced with the GUI in 3D demo from GitHub, could you please make an issue on the GitHub repository detailing the issue(s) found? That way, other users who look at the demo will know of the issue(s), as well as then the issue can (hopefully) be fixed for the demo as well.