Moving Separate Parts of Human Model

mec

Hey , this is Mason I'm recently working on a motion capture project on Godot, I have done all the electronic parts for IMU sensors. but there is problem with moving the body parts on Godot, IMU sensors sends Quaternion data which I send it over serial port and it's ready to use in Godot, 1.Which format should export my human model? dea , gltf or ... 2.What exactly should I do for moving different body parts? and also all the project will be published for all users for free at last thanks for your attention also i will be grateful for any help Mason

TwistedTwigleg

@mec said: 1.Which format should export my human model? dea , gltf or ...

GLTF is probably the best choice, as it has the most support in Godot currently, and should be reasonably future proof as well.

2.What exactly should I do for moving different body parts?

If the model is rigged to a skeleton, than you will need to use the set_global_pose_override function (documentation) to move the bones. The one gotcha with global poses is that they are not world transforms, like global_transform, but instead are the transform relative to the Skeleton node, so more like transform on a node that is a child of the Skeleton.

mec

@TwistedTwigleg thanks for your attention and time for better understanding the final goal I will attach a video file, also there is not enough information about the function you mentioned.(I mean there is nothing) Kind regards Mason

newmodels

You might find the function, under the Skeleton section of the programming manual online. 'Godot Skeleton' in google.com.

TwistedTwigleg

@mec said: @TwistedTwigleg thanks for your attention and time for better understanding the final goal I will attach a video file, also there is not enough information about the function you mentioned.(I mean there is nothing) Kind regards Mason

For the function, as @newmodels mentioned, it is under the Skeleton node's API. To use it, you will need to get a Transform, either from a node or elsewhere, and then call the function using something like $Skeleton.set_bone_global_pose_override(bone_id, your_transform_here, 1.0, true).

Here's a better example, which assumes the script is attached to the Skeleton node, there is a child node of the Skeleton called bone_node whose transform you want to use, and that you want to move the first bone:

extends Skeleton

var bone_node : Spatial
var bone_id_to_manipulate = 0

var is_bone_overridden = false

func _ready():
	bone_node = get_node("bone_node")
	start_bone_override()

func start_bone_override():
	is_bone_overridden = true

func update_bone_override():
	# if we are overriding the bone pose...
	if is_bone_overridden == true:
		# set's the bone override transform
		set_bone_global_pose_override(bone_id_to_manipulate, bone_node.transform, 1.0, true);

func stop_bone_override():
	# stop overriding the bone pose every _process call
	is_bone_overridden = false
	# reset the bone pose
	set_bone_global_pose_override(bone_id_to_manipulate, Transform(), 0.0, false)


func _process(delta):
	update_bone_override()

For motion capture, it is a bit more complicated, as you probably have bone chains you want to update and manipulate, but it's certainly doable! What I would recommend doing is making a Bone3D node/class to help make the bone manipulation easier and more straightforward. It is pretty much what I wrote above, but you'll want to make the script on a Spatial node instead of a Skeleton (so you can have multiple) and probably will want to add checks and exported variables, for easier customization.

If you do not mind using C# and a plugin, I have written a IK plugin for Godot that already has a Bone3D node that makes this process a bit easier. It is free for non-commercial use, so as long as you are not sharing the source code for the IK plugin and are not using it commercially, the free version should be usable for your project.

If you are looking to make your project commercial or release the source code, then the C# IK plugin will not work. However, this PR has modifications to the BoneAttachment node that basically makes it possible to optionally be a Bone3D node, so that may be worth looking at. It is in C++ though, but it may helpful as a reference.

mec

@TwistedTwigleg Thanks again for your attention , It totally works. but yet I have another question : If we consider W,X,Y,Z as Quaternion data from Sensors how should I relate them as rotation data to individual bone? in this command if I'm not wrong the transform data is related to "bone_node.transform" : set_bone_global_pose_override(bone_id_to_manipulate, bone_node.transform, 1.0, true); what should I use instead of "bone_node.transform" for quaternion data I already wrote a function to convert quaternion data to Transfer data type but the results are not valid at all :*

func quat_transform(x, y, z, w): var t = $"Physical Bone Bone".get_transform() #Physical Bone Bone is the exact bone that I want to move var myRot = Quat(x, y, z, w) var m = Basis(myRot) var n = Transform(m.x.normalized(), m.y.normalized(), m.z.normalized(), t.origin)

also @newmodels thanks for guiding. Thanks Again Mason

TwistedTwigleg

@mec said: @TwistedTwigleg Thanks again for your attention , It totally works. but yet I have another question : If we consider W,X,Y,Z as Quaternion data from Sensors how should I relate them as rotation data to individual bone?

You will need to construct a Transform with the desired rotation basis. The trickier part here is making sure the position is correct, especially if all you have is rotation data from a Quaternion.

To construct the Transform, you just need to do something like this:

func quat_transform(x, y, z, w):
	var bone_trans = $"Physical Bone Bone".get_transform()
	var myRot = Quat(x, y, z, w)
	var myBasis = Basis(myRot)
	
	# then you can just set the basis property of the bone trans
	bone_trans.basis = myBasis;
	# or return a transform with the desired rotation instead
	# var return_trans = bone_trans.duplicate()
	# return_trans.basis = myBasis;
	# return return_trans;

However, this does not take bone child relationships into account, which is something to be aware of if you have several bones in a row and you want to move only one of them (like the upper arm but not the lower arm, for example). The easiest way to keep bones in the correct parent-child relationship is to setup the Bone3D-like nodes in your scene where child bones are children of the Bone3D-like node that represents the parent. For example:

UpperArm_Bone3D
- LowerArm_Bone3D
- Hand_Bone3D

That way, when you rotate something like UpperArm_Bone3D, the child bones will be rotated as well, giving a more realistic (and likely expected) result.

mec

Hi again @TwistedTwigleg , Sorry for delay, but I was testing all the possible ways to make it work and it took so much time. Anyways, there are some real problems with location(origin) of rotation and parenting that I couldn't solve it. In this video I tried to move Arm and Forearm separately, as you can see rotation origin is (0,0,0) and when the arm is moving the forearm should move with that but it's not.

Here is the script, q variables are the Quaternion data which I'm getting from IMU sensors:

extends Skeleton

var loadedScript = load("res://Model/metarig.py")
var loadedScriptInstance = loadedScript.new()

var bone_node : Spatial
var bone_node_2 : Spatial

var bone_id_to_manipulate = 8
var bone_id_to_manipulate_2 = 9

var is_bone_overridden = false

var q0=0
var q1=0
var q2=0 
var q3=0
var q4=0
var q5=0
var q6=0
var q7=0

var trans
var trans2

func _ready():
#	bone_node = get_node("bone_node")
#	bone_node_2 = get_node("bone_node_2")
	start_bone_override()


func start_bone_override():
	is_bone_overridden = true


func update_bone_override():
	# if we are overriding the bone pose...
	if is_bone_overridden == true:
		# set's the bone override transform
		set_bone_global_pose_override(bone_id_to_manipulate, trans, 1.0, true);
		set_bone_global_pose_override(bone_id_to_manipulate_2, trans2, 1.0, true);


func stop_bone_override():
	# stop overriding the bone pose every _process call
	is_bone_overridden = false
	# reset the bone pose
	set_bone_global_pose_override(bone_id_to_manipulate, Transform(), 0.0, false)
	set_bone_global_pose_override(bone_id_to_manipulate_2, Transform(), 0.0, false)


func _process(_delta):
	var dataPacket = loadedScriptInstance.myProcess()
	var splitPacket=dataPacket.split(',')
	
	q0 = splitPacket[0]
	q1 = splitPacket[1]
	q2 = splitPacket[2]
	q3 = splitPacket[3]
	q4 = splitPacket[4]
	q5 = splitPacket[5]
	q6 = splitPacket[6]
	q7 = splitPacket[7]
	
	q0 = float(q0)
	q1 = float(q1)
	q2 = float(q2)
	q3 = float(q3)
	q4 = float(q4)
	q5 = float(q5)
	q6 = float(q6)
	q7 = float(q7)
	
	quat_transform(q1, q2, q3, q0)
	quat_transform2(q5, q6, q7, q4)
	
	update_bone_override()


func quat_transform(x, y, z, w):
	#var bone_trans = $"Physical Bone Bone".get_transform()
	var bone_trans = $"Physical Bone shoulderL/Physical Bone upper_armL".get_transform()
	var myRot = Quat(x, y, z, w)
	var myBasis = Basis(myRot)
	# then you can just set the basis property of the bone trans
	bone_trans.basis = myBasis;
	trans = Transform ( myBasis )
	# or return a transform with the desired rotation instead
	# var return_trans = bone_trans.duplicate()
	# return_trans.basis = myBasis;
	# return return_trans;

func quat_transform2(x, y, z, w):
	#var bone_trans = $"Physical Bone Bone".get_transform()
	var bone_trans2 = $"Physical Bone shoulderL/Physical Bone upper_armL/Physical Bone forearmL".get_transform()
	var myRot2 = Quat(x, y, z, w)
	var myBasis2 = Basis(myRot2)
	# then you can just set the basis property of the bone trans
	bone_trans2.basis = myBasis2;
	trans2 = Transform ( myBasis2 )
	# or return a transform with the desired rotation instead
	# var return_trans = bone_trans.duplicate()
	# return_trans.basis = myBasis;
	# return return_trans;

also I did tried the same process for a simple cube and here is the result:

The length of the cube is enough to cover the space between two black blocks, but it gets short when I'm trying to move it also as you can see deformation of mesh is not good at all because it seems a part of a cube is shifted inside. Thanks again for your attention. Kind Regards, Mason

TwistedTwigleg

It is probably because the child bone's transform isn't in global_pose space, so it's origin is offset incorrectly.

Try modifying the global transforms for the bones and using this function to convert it to a global pose:

func global_transform_to_global_pose(p_skeleton, p_transform):
	p_skeleton.global_transform.affine_inverse() * p_transform;

Then you just need to adjust a few functions. Adjustments below:

func update_bone_override():
    # if we are overriding the bone pose...
    if is_bone_overridden == true:
        # set's the bone override transform
        set_bone_global_pose_override(bone_id_to_manipulate, global_transform_to_global_pose(self, trans), 1.0, true);
        set_bone_global_pose_override(bone_id_to_manipulate_2, global_transform_to_global_pose(self, trans2), 1.0, true);

func quat_transform(x, y, z, w):
    var bone_trans = $"Physical Bone shoulderL/Physical Bone upper_armL".get_transform()
    var myRot = Quat(x, y, z, w)
    var myBasis = Basis(myRot)
    bone_trans.basis = myBasis;
    # just assign to bone_trans, as we want to keep the origin (position) of the bone the same
    trans = bone_trans;

func quat_transform2(x, y, z, w):
    var bone_trans2 = $"Physical Bone shoulderL/Physical Bone upper_armL/Physical Bone forearmL".global_transform
    var myRot2 = Quat(x, y, z, w)
    var myBasis2 = Basis(myRot2)
    bone_trans2.basis = myBasis2;
    # just assign to bone_trans2, as we want to keep the origin (position) of the bone the same
    trans2 = bone_trans2;

mec

Hi again, Dear @TwistedTwigleg I really appreciate for your help, there is a real progress with your guides. As you said I did some adjustments in code and there is some improvements , also some new issues : 1. When I'm moving just Left Arm everything is ok except scale (You will notice in first gif) 2. When I'm moving the Left Arm and Forearm the parenting concept problem is still remaining, also the scaling problem too. I will post the script and both condition gifs for better understanding the problem:

extends Skeleton

var loadedScript = load("res://Model/metarig.py")
var loadedScriptInstance = loadedScript.new()
  

var bone_id_to_manipulate = 8
var bone_id_to_manipulate_2 = 9

var is_bone_overridden = false

var q0=0
var q1=0
var q2=0 
var q3=0
var q4=0
var q5=0
var q6=0
var q7=0

var trans
var trans2

var x


func _ready():
	start_bone_override()


func global_transform_to_global_pose(p_skeleton, p_transform):
	x = p_skeleton.global_transform.affine_inverse() * p_transform;


func start_bone_override():
	is_bone_overridden = true


func update_bone_override():
	# if we are overriding the bone pose...
	if is_bone_overridden == true:
		# set's the bone override transform
		global_transform_to_global_pose(self, trans)
		set_bone_global_pose_override(bone_id_to_manipulate, x, 1.0, true);
		
		
		set_bone_global_pose_override(bone_id_to_manipulate_2, global_transform_to_global_pose(self, trans2), 1.0, true);
		global_transform_to_global_pose(self,trans2)
		set_bone_global_pose_override(bone_id_to_manipulate_2, x, 1.0, true);



func stop_bone_override():
	# stop overriding the bone pose every _process call
	is_bone_overridden = false
	# reset the bone pose
	set_bone_global_pose_override(bone_id_to_manipulate, Transform(), 0.0, false)
	set_bone_global_pose_override(bone_id_to_manipulate_2, Transform(), 0.0, false)
	




func _process(_delta):
	var dataPacket = loadedScriptInstance.myProcess()
	var splitPacket=dataPacket.split(',')
	q0 = splitPacket[0]
	q1 = splitPacket[1]
	q2 = splitPacket[2]
	q3 = splitPacket[3]
	q4 = splitPacket[4]
	q5 = splitPacket[5]
	q6 = splitPacket[6]
	q7 = splitPacket[7]
	q0 = float(q0)
	q1 = float(q1)
	q2 = float(q2)
	q3 = float(q3)
	q4 = float(q4)
	q5 = float(q5)
	q6 = float(q6)
	q7 = float(q7)
	
	quat_transform(q1, q2, q3, q0)
	quat_transform2(q5, q6, q7, q4)
	
	update_bone_override()
	
	
func quat_transform(x, y, z, w):
	var bone_trans = $"Physical Bone shoulderL/Physical Bone upper_armL".get_transform()
	var myRot = Quat(x, y, z, w)
	var myBasis = Basis(myRot)
	bone_trans.basis = myBasis;
	trans = bone_trans

	
	
func quat_transform2(x, y, z, w):
	var bone_trans2 = $"Physical Bone shoulderL/Physical Bone upper_armL/Physical Bone forearmL".global_transform
	var myRot2 = Quat(x, y, z, w)
	var myBasis2 = Basis(myRot2)
	bone_trans2.basis = myBasis2;
	trans2 = bone_trans2

and here is the gifs: First one is only for Left Arm which everything is ok but scale

This one is for moving two bones, interesting thing about this one is location of forearm is kind of correct but kind of

Kind Regards, Mason

mec

I Hope it's all clear

TwistedTwigleg

Right, that makes sense with the scale. Scale data is stored in the Basis along with rotation, so when you construct a new Basis you will need to preserve the scale as well.

Try adding this to the global_transform_to_global_pose function:

func global_transform_to_global_pose(p_skeleton, p_transform):
	x = p_skeleton.global_transform.affine_inverse() * p_transform;
	x.basis = x.basis.orthonormalized()

This will set the global pose's scale to be (1, 1, 1), which I think should fix the scale issue you are having. If not, then you'll want to get the global pose scale for each bone, and then apply that, using something like this:

x.basis = x.basis.orthnormalized().scaled(global_pose_bone_scale_here)

mec

Hey @TwistedTwigleg , R U a Orthopedic doctor? Cause you know how to deal with bones ;) appreciation As you said I just added the following line of code and the scaling problem is settled.

x.basis = x.basis.orthonormalized()

and this is the result for a single bone (Left_Arm)

but when I'm trying to move the forearm too , here is what happens

as you can see mesh deformation is not logical and there is no parenting concept too thanks again kind regards Mason

TwistedTwigleg

@mec said: Hey @TwistedTwigleg , R U a Orthopedic doctor? Cause you know how to deal with bones ;) appreciation As you said I just added the following line of code and the scaling problem is settled.
x.basis = x.basis.orthonormalized()
and this is the result for a single bone (Left_Arm)

but when I'm trying to move the forearm too , here is what happens

as you can see mesh deformation is not logical and there is no parenting concept too thanks again kind regards Mason

LOL, no, I'm not a doctor of any type! I've just written too many IK algorithms :lol:

What does your node tree look like? Also, I would double check to make sure that the code is getting the global transform for both bones, rather than the transform, since the global_transform_to_global_pose needs global transforms now.

mec

Hi @TwistedTwigleg , thanks again for your attention Here is my node tree: I forgot to say, I exported the model in blender with glTF Embedded format. I hope it helps to understand the problem cause I'm disturbing you so often. Kind Regards, Mason.

TwistedTwigleg

It looks like the scene is setup fine for the shoulder, upper arm, and fore arm. I wonder if the issue is the use of PhysicalBones then, perhaps their physics update is causing the issue? Looking at the GIF, it seems the issue is that when the upper arm rotates, the fore arm stays in the same position even though it's parent node rotated. This might be because it is a physics node and so it ignores the parent's transform.

If you add a MeshInstance as a child of the upper arm and position it roughly where the fore arm is (like make it a scaled cube that roughly encompasses the size of the fore arm) does it rotate correctly? If it rotates correctly with the upper arm, then it is probably the use of PhysicalBone nodes that is causing the issue since it's a physics node (and therefore doesn't take parent transforms into account).

mec

Hi @TwistedTwigleg Sorry for late reply, I tried adding Mesh Instance as a child for Upper Arm (I didn't write any scripts for that) but it didn't work. I don't know if its a good news or a bad one. Here is what I did: thanks again for your attention. Kind Regards, Mason

TwistedTwigleg

Great, that should help with figuring out the issue.

Does the added MeshInstance node position and rotate itself correctly when the upper arm bone rotates? If it is rotated and positioned correctly, then that would pretty much confirm that the physics in the PhysicalBone3D is causing the issue. If it doesn't work, then it may still be the physics, but it could be something else as well.

The hard part with debugging this kind of thing is that it's very fiddly! Sorry for having to ask so many questions, just trying to figure out where the problem might be :smile:

mec

Thanks for fast respond, @TwistedTwigleg The added MeshInstance is not moving with upper arm, Don't be sorry at all cause It's nice of you Thanks again for your attention Mason

TwistedTwigleg

Well unfortunate that it's not moving since it doesn't get us any closer to a solution.

I'll try to see if I can get a simple scene working using a similar 3D model and rotate it using the code a few posts back and see if I can figure out what is going on :+1: