Mastering 3D Model Transformation in Unity with Snapdragon Spaces Hand Gestures
The integration of hand gestures in head-mounted mixed reality (MR) devices has revolutionised the way we interact with virtual objects. With Snapdragon Spaces and Unity, controlling the scale, position, and rotation of 3D models has become even more intuitive and natural. In this blog, we’ll explore the capabilities of hand gestures in head-mounted MR devices and how they can be used to transform 3D models in Unity. Whether you’re just starting or looking to enhance your MR development skills, this guide will provide a comprehensive look at how hand gestures can bring your virtual creations to life in a whole new way.
Problem break down:
- Unity Project setup with Snapdragon Spaces.
- Rendering hands in the environment with a collider.
- Creating anchors.
- Detecting interaction with anchors and hand gestures.
- Transforming the 3D model based on hand movements.
The plan is to bring a new level of interactivity to 3D models, By adding a wireframe. The wireframe will have several anchors equipped with colliders to detect any collisions. Render the hand within the environment, equipped with a rigid body and collider. As the hand interacts with the 3D model, any collisions detected will result in real-time transformations to the model, including scaling, rotating, and repositioning.
Unity Project setup with Snapdragon spaces:
Snapdragon Spaces SDK provides multiple perception features for MR Devices that run on a Snapdragon processor. Hand tracking is one of the features of spaces SDK. You can set up the spaces SDK by following the below steps.
Create a new 3D Project in Unity and follow the below guide to import the Snapdragon Spaces SDK.
The above instructions will make necessary changes to player settings.
Rendering hands in environment with collider:
Import the code samples that come along with the Snapdragon Spaces and open the Hand Tracking scene. Build and scene and open the app. Once the hands are detected, you may see something like the below image.
Note: The position of virtual hands seems a bit off-place due to a bug in the recording software. While trying out the hands and virtual hands overlap accurately.
Let’s understand the outermost layer of Snapdragon Spaces Hand tracking we work on.
The Hand Tracking Sample Scene has an AR Session Origin, to which the Spaces Hand Manager script is attached. This script is responsible for monitoring and tracking hand movements in the real world. It requires a hand prefab, which serves as the visual representation of a single hand. Upon detection of both hands, the script will instantiate two hand game objects to accurately reflect their movements.
Open the Default Spaces Hand prefab by clicking on it. It should have two scripts attached to it Spaces Hand and Spaces Hand Joint Visualiser. Spaces Hand scripts contain essential information about the hand, such as whether it is the left or right hand, the position of each joint, and the type of hand gesture. Spaces Hand Joint Visualiser consumes this information and draws the virtual Hands.
The update method of Spaces Hand joint visualiser renders a mesh at the position of each joint. However, being simply a mesh, it does not possess the ability to interact with colliders. To overcome this limitation, we will modify the script to use joints with colliders and rigid bodies. This way, we can easily detect any interactions between the hands and the colliders, allowing us to determine which anchors are being locked.
Create a joint prefab which has a collider and rigid body. Make sure to turn off the gravity and remove drag and angular drag.
Attach 25 of these joints to the hand. All these joints represent one of the joints in spaces hand. And add the custom joint visualiser script we wrote.
using UnityEngine;
using UnityEngine.XR.ARSubsystems;
using Qualcomm.Snapdragon.Spaces;
public class HandJointVisualizer: MonoBehaviour
{
private SpacesHand _spacesHand;
private bool _isHandActive = false;
private void Start() {
_spacesHand = GetComponent<SpacesHand>();
}
private void Update() {
switch (_spacesHand.IsLeft)
{
case true when !StateManager.Instance.isLeftHanded():
case false when StateManager.Instance.isLeftHanded():
return;
}
if (_spacesHand.trackingState == TrackingState.None)
{
ActivateOrDeactivateJoints(false);
return;
}
if(!_isHandActive) ActivateOrDeactivateJoints(true);
for (var i = 0; i < _spacesHand.Joints.Length; i++)
{
transform.GetChild(i).transform.position = _spacesHand.Joints[i].Pose.position;
}
}
private void ActivateOrDeactivateJoints(bool state)
{
_isHandActive = state;
for (var i = 0; i < transform.childCount; i++)
{
transform.GetChild(i).gameObject.SetActive(state);
}
}
}Please note that in this current code, the user has the option to disable one hand based on their preference. This is due to the design of the HMD being used for this project, which has a hand-held component and therefore only allows for the use of one hand for active object manipulation.
Conclusion: After implementing the above changes. The code should now be configured with Snapdragon Spaces with hand tracking enabled, and colliders that facilitate interaction with virtual objects.
Creating anchors:
Anchors serve as a means to manipulate 3D models. In this implementation, three distinct types of anchors have been created for repositioning, rotating, and scaling. A cubic wireframe was created, with its centres serving as repositioning anchors, edges as rotating anchors, and vertices as scaling anchors.
Here I have only 4 Edge anchors since I only rotate my 3D Model along Y-Axis. If you want to rotate the model over all the axes then 8 more edge anchors may be needed.
The above image shows the structure of the anchors forming the wireframe.
Each of these anchors are equipped with Colliders with Is Trigger property enabled to detect collisions.
The wireframe will not have the cube in the centre. A 3D Model will be placed in the place of cube in the final result.
Detecting interaction with anchors and hand gestures.:
Interaction with anchors:
The hand joints are equipped with a rigid body and a collider and all the anchors have a collider. So we can detect these collisions whenever the hand enters the collider. There are 6 Repositioning Anchors, 4 Rotation Anchors and 8 Scaling anchors. To manage all these and orchestrate the whole flow I have State Manager. This state manager at any given point in time knows both the active anchors and position of Hands.
Furthermore, to manage collider triggers, a script must be attached to each anchor. This script functions as a mediator, transmitting information to the State Manager whenever an on-trigger event is activated.
using System.Collections.Generic;
using System.Linq;
using Qualcomm.Snapdragon.Spaces;
using UnityEngine;
public class Anchor : MonoBehaviour
{
[SerializeField] protected Material _default;
[SerializeField] private Material _selected;
[SerializeField] private Material _locked;
[SerializeField] protected GameObject _model;
protected List<string> triggered = new List<string>();
protected Renderer _renderer = null;
protected SpacesHand _holdingHand = null;
private void Start()
{
_renderer = GetComponent<Renderer>();
}
private void OnTriggerEnter(Collider other)
{
if (_holdingHand) return;
_renderer.material = _selected;
StateManager.Instance.AddActivatedAnchor(this);
if (!triggered.Contains(other.gameObject.name)) triggered.Add(other.gameObject.name);
}
private void OnTriggerExit(Collider other)
{
triggered = triggered.TakeWhile(t => t != other.gameObject.name).ToList();
if (_holdingHand) return;
if (triggered.Any()) return;
_renderer.material = _default;
StateManager.Instance.RemoveActivatedAnchor(this);
}
public void LockAnchor(SpacesHand hand)
{
_renderer.material = _locked;
_holdingHand = hand;
}
public void UnLockAnchor()
{
_holdingHand = null;
if (!triggered.Any())
{
_renderer.material = _default;
return;
}
_renderer.material = _selected;
}
protected SpacesHand.Joint GetJointByType(SpacesHand.JointType type)
{
if (!_holdingHand) return null;
return _holdingHand.Joints.First(joint => joint.Type == type);
}
}The script also implements a dynamic material change for visual feedback on trigger enter and trigger exit events. Additionally, a comprehensive list of joints interacting the collider is maintained, ensuring correct visual feedback even after a joint leaves the collider.
The script also has Lock and Unlock Anchor that we address in a bit after discussing the state manager.
Detecting hand gestures and locking the anchor:
The goal is to transform an object whenever a hand enters a collider and a grab or pinch gesture is executed. With the ability to detect interactions in place, the next step is to construct a logic to determine when a user initiates and releases a grab or pinch. State Manager comes into play here.
The State Manager integrates with the Spaces Hand Manager script attached to the AR Session Origin by setting a listener. This listener is dynamically triggered by the Spaces Manager whenever updates to the hand positions occur.
using System.Linq;
using Qualcomm.Snapdragon.Spaces;
using UnityEngine;
using UnityEngine.SceneManagement;
using Toggle = UnityEngine.UI.Toggle;
public class StateManager : MonoBehaviour
{
[SerializeField] private Toggle isLeftHandedToggle = null;
public static StateManager Instance;
private Anchor _activatedAnchor = null;
private Anchor _locked = null;
private SpacesHandManager _spacesHandManager;
private void Start()
{
if (Instance == null) Instance = this;
_spacesHandManager = FindObjectOfType<SpacesHandManager>();
_spacesHandManager.handsChanged += LockTarget;
}
private void LockTarget(SpacesHandsChangedEventArgs args)
{
if (args.added.Count < 1 && args.updated.Count < 1 && _locked)
{
_locked.UnLockAnchor();
_locked = null;
}
var hands = args.added.Concat(args.updated).ToList();
foreach (var hand in hands)
{
switch (hand.IsLeft)
{
case true when !Instance.isLeftHanded():
case false when Instance.isLeftHanded():
continue;
}
if (IsHolding(hand))
{
if (_locked || !_activatedAnchor) continue;
_locked = _activatedAnchor;
_locked.LockAnchor(hand);
}
else
{
if (!_locked) continue;
_locked.UnLockAnchor();
_locked = null;
_activatedAnchor = null;
}
}
}
private bool IsHolding(SpacesHand hand)
{
return hand.CurrentGesture.Type == SpacesHand.GestureType.GRAB
|| hand.CurrentGesture.Type == SpacesHand.GestureType.PINCH;
}
public bool isLeftHanded()
{
return isLeftHandedToggle.isOn;
}
public void AddActivatedAnchor(Anchor anchor)
{
if(_locked) return;
if(_activatedAnchor && _activatedAnchor.gameObject.name == anchor.gameObject.name) return;
_activatedAnchor = anchor;
}
public void RemoveActivatedAnchor(Anchor anchor)
{
if(_locked || !_activatedAnchor) return;
if(_activatedAnchor.gameObject.name != anchor.gameObject.name) return;
_activatedAnchor = null;
}
public void Reset()
{
SceneManager.LoadScene(SceneManager.GetActiveScene().name);
}
}In the above script in the Start method we are adding LockTarget method as the listener.
The State Manager dynamically updates the active anchor whenever a collider of the anchor is triggered. Once a grab or pinch gesture is made, the active anchor becomes locked, and when the grab or pinch gesture is released, the lock on the anchor is lifted. The lock method in anchor script expects the hand that initiated the lock, allowing for seamless transformation of the model.
Importantly, the anchor remains locked even upon trigger exit to ensure continued transformation as the hand moves, as there is a high likelihood that the hand may move outside the collider. This approach maximizes the accuracy and consistency of the transformation process.
Transforming the 3D model based on hand movements:
With the ability to detect collisions and hand interactions with the anchors and lock the anchor, the final piece of the puzzle is the transformation of the model as the hand moves. The implementation of this component completes the seamless integration of hand movements and object transformations.
The system employs three distinct types of anchors Repositioning, Rotating, and Scaling. Each anchor is equipped with their respective script that enables manipulation of the model according to hand movement.
Repositioning:
Repositioning is the simplest of all transformations, requiring only the placement of the model at the location of the hand.
To facilitate the repositioning process, a new script should be created and the Repositioning class should extend the Anchor class to incorporate the previously developed functionality.
public class RepositionAnchor : Anchor
{
private void Update()
{
if (!_holdingHand)
{
if (triggered.Count < 1) _renderer.material = _default;
return;
}
var palmPosition = GetJointByType(SpacesHand.JointType.PALM).Pose.position;
_model.transform.position = palmPosition - (transform.localPosition * _model.transform.localScale.x);
}
}To implement repositioning, the local position of the anchor multiplied by the scale of the model is subtracted from the palm position. This approach ensures that after repositioning, the hand will touch the anchor instead of the model itself. Additionally, the multiplication by scale corrects the distance between the centre of the model and the anchor, taking into account any changes in scale that would not affect the local position of the anchor relative to its parent.
Attach this script to all the RepositionAnchors of the model wrapper.
Rotating:
To determine the necessary rotation of the model, a GameObject called a Compass is added to the centre of the model. Each edge has its Compass that continuously looks at the edge. Upon locking the edge, the Compass looks at the hand. The difference in the angle that Compass has rotated is then added to the model’s rotation.
using UnityEngine;
public class RotationAnchor: Anchor
{
[SerializeField] private GameObject compass;
private void Update()
{
if (!_holdingHand)
{
if (triggered.Count < 1) _renderer.material = _default;
return;
}
var palmPosition = GetJointByType(SpacesHand.JointType.PALM).Pose.position;
var defaultCompassRotation = compass.transform.localEulerAngles;
compass.transform.LookAt(palmPosition);
var y = compass.transform.localEulerAngles.y - defaultCompassRotation.y;
_model.transform.localEulerAngles += new Vector3(0f, y, 0f);
compass.transform.localEulerAngles = defaultCompassRotation;
}
}Similar to repositioning anchor a new script should be created for this logic and should be added to the edge anchor.
Scaling:
Just like repositioning and rotation, a new script needs to be created and attached to all of the scaling anchors.
In the current case, I want to scale the object symmetrically so the scale on all the axes will be the same. If we need to scale an object over Y-Axis between two points, we can calculate the distance between the 2 points and divide it by 2. That will become the scale of the object on the Y-Axis.
Scaling can be solved in two ways, depending on the shape of the wireframe. For a cubical wireframe, the scale can be calculated using the Pythagorean theorem. Yes, You read that correctly. For a cuboidal wireframe, the scale can be determined using the properties of similar triangles.
In the above diagram, for calculating the scale of a cubical wireframe we just need to calculate the length of the hypotenuse (AB) and divide it by 2. The number we get will be the scale of the model in all 3 axes.
The cuboidal frame forms a pair of similar triangles ABC and ADE. So the ratio of AD/AB should be equal to DE/BC. By BC will be the distance between 2 anchors on the vertices. Equating the above equation gives the length of BC and BC/2 will give the scale of the model. That scale can be applied to all axes.
Note: Below code is applicable only for scaling cubical wireframes.
using System;
using UnityEngine;
public class ScalingAnchor: Anchor
{
private void Update()
{
if (!_holdingHand)
{
if (triggered.Count < 1) _renderer.material = _default;
return;
}
var palmPosition = GetJointByType(SpacesHand.JointType.PALM).Pose.position;
var modelPosition = _model.transform.position;
var side = Vector3.Distance(palmPosition, modelPosition);
var length = Math.Sqrt(side * side + side * side);
var magnitude = (float)length / 2;
_model.transform.localScale = Vector3.one * magnitude;
}
}
Attach a model as a child to the wireframe and adjust the scale so that the model is enveloped by the wireframe. All these scripts alter the model wireframe, since the model is a child of the wireframe the model should also alter along with wireframe.
Build the application and run it in a MR Device.
In conclusion, the integration of hand gestures in MR is a step forward towards a more intuitive and immersive experience. I hope that this blog has provided a comprehensive overview of the process and has inspired you to explore this technology further.
