Predicting Someone’s Movement Before They Move
At Cionic, we are building a platform for human augmentation. The platform consists of three pillars: analysis, prediction, and augmentation.
Pillars of the Cionic Human Augmentation Platform
- ANALYZE how someone is moving
through multiple sensor modalities - PREDICT what the person intends to do next
through predictive modeling and AI - AUGMENT the person to achieve their intended movement
through modalities ranging from FES to exoskeletons
These are the necessary ingredients to help people transcend physical limitations.
Sensorized Clothing
For the first pillar — analysis — we have built clothing that is fully sensorized with both IMU and EMG sensors. Each of these sensors provide complementary but distinct measurements of human movement.
IMUs, or inertial measurement units, are sensors that use a combination of accelerometers and gyroscopes to measure motion. An IMU is how your fitness tracker or smartphone counts how many steps you did.
EMG, or electromyography, sensors measure muscle electrical signals. These signals can be picked up at the surface of the skin whenever a muscle contracts. EMG is often used by doctors to diagnose movement disorders or conditions.
EMG sensors are harder to productize than IMUs, but we invested the effort to include it in our platform because EMG provides a critical piece of information that IMUs cannot: a user’s movement intention.
How early can we predict a person’s movement?
To showcase EMG’s ability to predict a user’s intent, we ran an experiment. Our goal was to see how early we could predict a person’s movement using IMU versus EMG signals.
We trained recurrent neural networks (RNNs) to predict different movements types, specifically side steps, forward steps, back steps, and center stomps from a neutral position. The beginning of all of these movements looks nearly identical.
For the EMG-only model, we fed the 16 EMG leg signals into the model (8 from each leg). The IMU-only model was fed 6 quaternion-formatted sensor streams (one from each foot, shin, and thigh).
Foot pressure insoles were used as the gold standard label of when the movement had started. Foot pressure data allowed us to see exactly when the foot started lifting off the ground.
The results of the experiment were stark. EMG accurately predicted which movement the person was about to do 250ms, on average, before the foot started to lift off the ground.
IMU could also accurately predict the person’s movement, but only 100ms, on average, after the foot started to lift off the ground.
Here is a video showing the EMG model movement predictions on a test case:
EMG is able to do this early prediction because there is a delay between the onset of a muscle’s electrical activity and its subsequent force production and motion [1]. This delay is called electromechanical delay (EMD).
IMU, on the other hand, measures motion as it happens. In the case of side, forward, back, and center steps, the movements all look similar at the start: weight shifting to one leg and the other foot lifting off the ground. It’s not until a little bit later in the movement when the foot starts to go slightly forward or sideways that the IMU model can distinguish between the movements.
What this means for human augmentation?
For a human augmentation platform, predicting a user’s intention is critical. It’s critical because the intended and actual movements do not always match.
For example, let’s imagine an exoskeleton that is supposed to help people lift heavy things. A person tries to lift a heavy box to a higher shelf, but they can’t even begin to move the box because it is too heavy. A model based solely on IMU motion signals will not be able to infer that the person wants to lift up their arms. EMG signals, on the other hand, can easily see the biceps firing and tell the exoskeleton to help the person lift. EMG signals can also tell when muscle fatigue is setting in and increase the amount of assistance the exoskeleton is providing to the person.
For people with movement disorders, there are often examples like this one where the intended and realized movement do not match. That is why intention prediction is critical.
If we can sense how a person is moving, predict what they intend to do next, and then augment their body to achieve that movement, then we can help folks transcend their physical limitations.
