MyoChallenge ’23: A Retrospect
TL;DR: Exploring human motor control through digital twins offers a safe and cost-effective method, utilizing machine learning for the synthesis of behaviors. The introduction of MyoSuite in 2021 and MyoChallenge in 2022 has established a connection between AI and biomechanics experts to simulate complex human actions. In MyoChallenge 2023, participants were tasked with addressing both manipulation and locomotion challenges using the MyoArm and MyoLeg musculoskeletal models from MyoSuite. Engaging over 50 teams and receiving more than 500 submissions, we gathered policies that adapt to unseen conditions and variations. This achievement marks a significant step towards replicating human-like behavior control. We look forward to seeing more amazing results in MyoChallenge 2024!
Human beings are capable of unparalleled feats of motor behaviors and are capable of generalizing them to novel situations. This skill set, derived from synthesizing years of experience into a generalizable machinery within a complex, overactuated system, highlights the distinct nature of human movement, a product of millions of years of evolution. The underlying mechanism enabling such nimble and expressive moments has been an active area of study for multiple communities such as biomechanics, neuroscience, robotics, and animation to name a few.
This leads us to create MyoSuite in 2021, which is a collection of environments and tasks to be solved by a contact-rich musculoskeletal simulation that captures the physiological realism of a human body. Today, MyoSuite includes over 300 tasks, including physiological variations and exoskeleton integration. By developing this accurate digital replica of the human body that adheres to its physical constraints, MyoSuite’s goal is to provide researchers and scientists across diverse fields a common playground for experimentation and knowledge sharing. The MyoSuite platform further exposes these twins to data-driven machine-learning techniques for behavior synthesis. MyoSuite enables risk-free and cost-effective exploration of different skills and underlying mechanisms, leading to better-informed decisions and theories for real-world applications. With the advancements in machine learning algorithms, particularly reinforcement learning and data collection at scale, we see the potential of leveraging them in combination with digital twins to understand and perform complex human behaviors in a faster and cheaper way leading to significant advancements in sports, medicine, and rehabilitation science, as well as to advance our understanding of the mechanisms behind human movement.
Nevertheless, within those frameworks, how can we achieve robust controllers for overactuated systems that generalize to unseen variations? After all, human children can do so effortlessly. To address this difficult question of human motor control, multidisciplinary effort from respective experts is needed in an open-source environment. Since 2017, the Learning to Run competition hosted at NeurIPS has been at the forefront of this effort. Over the years, it continues to expand to include AI For Prosthetics (2018) and Learn to Move — Walk Around (2019), pioneering the approach to under human motor control through reinforcement learning with osim-rl. Unfortunately, making these challenges work on a larger scale has proven difficult.
To address this, we launched the MyoChallenge host on the EvalAI platform to tackle the complexity of human dexterity and agility. Last year’s MyoChallenge’22, as part of the NeurIPS 2022 competition track, was launched to further our understanding of human dexterity. Acquisition of physiological dexterity was evaluated using tasks: solving both single and double object manipulation using the MyoHand: reorienting a die to a goal configuration and simultaneous rotation of two balls over the palm. The participating teams from last year demonstrated how reinforcement learning techniques with a focus on curriculum learning, reward shaping, and exploration strategies can be used to tackle the complexity of hand object manipulation. The results highlighted the progress made in using data-driven methods for musculoskeletal control tasks in high-dimensional environments. Nevertheless, the challenges associated with the generalization of musculoskeletal agents remain unresolved.
This year, we introduced the MyoChallenge’23 as part of the NeurIPS 2023 Competition track to continue the exploration of human motor control. This new challenge aims to push the motor control skills of our digital twin to a higher level by requiring difficult feats of both dexterity and agility so that we can achieve generalization and versatility beyond traditional biomechanical controllers. For this reason, this year’s competition contained two tracks:
- Manipulation track: coordination of both small and large muscles to enable complex dexterous object manipulation
- Locomotion Track: nimble and agile dynamic control and decision-making in the lower body, adapting to goal objectives with increasing difficulties with the presence of other agents or external factors.
Manipulation Track: Object Manipulation and Placement with MyoArm
The goal of this track is to develop a policy capable of controlling the manipulation agent to interact with complex objects that require deleterious manipulations. In contrast to the previous year, participants needed to control the entire arm — a much more difficult task than controlling the single hand of the previous year’s challenge — while the policies were evaluated on unseen test objects. It is easy for humans to blindly grasp and relocate an unknown object with a bit of effort, but this skill has never been achieved before in a musculoskeletal simulation of the full arm.
In this track, the agent controls the MyoArm (63 muscles and 27 DoF) and manipulates the arm towards a random object, which has to be stably grasped and moved into a receptacle bin. The policy was evaluated based on the percentage of successful episodes and the overall muscular effort.
Phase 1
The agent needs to pick up an object initially located near the hand of MyoArm and place it within a target bin whose location is randomly positioned over the table surface.
Phase 2
In the second phase, the participants are introduced to objects with new geometries and physical properties (e.g., mass and friction). Additionally, the object and MyoArm’s initial configuration is randomized.
Locomotion Track: Bipedal Agility with MyoLegs
This track, inspired by the World Chase Tag Competition, is a completely new challenge that focuses on lower body control. Featuring the novel 80 muscle-driven bipedal MyoLeg model, a randomly changing terrain, and an opponent with several different behaviors, participants have to maintain stable and agile control of the MyoLeg while generating successful high-level strategies to win. Combining high and low-level reasoning capabilities, the participants are faced with a randomized opponent they have to overcome.
Phase 1: Chasing an opponent
The agent has 20 seconds to chase the opponent within a flat arena. They are evaluated based on the time required to reach the opponent. The opponent is either stationary or performs random movements.
Phase 2: Evading & Chasing
In the second phase, the agent has to alternately chase and evade the opponent within an arena with uneven terrain. Not only is the terrain randomly changing, but participants have to either evade or chase the opponent, who features unseen behaviors during the performance evaluation. In the evading task, the agent has to avoid the opponent as long as possible without leaving the arena.
Engagements and Results
This year’s MyoChallenge has drawn in a total of 59 teams from over fifteen countries, resulting in a remarkable 536 submissions throughout the two phases. This widespread competition has also yielded remarkable results for MyoSuite, with over 6,000 total downloads during the competition phase, underlining its growing impact in the field.
Furthermore, over 70% of our participants are newcomers to the challenge, demonstrating its growing appeal within a broader community. Among these participants, we’re proud to welcome 16.7% post-graduate researchers 50% graduate students, and one-third of master-level students.
We are especially thrilled about the large number of newcomers, as we made a substantial effort to render this year’s challenge more accessible, executing on feedback we gathered from the previous year. This year, we provided automated scripts for the submission of solutions, detailed tutorials, several pre-trained baselines, colab notebooks, and virtual coaching sessions with interested participants.
Manipulation Winning Solutions
Team Lattice from EPFL, whose members consist of Alberto Chiappa, Alessandro Marin Vargas, and Alexander Mathis, stood out in the manipulation track by reaching a final score of 34% and 0.052 muscular effort during evaluation. Despite only focusing their efforts quite late in the challenge on the manipulation task, they were quickly able to climb the leaderboard and finally win with the best policy.
Their recipe for winning includes the PPO algorithm, curriculum learning, reward shaping, and LATent TIme-Correlated Exploration (Lattice), a method which they later published at the main track of the NeurIPS 2023 conference.
Lattice exploration is a new technique that enhances the way algorithms learn from random data or noise by using connected patterns (i.e., correlated weights) found in the final stage of the system’s decision-making process (i.e., policy network). By perturbing muscle in a highly correlated manner, they are able to explore in a more coordinated, and effective, way than comparable uncorrelated exploration methods. Similar strategies were found in DEP-RL, one of the finalists of last year’s track, and in the previous OpenSim-based NeurIPS competitions. It’s generally very difficult to find out how to move in the world by randomly pulling on single muscles at a time, so methods that figure out how to explore with entire muscle groups can achieve more meaningful behaviors.
Another important feature central to achieving those results is curriculum learning, which is analogous to how children start with easy movements before trying to run. Similar to SDS, the algorithm proposed by the team in the MyoChallenge 2022 (back then under the name of stiff_fingers), the agent benefited from starting with easier variations of the task, before taking on the real challenge environment.
In contrast to last year, the team relied more on tuning reward functions to promote the achievement at each iteration as well as highly parallelized environment setups, leveraging the high computational speed of the MuJoCo engine even more. They did not make use of prior data or mocap recordings.
During the MyoChallenge, the Lattice team showed that the lattice-PPO policy outperforms unstructured exploration in both reach and object manipulation tasks to achieve higher rewards while being more energy-efficient to win the competition. We are excited to see the success of the Lattice algorithm as an encouraging step towards realizing embodied AI, pushing us closer to achieving human-like dexterity and biological motor learning.
Locomotion Winning Solutions
The winning locomotion solution developed by Seoul National University’s GaitNet team, with team members Jungnam Park and Jungdam Won, is extremely impressive. Considering the track was only added this year, they achieved a 66.6% success rate with 0.63 effort, leading the second place by over 30%. They used the Generative GaitNet algorithm, which is a pre-trained, cohesive system of artificial neural networks that have been trained in a continuous 618-dimensional domain. This domain includes gait conditions such as stride and cadence as well as anatomy conditions such as mass distribution, body proportion, bone deformity, and muscle deficits.
The GaitNet framework generates a diverse array of human gaits from single motion clips of a reference gait cycle, which act as a guide for cyclical patterns. These gaits go beyond mere motion tracking and imitation learning but can diverge from the original motion using biologically inspired rewards and incorporate hierarchical motion displacement mapping and adaptive phase stepping. These latter two methods are implemented through an innovative Cascaded Subsumption Network (CSN) architecture developed also by the GaitNet Team. CSN adeptly captures the interplay between muscle activations and their impact on the simulated gait hierarchically and ensures that previous learning is not lost. In contrast to using curriculum learning (CL) with a simpler, non-layered network, CSN maintains the integrity of the base network while adding and learning from subsequent layers.
The demonstrated techniques enable the agent to chase and evade its opponent with biomimetic agility, as showcased in the video above. This proves the exciting potential of imitation learning to transcend its initial reference tasks and adapt to a wider range of action spaces, achieving generalization in the reinforcement learning agent.
Unseen in previous years or other teams, the locomotion track teams made ample use of imitation learning. Some of the training procedures are reminiscent of the learning of children, starting with simple maneuvers such as standing and slowly walking in different directions after observing their parents. While the digital agents learn from datasets instead of human teachers, they start with the same imitation of behaviors, before going beyond mere replication through a gradual increase in task difficulty. With this learning curriculum, the agents can go from imitation to generalization, which is immensely difficult for digital and flesh-and-blood learners alike. We cannot wait to see the application of the GaitNet algorithm in generalizing a variety of other locomotion tasks, including stair-climbing, sports activities, and more. Well done team GaitNet!
We launched this year’s MyoChallenge competition to leverage the global machine-learning community’s expertise through AI-driven approaches to advance our understanding of how humans coordinate muscles to achieve complex movements. The winning solution from the manipulation track showed that in addition to MyoDex, MyoArm is also capable of grasping and lifting novel objects under a unified policy. Beyond our expectations, the newly introduced locomotion track already received mature solutions from the community that demonstrated human-level agility in unseen terrains and randomized initial conditions. Furthermore, MyoChallenge this year was able to lower the entry barriers by providing computational resources to individuals lacking them, creating personalized tutorials, and supplying baselines so even high school students can participate. We hope to continue MyoChallenge as a power engine for new insights and methodologies to solve the mystery of human motor control and to achieve our final goal of closing the gap between lab research and clinical practice.
MyoSymposium ’23
As part of MyoChallenge, aimed at advancing our understanding of human movement for the betterment of rehabilitation and assistive technologies, we organized the MyoSymposium ’23 at the Neural Information Processing Systems Conference (NeurIPS’23).
During MyoSymposium ’23, we were honored to feature renowned experts in AI, biomechanics, and neuroscience. Professor Dario Farina delivered the keynote address, joined by distinguished scholars and researchers: Professor Andrea D’Avella, Professor Scott Delp, Sasha Salter, and Richard Warren from Meta.
Another highlight of MyoSymposium was undoubtedly the presentations from the winning teams of both competition tracks. The winners shared invaluable insights into the strategies and methodologies that set them apart from the rest of the competitors.
During the poster session, other participants of the MyoChallenge also showcased their solutions. Notably, Team JustRandom (from Tsinghua University), who earned the Student Award, and Team MSKBioDyn, who secured the second spot in the locomotion track, were among the presenters. Additionally, members of the MyoSuite team also shared their ongoing research using MyoSuite as a simulation platform.
This workshop allows us to bring together scholars and experts in the fields of biomechanics, ML, neuroscience, and health care. We’re thrilled to witness how MyoChallenge has contributed to the creation of advanced algorithms and moves us closer to achieving human-like dexterity.
Looking Ahead
As MyoChallenge ’23 concludes, we are thrilled to announce that its influence will persist into the future. Firstly, all MyoChallenge ’23 models and environments have been integrated into the latest version of MyoSuite, enhancing our digital twin library. Secondly, we are excited to unveil MyoChallenge ’24 soon, so please stay tuned for updates on the themes via our Twitter announcements.
Lastly, we are excited to announce the launch of the MyoChallenge Podcast in December of the previous year. Hosted by the University of Twente Robotics Center, this podcast aims to uncover the stories and motivations behind the creation of MyoSuite and MyoChallenge.
The podcast features discussions with the founders of MyoSuite — Professor Massimo Sartori, Vittorio Caggiano, and Vikash Kumar. They share insights into how AI, digital twin technology, and neuro-mechanics reshape healthcare and enhance our understanding of human-motor control. Additionally, the podcast provides a closer look at our global competition, MyoChallenge, with a chance to hear from the winners personally about their innovative solutions. Don’t miss our chance to learn about the potential of AI and engineering in healthcare by tuning in to our podcast!
Thank you to our generous sponsors throughout the competition!
Written by: Pierre Schumacher & Cheryl Wang :)
