Understanding NVIDIA’s Project GROOT

Discover how NVIDIA’s Project GROOT is reshaping the future of robotics by blending AI and cutting-edge technology. Explore its groundbreaking principles, innovative training methods, and transformative applications in this blog.

General-Purpose Foundation Models

Foundation models, also known as General-Purpose AI or GPAI, are capable of a range of general tasks such as text synthesis, image manipulation, and audio generation. They are trained on broad data and can be adapted to a wide range of downstream tasks. Notable examples include OpenAI’s GPT-3 and GPT-4.

Foundation models are characterized by their scale, which includes a lot of memory, data, and powerful hardware. They rely on transfer learning, which is the application of knowledge from one task to another. These models are sometimes referred to as ‘foundation models’ and are characterized by their widespread use as pre-trained models for other, more specialized AI systems.

For example, a single general-purpose AI system for language processing can be used as the foundation for several hundred applied models (e.g., chatbots, ad generation, decision assistants, spambots, translation, etc.), some of which can then be further fine-tuned into a number of applications tailored to the customer.

Reinforcement Learning

Reinforcement learning is a machine learning technique where an algorithm learns to find the optimal behavior or path in a specific situation through trial and error. It uses algorithms that learn from outcomes and rewards, and can be used for various applications such as robotics, chess, and data processing.

What is Project GROOT?

Project GROOT (Generalist Robot 00 Technology) is a general-purpose foundation model developed by NVIDIA. It aims to transform humanoid robot learning in simulation and the real world. Trained in NVIDIA GPU-accelerated simulation, GROOT enables humanoid embodiments to learn from a handful of human demonstrations with imitation learning and NVIDIA Isaac Lab for reinforcement learning. The GROOT model takes multimodal instructions and past interactions as input and produces the actions for the robot to execute.

How is GROOT trained?

NVIDIA developed Isaac Lab to train GR00T at scale. They also built NVIDIA OSMO, a compute orchestration service that coordinates the training and inference workflows across various NVIDIA systems. These include NVIDIA DGX systems for training, NVIDIA OVX systems for simulation, and NVIDIA IGX and NVIDIA AGX systems for hardware-in-the-loop validation.

NVIDIA Issac Lab

NVIDIA Isaac Lab is a lightweight reference application built on the NVIDIA Isaac Sim platform, specifically optimized for robot learning. It is pivotal for robot foundation model training. Here are some key points about Isaac Lab:

• Isaac Lab is the successor to Isaac Gym and benefits from NVIDIA Omniverse technologies for physics-informed, photorealistic, perception-based reinforcement learning tasks.
• It is an open-source, performance-optimized application for robot learning built on the Isaac Sim platform.
• Isaac Lab optimizes for reinforcement, imitation, and transfer learning, and is capable of training all types of robot embodiments

NVIDIA OSMO

NVIDIA OSMO is a cloud-native workflow orchestration platform that lets you easily scale your workloads across distributed environments-from on-premises to private and public cloud3. Here are some key points about OSMO:

• OSMO provides a single pane of glass for scheduling complex multi-stage and multi-container heterogeneous computing workflows.
• Developers can easily share accelerated computing clusters for workflows across multiple heterogeneous compute nodes in multiple stages, without needing to know Kubernetes3.
• OSMO supports location-agnostic deployment on Kubernetes clusters with mixed compute, such as x86 and Arm and NVIDIA GPU’s for training, inference, or rendering.
• It offers data traceability for auditing deployed models and maintaining data lineage for safety.

Other Notable Humanoid Projects

Ameca by Engineered Arts

Ameca is designed specifically as a platform for development into future robotics technologies, Ameca is the perfect humanoid robot platform for human-robot interaction.

Alter 3 by Osaka University and mixi

Alter 3 is a humanoid robot with a bare body exposing the machine inside, a face without age or sexuality. It was created by roboticist Hiroshi Ishiguro of Osaka University and Mixi Corporation. It is embedded with an artificial neural network developed by artificial life researchers of the University of Tokyo.

ARMAR-6 by Karlsruhe Institute of Technology

ARMAR-6 is a collaborative humanoid assistant robot for industrial environments. It can interact with humans and provide help when needed in a proactive way. The robot is the 6th generation and youngest member of the ARMAR family of humanoid robots developed at the Karlsruhe Institute of Technology (KIT).

Apollo by Apptronik

Apollo was developed from Apptronik’s experience and expertise in building over 10 previous robots including NASA’s Valkyrie robot. Apollo will operate in warehouses and manufacturing plants in the near term eventually extending into construction, oil and gas, electronics production, retail, home delivery, elder care and countless more areas.

Atlas by Boston Dynamics

Atlas is a bipedal humanoid robot primarily developed by the American robotics company Boston Dynamics. The robot was initially designed for a variety of search and rescue tasks.

Beomni by Beyond Imagination

Beomni is a fully mobile robot that is designed to work safely around humans. It is the world’s first fully functional general-purpose robotic system, and the first to be beta tested at a medical facility.

Digit by Agility Robotics

Digit is a humanoid robot designed to navigate our world. It can walk into existing facilities and address the hardest-to-automate portions of your workflow. Digit is made for work, and its technology has been proven in real-world distribution, 3PL and manufacturing sites.

Jiajia by University of Science and Technology of China

Jiajia is designed to hold realistic conversations with humans and displays a wide range of facial expressions while doing so. Its speech is synced with its lips, it blinks and moves its eyes.

Applications of Project GROOT

Project GROOT is designed to understand natural language and emulate movements by observing human actions. This enables robots to quickly learn coordination, dexterity, and other skills in order to navigate, adapt, and interact with the real world. This technology can enhance the capabilities of humanoid robots and make it very easy to develop and deploy them.

As part of the project, NVIDIA announced a new computer, Jetson Thor, for humanoid robots based on the NVIDIA Thor system-on-a-chip (SoC). They also announced significant upgrades to the NVIDIA Isaac robotics platform.

Project GR00T marks a significant step forward in the field of robotics and AI. It’s part of NVIDIA’s initiative to drive breakthroughs in robotics and embodied AI.

How to Access Project GROOT

As of now, NVIDIA has not provided specific details on how to access Project GROOT. However, you can sign up on the NVIDIA Developer website to get notified about Project GROOT’s availability.