Moback and MediaPipe — Google’s open-source hand tracking algorithm

Tracking real-time hand movements has proved to be a challenging computer vision task because often all segments of a hand aren’t visible. Hands movements are also usually quick; it can be difficult for computers to catch them perfectly.

MediaPipe hands is a high-fidelity hand tracking solution that employs machine learning to create a map of the hand using only a smartphone and its camera. It utilizes 3 AI models — a palm detector, a hand landmark model, and a gesture recognizer — that provides an algorithm that’s both fast and accurate.

Moback team implemented a Unity framework for working with MediaPipe which allowed for faster prototyping and development. We also refined and refactored parts of the MediaPipe code to make it compatible with Unity and to surface cleaner data.

To learn more about MediaPipe’s hands visit: https://google.github.io/mediapipe/solutions/hands