WebAR Face Tracking with 10 lines of code
MindAR is an open source SDK for developing web based augmented reality applications extremely easily with as minimal as 10 lines of code. It supports different tracking abilities, and in this article, I’ll focus on Face Tracking.
The face tracking module is based on Google mediapipe, which offers state-of-the-art Machine Learning solutions in numerous tasks. The tracking quality is comparable to, if not better than, commercial alternatives.
Why not using mediapipe directly?
The first question is: Why don’t we use mediapipe directly then?
Mediapipe facemesh javascript solution returns a list of landmarks in the format of 2D or 3D coordinates. While drawing something primitives like dots or lines is not difficult, implementing more complex 3D effects like augmenting 3D objects like hats or glasses is not straight forward. For application developers, we normally rely on high level 3D rendering frameworks like ThreeJS or ARAME.
AFRAME and ThreeJS
MindAR bridged this gap by integrating the core tracking module with web 3D rendering frameworks and provide high level APIs for application developers. Using the provided AFRAME extension, a face AR applicaion can be as simple as this:
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<script src="https://cdn.jsdelivr.net/gh/hiukim/mind-ar-js@1.1.4/dist/mindar-face.prod.js"></script>
<script src="https://aframe.io/releases/1.2.0/aframe.min.js"></script>
<script src="https://cdn.jsdelivr.net/gh/hiukim/mind-ar-js@1.1.4/dist/mindar-face-aframe.prod.js"></script>
</head> <body>
<a-scene mindar-face embedded vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false">
<a-camera active="false" position="0 0 0"></a-camera>
<a-entity mindar-face-target="anchorIndex: 1">
<a-sphere color="green" radius="0.1"></a-sphere>
</a-entity>
</a-scene>
</body>
</html>
This is a 100% complete source code that can be deployed directly as a static website. Here is a live demo. A step-by-step quick start guide can be found in MindAR documentation.
Setting aside all the standard and copyable HTML stuff, the actual code for the above application is merely:
<a-scene mindar-face ....>
... <a-entity mindar-face-target="anchorIndex: 1">
<a-sphere color="green" radius="0.1"></a-sphere>
</a-entity>
</a-scene>
This is a minimal face AR effect, in which a green sphere is attached on the face anchor point 1 (i.e. nose tip). The result is something like this:
You can also choose to attach 3D models (e.g. accessories) on other anchor points like eyes, forehead, mouth, etc.
AFRAME is the easiest way to kickstart a project even for someone without programming experiences. For developers who can handle basic level of coding, MindAR also provides a ThreeJS API for more advanced effects.
Last but not least…..
MindAR is open source under MIT license. It’s completely free to use! Here is the source repo:
