Unity AR Foundation and CoreML: Hand detection and tracking

0x00 Description

The AR Foundation package in Unity wraps the low-level API such as ARKit, ARCore into a cohesive whole.

The CoreML is a framework that can be harnessed to integrate machine learning models into your app on iOS platform.

This article and the demo project at the end of the article show how to enable the CoreML to work with AR Foundation in Unity. With AR Foundation in Unity and CoreML on iOS, we can interact with virtual objects with our hands.

This article refers to Gil Nakache’s article and uses the mlmodel used in his article. In his article, he describes how to implement these on the native iOS platform with Swift.

Version

Unity Version: 2018.3.13f1

Xcode Version: 10.2.1

The AR Foundation Plugin: 1.5.0-preview.5

iPhone 7: 12.3.1

0x01 Implementation

Import AR Foundation Plugin

For convenience, I use the local package import. This is very simple, just modify the manifest.json file in the Package folder and…