Tensorflow Lite(TFLite) with Golang

Tensorflow Lite commonly known as TFLite is used to generate and infer machine learning models on mobile and IoT(Edge) devices. TFLite made the on-device(offline) inference easier for multiple device architectures, such as Android, iOS, Raspberry pi and even backend servers. With TFLite you can build a lightweight server based inference application using any programming language with lightweight models, rather than using heavy Tensorflow models.

As developers, we can simply use existing optimized research models or convert existing Tensorflow models to TFLite. There are multiple ways of using TFLite in your mobile, IoT or server applications.

• Implement the inference for different architecture(Android, iOS etc…) using the standard libraries, SDKs provided by TFLite.
• Use the TFLite C API for inference along with platform independent programming language like Golang. And cross-compile for platforms like Android, iOS etc...

In this post I’m going to show case the implementation of TFLite inference application using platform independent language Golang and cross-compiling to a shared library. Which then can be consumed by Android, iOS etc…

First thanks to mattn who created the TFLite Go bindings and you can find the repo here. We will start the implementation of a simple Golang application for TFLite inference(You can find the example here). Here I’m using a simple text classifier which will classify to ‘Positive’ or ‘Negative’.

Here is the classifier.go, which has Go functions and are exported for use by C code.

Build() : function will initialize all the object references in go runtime environment and return the memory pointer for future use.

Classify(appPointer unsafe.Pointer, word *C.char) : function will accept the input of appPointer returned in Build() function and word a C string.

//export : is used to export Go functions to be used in C code.

More info about cgo you can find here.

Now we already have the Golang implementation of text classification in classifier.go. Next big challenge is the cross-compilation of above text classifier to a shared library. In Golang you can build your application targeting to a specific operating system and an architecture. Important Go commands used in our example.

GOARCH      - target architecture
GOOS        - target operating system
buildmode   - which kind of object file is to be built. More info here
CGO_ENABLED - Whether the cgo command is supported. In this case it's 1

You can find the supported GOOS and GOARCH info here or you can get it by running the below command.

go tool dist list | column -c 75 | column -t
Output :
aix/ppc64        freebsd/amd64   linux/mipsle   openbsd/386
android/386      freebsd/arm     linux/ppc64    openbsd/amd64
android/amd64    illumos/amd64   linux/ppc64le  openbsd/arm
android/arm      js/wasm         linux/s390x    openbsd/arm64
android/arm64    linux/386       nacl/386       plan9/386
darwin/386       linux/amd64     nacl/amd64p32  plan9/amd64
darwin/amd64     linux/arm       nacl/arm       plan9/arm
darwin/arm       linux/arm64     netbsd/386     solaris/amd64
darwin/arm64     linux/mips      netbsd/amd64   windows/386
dragonfly/amd64  linux/mips64    netbsd/arm     windows/amd64
freebsd/386      linux/mips64le  netbsd/arm64   windows/arm

Here is the Makefile I created to generate shared library for Linux, Macos, Android and iOS.

Finally, you can use above generated shared libraries(.so or .a) in Android and iOS. In Android, you can use shared library using JNI(Java Native Interface) and in iOS you can use it as a framework module. Also with flutter you can use dart:ffi (foreign function interface).

You can find the implementation of text classifier https://github.com/duladissa/go-tflite/blob/cross_compilation_support/_example/cross_complie