How to use the Style Transfer API in React Native with Fritz

Fritz is a platform that’s intended to make it easy for developers to power their mobile apps with machine learning features. Currently, it has an SDK for both Android and iOS. The SDK contains ready-to-use APIs for the following features:

1. Object Detection
2. Image Labeling
3. Style Transfer
4. Image Segmentation
5. Pose Estimation

Today, we’ll explore how to use the Style Transfer API in React Native.

I was only able to develop and test in Android (no Macs here!) and got a working application.

The Style Transfer API styles images or video according to real art masterpieces. There are 11 pre-trained artwork styles, including Van Gogh’s Starry Night and Munch’s Scream, among others.

The app we’ll be developing allows the user to take a picture and convert it into a styled image. It will also allow the user to pick the artwork style they wish to use on the image.

The app will contain a Home page, where the user can pick the art style. It will also include a separate Camera View, where the user captures the image.

Note: The following tutorial is for the Android platform only.

Prerequisites

1. React Native CLI: run npm i -g react-native-cli to globally install the CLI

Since there is no default React Native module for Fritz, we’ll need to write our own. Writing a native module means writing real native code to use on one or both platforms.

Step 1 — Creating the RN app and install modules

To create the app, run the following command in the terminal:

react-native init <appname>

Move into the root of the folder to begin configuration.

For navigation, we’ll be using React Navigation and React Native Camera for the Camera View.

To install both dependencies, run the following command in the terminal:

npm i --save react-navigation react-native-camera

Follow the instructions here to configure React Navigation for the app. We’ll need to install react-native-gesture-handler as well, as it’s a dependency of React Navigation.

Follow the instructions here to configure the React Native Camera for the app. We can stop at step 6, as for this example we will not be using text, face, or barcode recognition.

Step 2 — Including Fritz SDK in the app

First, we need to create a Fritz account and a new project.

From the Project overview, click on Add to Android to include the SDK for the Android platform. We’ll need to include an App Name and the Application ID. The Application ID can be found in android/app/build.gradle, inside the tag defaultConfig.

Upon registering the app, we need to add the following lines in android/build.gradle:

allprojects {
    .....
    repositories {
        .....
        maven { url "https://raw.github.com/fritzlabs/fritz-repository/master" } //add this line
    }
}

Afterward, include the dependency in the android/app/build.gradle:

dependencies {
    implementation 'ai.fritz:core:3.0.2'
}

We’ll need to update the AndroidManifest.xml file to give the app permission to use the Internet and register the Fritz service:

<manifest xmlns:android="http://schemas.android.com/apk/res/android">
    .....
    <uses-permission android:name="android.permission.INTERNET" />

    <application>
        .....
        <service
            android:name="ai.fritz.core.FritzCustomModelService"
            android:exported="true"
            android:permission="android.permission.BIND_JOB_SERVICE" />
    </application>
</manifest>

We then need to include the following method within the MainActivity.java:

import ai.fritz.core.Fritz;
import android.os.Bundle; //import these two as well
public class MainActivity extends ReactActivity {
    .....
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        // Initialize Fritz
        Fritz.configure(this, "<api-key>");
    }
}

Step 3 — Create the Native Module

Since the SDK only supports iOS and Android, we’ll need to make the native module. To get a better understanding of this, refer to the docs here:

Native Modules · React Native

To make an Android Native module, we’ll need to make two new files. They will be within the root package of the Android source folder.

1. FritzStyleModule: This contains the code that will return the styled image
2. FritzStylePackage: This registers the module so that it can be used by the JavaScript side of the app.

FritzStyleModule

The React method being used has a success and error callback. The chosen artwork style and a base64 of the original image are sent to the method. The error callback is invoked when an Exception is thrown and returns the error. The success callback returns a base64 encoded string of the converted image. On a high-level, the above code does the following:

1. Initializes the style predictor with the user’s choice of artwork.
2. Converts the original base64 image into a Bitmap.
3. Creates a FritzVisionImage, which is the input of the style predictor.
4. Converts the FritzVisionImage into a styled FritzVisionStyleResult, which is the converted image.
5. Gets a Bitmap out of the FritzVisionStyleResult.
6. Converts the Bitmap into a base64 to be sent back to the JavaScript side of the app.

FritzStylePackage

This class is used to register the package so it can be called in the JavaScript side of the app.

This class is also initialized in the getPackages() of MainApplication.java:

@Override
protected List<ReactPackage> getPackages() {
  return Arrays.<ReactPackage>asList(
    new MainReactPackage(),
    ......, 
    new FritzStylePackage() //Add this line and import it on top
  );
}

Now on to the JavaScript side of the application.

Step 4 — Creating the UI

To do this, we’ll be creating/updating the following pages:

1. Home.js — Display the picker of artwork styles and the final result.
2. CameraContainer.js — Display the camera view to capture an image.
3. FritzModule.js — Export the above-created Native module to the JavaScript side.
4. App.js — Root of the app which includes the navigation stack.

Home.js

This page contains:

1. Text to display the app description.
2. Picker to allow the user to select the artwork style of the converted image.
3. Button to redirect the user to the Camera page. It will pass the selected artwork style to the CameraContainer.
4. If the navigation prop contains the original and converted image, it will be displayed.

The page currently looks like this;

Home page before taking a picture

CameraContainer.js

The CameraContainer page displays a full page CameraView. It includes a button to take the picture at the bottom of the page. Upon clicking it, a spinner will be displayed to convey to the user that an action is taking place.

The image is first captured using the react-native-camera method takePictureAsync(). The original image is then saved into the state of the page. The setState method is asynchronous and thus has a success callback that runs after the state is set.

The getNewImage method from the FritzModule is run within this success callback. The original image and the filter (artwork style) picked from the Home Page is passed to the method. On the error callback, an alert is displayed to the user to convey that an error has occurred. On the success callback, the new styled image is saved into the state. On this second setState methods’ success callback, the user is redirected to the Home page with both the original and styled images.

CameraContainer on emulator

FritzModule.js

import { NativeModules } from 'react-native';
export default NativeModules.FritzStyle;

This page exposes the Native module, FritzStyle. This allows the JavaScript side to make calls to the method getNewImage.

App.js

import React, { Component } from 'react';
import Home from './src/Home';
import CameraContainer from './src/CameraContainer';
import { createStackNavigator, createAppContainer } from 'react-navigation';
const AppNavigator = createStackNavigator({
  Home: { screen: Home },
  Camera: { screen: CameraContainer }
});
const AppContainer = createAppContainer(AppNavigator);
export default class App extends Component {
  render() {
    return (<AppContainer />);
  }
}

First, we create the Stack navigator with the Home Page and Camera View. The key ‘Home’ is used when navigating to the Home Page, and the key ‘Camera’ when navigating to the CameraContainer.

The AppContainer becomes the root component of the App. It’s also the component that manages the app’s state.

Now to see the entire app in function;

To recap, we have;

1. Created a React Native app,
2. Included the Fritz SDK in it,
3. Created a Native Module that makes use of the Style Transfer API, and
4. Designed a UI to display the styled image.

Find the code repo, here.

For native iOS or Android implementations of Fritz’s Style Transfer API, check out the following tutorials:

Real-Time Style Transfer for Android — Transform your photos and videos into masterpieces

Real-Time Style Transfer for iOS— Transform your photos and videos into masterpieces