Building with Flutter + Unity (AR Experience Toolkit)

Published in

@potato

6 min readApr 6, 2021

Wallace from The Big Fix Up is in his workshop surrounded by a pile of urgent job requests.

Wallace & Gromit: The Big Fix Up launched in January 2021, thanks to the collaboration between multi-award-winning independent animation studio Aardman and storytelling venture Fictioneers (a consortium of Potato, Tiny Rebel Games, and Sugar Creative).

Available on iOS and Android devices in the UK, US, and Canada, the free app creates a narrative-driven experience, taking the user through AR gameplay, CG animations, in-character phone calls, extended reality (XR) portals, and comic strips. In the new storyline, inventor Wallace and his dog Gromit’s new business venture takes on a contract to “Fix-Up” the city of Bristol, UK.

It’s been three months since The Big Fix Up app was made public and it has gained a lot of attention. This AR masterpiece, which was built on Flutter with Unity, was a first of its kind and raised a lot of interesting questions about our approach to building this real-time storytelling experience:

Why Flutter?
Embedding Unity3D app in Flutter?
Communication between Unity and Flutter
Performance implications for embedding Unity with Flutter,
App bundle size, e.t.c.

Here we’ll break down how we answered these questions and eventually solved the problem of embedding Unity in Flutter. The result was a Flutter Unity View plugin which is published opensource for the community here: https://github.com/juicycleff/flutter-unity-view-widget.

Why Flutter?

If you’re in the mobile application development or engineering space, you most likely won’t need an introduction to Flutter, but if not,

Flutter is Google’s open-source toolkit for building beautiful, natively compiled applications for mobile, web, desktop, and embedded devices from a single codebase.

Flutter enables a mobile developer to build amazing, high-performing, and beautiful apps with minimal effort with widgets for nearly all your needs. Learn more about Flutter.

After testing several popular mobile cross-platform frameworks out there, we chose Flutter for 3 reasons;

Performance
Stability
Rich UI

Performance and Stability were a big concern for us because our app was a combination of a Flutter app and Unity app. When tested on other cross-platform frameworks, we had issues such as Unity not properly rendering, or sometimes rendering blanks on a lot of devices.

Also, bidirectional communication between framework (other cross-platform) and Unity was comparatively slow when compared to Flutter when sending, such as Touch and Gesture events to Unity, but with Flutter, we benchmarked more efficient events notification and message communication to and from Unity which can only be rivaled by pure native mobile development. The Flutter rendering is so powerful that we never had any issues with rendering our Unity App on top of it and because Flutter widgets are the most beautiful, eye-catching stock UI components around, it takes little effort to make beautiful apps in Flutter.

Embedding Unity3D app in Flutter

We needed to export the Unity project as a native iOS and Android project so we can convert it into a native package that can be added to the native codebase of our Flutter app. The problem was that Unity at the time only had support for exporting Unity projects as an app instead of a package (This was pre UaaL). This required modification to the actual Unity project which heavily involves modifying our exported Unity app source and converting it into a library. But today with support for Unity as a library (UaaL), which now does support a lot better for reusing Unity inside native apps, there was little modification to be done, but we still needed a few like;

Notifying Flutter when Unity player is ready to be used in Flutter. For instance, we added a new notification event to the Unity iOS post-build processors by extending the codebase just like so.

/// <summary>
/// Edit 'UnityAppController.mm': triggers 'UnityReady' notification after Unity is actually started.
/// </summary>
private static void EditUnityAppControllerMM(string path)
{
    var inScope = false;
    var markerDetected = false;

    EditCodeFile(path, line =>
    {

        inScope |= line.Contains("- (void)startUnity:");
        markerDetected |= inScope && line.Contains(TouchedMarker);

        if (inScope && line.Trim() == "}")
        {
            inScope = false;

            if (markerDetected)
            {
                return new string[] { line };
            }
            else
            {
                return new string[]
                {
                    "    // Modified by " + TouchedMarker,
                    @"    [[NSNotificationCenter defaultCenter] postNotificationName: @""UnityReady"" object:self];",
                    "}",
                };
            }
        }

        return new string[] { line };
    });

}

We also had to modify the Unity project Android export to work with Flutter seamlessly by modifying the Gradle build script automatically by replacing all Gradle android application-related codes to an android library like so;

public static void DoBuildAndroid()
{
    ...

    // Modify build.gradle
    var build_file = Path.Combine(androidExportPath, "build.gradle");
    var build_text = File.ReadAllText(build_file);
    build_text = build_text.Replace("com.android.application", "com.android.library");
    build_text = build_text.Replace("bundle {", "splits {");
    build_text = build_text.Replace("enableSplit = false", "enable false");
    build_text = build_text.Replace("enableSplit = true", "enable true");
    build_text = build_text.Replace("implementation fileTree(dir: 'libs', include: ['*.jar'])", "implementation(name: 'unity-classes', ext:'jar')");
    build_text = Regex.Replace(build_text, @"\n.*applicationId '.+'.*\n", "\n");
    File.WriteAllText(build_file, build_text);

    // Modify AndroidManifest.xml
    var manifest_file = Path.Combine(androidExportPath, "src/main/AndroidManifest.xml");
    var manifest_text = File.ReadAllText(manifest_file);
    manifest_text = Regex.Replace(manifest_text, @"<application .*>", "<application>");
    Regex regex = new Regex(@"<activity.*>(\s|\S)+?</activity>", RegexOptions.Multiline);
    manifest_text = regex.Replace(manifest_text, "");
    File.WriteAllText(manifest_file, manifest_text);

    ...
}

Knowing we can automate the export from Unity and modify Unity export on the fly, Unity was ready to be embedded into Flutter. This then was a good foundation to enable Flutter to render unity. With the highly performant native extension support in Flutter, we were able to use Flutter Native API to achieve rendering Unity on Flutter, some of which are;

These APIs allowed us to attach the Unity native view in Flutter for both Android and iOS platforms. With rendering out of the way, we needed to have full bidirectional communication between Flutter and Unity.

Communication between Unity and Flutter

Flutter does support a high-performance messaging transport from Flutter to the native side with the MethodChannel API support, one good reason Flutter outperforms other cross-platform frameworks. The MethodChannel API gives us means for communication with the native code however, Unity as a Library only allows one-way communication, which was from Flutter to Unity only and so we had to figure out how Unity can message Flutter. Luckily Unity has support for calling native code classes and static methods from within Unity and this led to writing the Unity Message Manager whose job is to call Kotlin/Java/Obj C/Swift static class methods, which in turn notifies or calls a Flutter method. The Unity message manager looks like this;

public class NativeAPI
{
    #if UNITY_IOS && !UNITY_EDITOR

    [DllImport("__Internal")]
    public static extern void OnUnityMessage(string message);
    #endif


    public static void SendMessageToFlutter(string message)
    {
        #if UNITY_ANDROID
        try
        {
            AndroidJavaClass jc = new AndroidJavaClass("com.xraph.plugin.flutter_unity_widget.UnityPlayerUtils");
            jc.CallStatic("onUnityMessage", message);
        }
        catch (Exception e)
        {
            Debug.Log(e.Message);
        }
        #elif UNITY_IOS && !UNITY_EDITOR
        NativeAPI.OnUnityMessage(message);
        #endif
    }
}

Here we invoke the native Android class UnityPlayerUtils method UnityPlayerUtils and also an ObjectiveC static method NativeAPI.OnUnityMessage(message) while using CSharp preprocessor directives. On the Flutter native end, the methods look like this for Android Kotlin;

class UnityPlayerUtils {
  ...  /**
  * Invoke by unity C#
  */
  @JvmStatic
  fun onUnityMessage(message: String) {
    for (listener in mUnityEventListeners) {
        try {
            listener.onMessage(message)
        } catch (e: Exception) {
            e.message?.let { Log.e(LOG_TAG, it) }
        }
    }
  }  ...
}

and on iOS as simple as this;

void OnUnityMessage(const char* message)
{
    [UnityPlayerUtils unityMessageHandler:message];
}

and it calls the Flutter MessageChannel instance to notify flutter;

@objc public class UnityPlayerUtils: UIResponder, UIApplicationDelegate, UnityFrameworkListener {  @objc
  public static func unityMessageHandler(_ message:        UnsafePointer<Int8>?) {
    if let strMsg = message {        globalChannel?.invokeMethod("events#onUnityMessage", arguments: String(utf8String: strMsg))    } else {        globalChannel?.invokeMethod("events#onUnityMessage", arguments: "")    }
  }}

With these, we had bidirectional communication between Flutter and Unity through the MethodChannel API with very impressive performance, which was vital to our choosing Flutter in the first place.

Performance implications for embedding Unity with Flutter

Rendering Unity on Flutter was very much like rendering a native view in a native view, it was just that efficient! There were no performance issues with Unity’s performance inside of Flutter, you just need to manage your Unity CPU and memory usage by making sure Unity is unloaded or in a scene with fewer resources when not interacting with your Unity scenes.

Using Flutter & Unity for your AR Project

Finally, the combination of Unity and Flutter is a great fit for your next AR project, because you can combine the beautiful UI of Flutter with the easy and powerful game engine to have a real nongame mobile app with the possibilities from a game engine. We have the package for Flutter Unity integration public here, so you don’t have to start from scratch. Here it is and good luck on your next project.