Rapid Prototyping with Unity and React

Web applications are great way to quickly reach a lot of users without the hassle of native client installs and play store downloads. Web frameworks such as React allow page components to communicate and respond to user interaction much like traditional applications. Being web-focused, React also offers exceptionally strong support for adaptive and responsive layout, e.g. smoothly re-flowing the page content for a vertically-oriented mobile device or browser window.

However, applications with complex UI and internal state — like a 3D game or simulation — are largely out of scope for React. Instead, such tasks are the domain of game engines like Unity3D. Unity can export WebGL applications which run self-contained game worlds with 3D rendering, physics interactions, and so on. Unity WebGL can also, in principle, communicate with the browser via jslib plug in and react-unity-webgl.

In theory, then, we can get the best of both worlds for rich interactive web applications by implementing one or more complex worlds in Unity WebGL, then allowing them to communicate with each other via the browser.

Unity and React Example Project

Using this technique, we were able to implement a Machine Vision training simulator composed of three distinct panels — two have unique Unity3D WebGL instances — in a React-based page. Each page has two way communication between the instances of Unity and React’s front end.

Panel #1 displays an assembly line with 3D assets coming down a conveyer belt. Panel #2 handles dataset generation by letting users take snapshots of parts that had been incorrectly classified as they came down the belt. Panel #3 allows players to define ML classes and populate them with snapshots from #2 (a form of Transfer Learning), in order to train the Machine Vision model to better classify parts coming down the belt. The results of this model training are visible in Panel #1, as parts in the next batch are classified under the retrained model.

Again, “under the hood”, this is all accomplished by embedding two separate instances of Unity WebGL in a single React page, and passing key data (3D asset identifiers, bitmap snapshots, ML model output) back and forth between them in messages.

See a live example of this project here.
Also, check out the open source project here.
For more information on what inspired this template, check out our article here.

A Template for Messaging Integration

In practice, combining these two frameworks is a bit tricky. Unity provides great documentation on how to communicate with the browser, but no out of the box solution. That’s where cra-template-unity-rect (CTUR) template comes in.’

CTUR is a Create React App template, installed with a single CLI command, that sets up a messaging architecture for Unity WebGL games to send and receive messages to and from the browser. These messages are reusable and customizable.

Ultimately, we hope that CTUR will significantly speed up development of web applications that can take advantage of both Unity’s beautiful 3D environments and React’s flexibility and familiarity.

CTUR is completely open source and can be found here.

Installing

1. Install npx and node
2. Open a terminal and go to a folder where you want to download the template.
3. Enter the command: npx create-react-app my-unity-react-app --template @carnegie-mellon-robotics-academy/cra-template-unity-react
4. See Readme for how to build and develop.

How it works

We send messages back and forth as serialized objects. Our project follows a basic structure on both sides. Each message has a topic. This topic is always a string. Components can subscribe to these topic in order to receive messages. Messages also have a message object. This object contains a method string and a parameters object.

{
  topic: "test",
  message:
  {
    method: "startTest",
    parameters:
    {
      i_testTime: 3,
      resolver: "endTest"
    }
  }
}

Unity communicates to the browser using a JS lib plugin. Unity has great documentation on how to do it here. This plugin allows us to communicate to the browser window. You can add other functions as needed. ReactUnityWebGL is an open source package that we used included in the project. It does a nice job of wrapping unity content in a react element.

MessageUnityOutgoing: function (str) {
  // Send string to ReactUnityWebGL…    
  ReactUnityWebGL.MessageFromUnity(Pointer_stringify(str));
}

Using our unity_api methods, React components can subscribe to topics when it mounts to the DOM. Each method can be imported as needed. This examples shows how to call the method handleTestMessage when Unity sends a message with a topic of “unity_awake”.

const unityAwakeSubscriber = { topic: "unity_awake", callback: handleTestMessage }
useEffect(() => {
  addTopicListener(unityAwakeSubscriber);
}, [])

We also provide synchronous and asynchronous methods to Unity. You can uses asynchronous methods to wait for a response from Unity. In this example, a react component starts a test. By passing the resolver as the parameter. Unity will know that this promise is waiting on a message with a topic of “endTest”.

const handleTest = async () => {
const msg = { topic: "test", message: { method: "startTest",     parameters: { i_testTime: 3, resolver: "endTest" } } }
setTesting(true);
await sendUnityMessageAsync(msg, "endTest").then((message) => {
  console.log("Test Ended ", message);
  setTesting(false);
});
}

Unity has a similar way of sending and receiving these messages. All messages from react get passed from MessageDispatcher.cs (MD). MD provides methods similar to unity_api. This example shows how to subscribe to a topic and method from another mono-behavior.

messageDispatcher.addMessageListener(new Subscriber("test", startTest));

A subscriber consists of a topic and a callback. The callback startTest must match the method that it expects. Callbacks must receive an object as its parameters. Once you have the parameters, you will need to deserialize them with into the structure it expects.

private void startTest(object parameters) {
 TestParam testParameters = JsonConvert.DeserializeObject<TestParam>(parameters.ToString());
 
 // Start the test with the commands. Pass the resolver to end the 
}

Sending messages is also easy. MD has a method to send data through the js lib plugin. It is passed an object which can be null, and a topic. The object will get serialized and sent as a message with the topic.

messageDispatcher.SendData(null, "endTest");
// message output
// {topic: endTest, message: null}

How to Contribute

The main goal of the template is to help Unity and Web developers integrate unity with modern front end frameworks like React. React offers rich GUI affordances and is a convenient middleware layer to the server.

This template is in its early stages. There are a lot of additional features that we would like to add to the project. These features all come in handy when working with React and Unity.

These include:

• Support for managing redux.
• Automatic focus for unity windows and keyboard capture
• Unity build script to build to “dist” automatically.
• Easy setup for multiple instances of unity.
• Unity editor interface where you can build your own messages.
• Support for other frontend frameworks.

If you think there are any other great frameworks or want to improve the template please become a contributor. https://github.com/cmroboticsacademy/cra-template-unity-react/

Carnegie Mellon Robotics Academy

Carnegie Mellon’s Robotics Academy studies how teachers use robots in classrooms to teach Computer Science, Science, Technology Engineering, and Mathematics (CS-STEM). Our mission is to use the motivational effects of robotics to excite students about science and technology. The Robotics Academy fulfills its mission by developing research-based solutions for teachers that are classroom-tested and foreground CS-STEM concepts.

This material is based upon work supported by the National Science Foundation under Grant Number 1937063.