Fusing 8th Wall and react-three-fiber

A powerful combination to build AR Web Apps.

A powerful combination to build Web Augmented Reality Experience is to use ThreeJS framework on the 8th Wall platform.

8th Wall is a very powerful platform, which helps you to write AR applications quickly. It’s AR engine provides you features like image detection, surface detection, face detection, etc in a web browser.

ThreeJS on the other hand is a popular Javascript 3D library. It enables you to create 3D objects in a web app.

One caveat is that ThreeJS is an imperative javascript API. And I am more of a ReactJS fanboy who loves to develop in declarative syntax.

This is where React-Three-Fiber (R3F) comes to the rescue. It is a React reconciler for ThreeJS. It enables you to build ThreeJS objects as declarative reusable components. All the cool and powerful React ecosystem unlocks for you, and it also supports interactivity out of the box.

To continue with this article, you would require a basic understanding of the 8th Wall platform, ThreeJS and R3F

When I started exploring 8th Wall with ThreeJS, all the examples or discussions were in an imperative approach. They worked very well and were amazing. But I was curious to rewrite them in a declarative manner.

I tried to look up if there were any existing implementations available which used 8th Wall and R3F together but could not find one.

Well, then I decided to do it myself 🚀

Following is a rudimentary implementation, in which I am trying to render relevant details around a card. At first, it was implemented in an imperative manner, and then I was able to rewrite it in a declarative approach without losing any capability. In fact, it enabled me to write it with much more control and a cleaner way.

Here 8th Wall is helping me to detect the card and its position.
Based on this information, I am rendering ThreeJS objects around it, which are implemented in R3F.

• This is a Web App.
• Every element is a React component.

There were two major changes that my project required.

1. Use of the Canvas component.

All Web AR applications work on the <canvas /> tag.

8th Wall also needs one.

All 8th Wall examples have a <canvas /> tag, and when the application initializes, it grabs a reference of this canvas to start painting the camera feed.

But when you want to use R3F, you do not need to write <canvas /> tag yourself.

It comes with a Canvas component, which is our portal into the ThreeJS.

This component is responsible for rendering all our declarative ThreeJS components.

Your component will look something like this:

import { Canvas } from 'react-three-fiber'
return (
    <Canvas>
        
    </Canvas>
)

This Canvas component internally renders a <canvas /> tag.

Hence as part of our integration, we need to attach the reference of this <canvas /> tag to the 8th Wall implementation.

The easiest way to grab reference of <canvas /> element as follows:

// grabbing reference of canvas
let canvasEl = document.getElementsByTagName(‘canvas’)[0]
//attaching canvas reference to 8th Wall
XR8.run({ canvas: canvasEl})

In order for this step to work properly, you may have to avoid using XRExtras.FullWindowCanvas.pipelineModule() in your 8thWall setup and style the canvasEl a bit to appear full screen.

2. Sharing ThreeJS resources between 8th Wall and R3F.

This is the major part of the integration.

A ThreeJS application works on the crucial objects like Scene, Camera and Renderer.

8th Wall initializes its own set of resources to work with ThreeJS, and R3F also creates its own resources. Hence in this step, we want to be able to make them work on the same set of resources. The way we make them work on the same <canvas/> tag in step 1.

I have observed that 8th Wall is unable to work without its own set of resources, hence we will ask R3F to use 8th Wall’s resources.

8th Wall’s set of ThreeJS resources can be generated using:

const { camera, scene } = XR8.Threejs.xrScene()

Now, we will pass these resources to our App in Canvas component:

import { Canvas } from 'react-three-fiber'
return (
    <Canvas>
        <App
            scene={scene}
            camera={camera}
        />
    </Canvas>
)

App.js is where we do our integration:

import React, { useRef, useEffect } from ‘react’
import { useThree } from ‘react-three-fiber’
const App = (props) => {
    const { scene, camera } = props
    const { setDefaultCamera } = useThree()

    // ref to root of our ThreeJS app
    const appRef = useRef()


    // add our app to 8thWall's ThreeJS scene
    useEffect(() => {
        if(scene) {
            scene.add(appRef.current)
        }
    }, [scene])

    // set 8thWall's ThreeJS camera as default camera of 
    // react-three-fiber
    useEffect(() => {
        if(camera) {
            setDefaultCamera(camera)
        }
    }, [camera, setDefaultCamera])
    return (
        <group ref={appRef} visible={true}>
            {... your R3F components}
        </group>
    )
}

Now our R3F app will start appearing in the 8thWall’s ThreeJS world.

Benefits?

Once this integration worked, I was able to rewrite my 8thWall project very quickly in a declarative manner and with much fewer lines of code.

Now I can focus more on writing quick and cool ThreeJS objects in the 8thWall world, rather than feeling frustrated with the complexity of imperative code.

Hope this article helped you. Happy coding!