How we put TiltBrush art into AR
While working on our favourite project with ARTLIFE we experimented with XR-tool TiltBrush. And now I want to tell you in detail how to bring VR and AR together, and cover all the nuances of exporting art from Oculus Quest 2 into Unity URP!
What we had:
- Oculus Quest 2
- Unity URP project
- TiltBrush art
The first difficulty was in the rendering pipeline. For the standard pipeline, there’s TiltBrush Unity SDK. It allows you to open the fbx model exported from the application in Unity without losing materials and shaders.
This was not suitable for our URP project.
The second difficulty is the Oculus Quest headset. You cannot export objects from it in .fbx, only .gltf. Despite the fact that the latest SDK update allows you to open in Unity .gltf, for some reason it didn’t work for us.
What we did
We opened TiltBrush via Steam with Oculus AirLink and exported .fbx to the computer. You will need Oculus Link software for that.
The algorithm is pretty simple:
- Turn on AirLink in the software and headset settings
- Launch Steam
- Open TiltBrush
- Export as usual. The folder with models will appear in Documents.
At the same time, I took folders with textures for brushes, so that later they could be used to import not only to Unity, but also to Blender, for example. You can find these folders here.
Moving on to Unity
We import the model into the project and add it to the scene.
Yes, everything is pink, the materials crashed — it’s nothing scary. For now, our task is to transfer the shader from standard to URP, so we are creating our own shader. It’s more convenient for me to work with a shader graph, but you can rewrite it with code.
Information about the colour with which we drew is stored in VertexColor. From the folder with the brushes above, we look for a suitable one, add the bump and main texture to the project, and link them into the material.
Few more things
Previously, we had already worked with VR drawings and knew which brushes had more problems, so we chose the simplest ones so as not to waste time remaking the shader.
For animation and further convenient work with the drawing, before exporting, we advise you to think about how to group objects, since each brush stroke is a separate object.
In the drawing, the artist used a mist brush. Of course, it was easier for us to recreate the fog with ordinary particles in Unity, so we asked to group it separately and then just replaced it using the original texture.
As for optimization, the drawing was created for AR, so all unnecessary details, layers and backdrops were removed immediately. The hypothesis “what launched and worked well in a standalone headset, should have no problems with integration into AR” was confirmed.
We also wanted to add the artist to the final image, so we recorded her on a chroma key and stylized it in Phygital.Plus. This is our internal product, where you can easily work with neural networks, create and edit content. If you are interested, subscribe to the WaitList. The product will be legendary 🔥.
We added the recorded video into the editor, removed the background and ran it through StyleFAN. As a result we got a video and a mask, and added everything to Unity.
The reference is on the left, and on the right we have the styled artist
To the production
Now we can add animation, particles, fog, interactive elements — whatever you want :)
Afterwards, you can build the app and enjoy the result in augmented reality.
This and other 39 paintings you can view in the ARTLIFE app:
Ios / Android
Markers can be found here. Write to us if you want more reviews on some particular paintings!
Summing up
- For a standard pipeline, when importing from TiltBrush, use the Toolkit SDK. If you just need a drawing from the headset, this is more than enough;
- If you are working on UPR or LRP, you will have to rewrite the shaders. You can use Shader Graph;
- Export .fbx is possible using Oculus Link/Airlink;
- Maybe you will be lucky and will be able to import .gltf in Unity via the updated Toolkit SDK;
- During the drawing process think where the drawing will be used: group objects, separate what you plan to animate, remove unnecessary details if optimization is required;
- If you want to be the first to get access to our tool Phygital.Plus to work with neural networks without code, subscribe to the Waitlist.
Special thanks to the artist @margaritainwonderland for such a cool drawing in VR.
The physical painting was made by @aboohamid___asadi, for whom we have been drawing in VR for two years already.
Thanks to Daria Wind for translation.
