Sculpture AR: Exploring Google’s ARCore
In August of this year (that’s 2017 if you’re reading this in the distant future), Google announced ARCore, their mobile augmented reality platform that promises a high-tech future for the world’s most popular mobile OS. At the time of writing, it is available on a grand total of 2 phones (the Google Pixel, and Samsung Galaxy 8), but with Google committing to hitting 100 million devices by the end of their preview program, and with manufacturers looking for the next shiny new feature to sell you next year’s phone, expect ARCore to land in a big way.
Both Apple and Google are investing heavily in mobile AR this year, promising new ways to navigate the streets, preview virtual furniture in your home, mark your friends with arrows at a crowded concert, or place an animated lion on your co-worker’s desk. AR is demonstrating out of the gate the practical consumer appeal that VR has be struggling to nail. We had to get our hands on this.
Exploring ARCore
We set out with the simple goal of exploring ARCore. We wanted to see what it could do, and gain a familiarity with it’s inner workings.
After a day or so hacking away at ARCore, we put together a very basic functional PoC running on a Google Pixel. The app simply let you place a 3D model of a chair on a surface plane in your room.
So what is it actually doing? ARCore uses the phones’s camera to detect points in meatspace and returns this information to the as a point cloud. It then analyzes the point cloud, looking for points that represent a flat surface. When it finds a large enough horizontal plane, it defines it as a surface and lets the user tap within that plane to place an anchor point.
We can then attach any 3D model to that anchor point and keep it in place as the phone moves around in meatspace. The phone’s camera will also detect the average light intensity, which we can use to light our 3D model.
The Concept
Pretty cool right? Well kinda. It was working, but placing a chair is far from exciting. We decided it needed a little more flash and polish. So a team of 3 of us booked a day at Devhub in Toronto’s downtown tech core, and had a mini-hackathon to see what the Lab could come up with.
We wanted to build something fun, demo-able, educational and it had to lend itself well to AR. After brainstorming a few ideas (building a mini village, decorating your room with Game of Thrones style medieval decor), we landed on the idea of taking historic sculptures out of the museum and into your room. We could let users get up close and explore art in a way they would otherwise have to travel the world to see. This idea hit our educational requirement, and was simple enough to hack together in a day. We were all pretty jazzed about it.
Building Sculpture AR
With Amanda researching famous European statues, myself (Dave) designing the UI assets, and Jordan coding away, we got to it. The concept was clear, but naturally we ran into a few hurdles along the way.
We didn’t allocate ourselves time to get into 3D modeling original assets. That would take days. There were open source 3D scans of sculptures available, but the file sizes were far too massive for a mobile app, and were not optimized well for the level of performance we were after. Our solution was to purchase game assets from the Unity Asset Store. Game assets tend to be built for performance, and with a few little tricks we could convert them into usable OBJ 3D models and MTL texture files.
ARCore itself was straightforward and easy to use, some of the actual rendering work was effort intensive. To render a wavefront obj with materials required writing the OpenGL vertex and fragment shaders by hand. ARCore has an SDK for Unity, which would have abstracted away a lot of the graphics work, but Jordan being the brilliantly curious developer he is, wanted to work in the code to understand ARCore from the ground up. And who wants to do it the easy way? (Matt did)
ARCore being in preview release, documentation was lacking and tutorials are non existent.
In the end it came together pretty nicely! While a little rough around the edges, we ended up with a fully functional sculpture-viewing AR application that lets you:
- Place 10 different historic european sculptures in AR space
- Move, rotate and scale the sculpture to your liking
- Pull up an information plaque on each sculpture to learn about it’s history
The end result was pretty stunning for about 2 days of effort, and equipped our team with a firm understanding of Google’s AR Core platform. Nice.
Try it Yourself
ARCore apps are not yet available through the Play Store, but if you want to install and play with our Sculpture AR app, check out the steps below:
Required a Google Pixel/XL, Samsung Galaxy S8, or other ARCore compatible device
- Install Android Studio version 2.3 or higher with Android SDK Platform version 7.0 (API level 24) or higher.
- Enable developer options on the device.
- Enable USB debugging on the device
- Install the ARCore service on the device:
- Download the ARCore Service
- Connect your Android device to the development machine with a USB cable
- Install the service by running the following adb command:
adb install -r -d arcore-preview.apk
5. Install the Sculpture AR app on the device:
- Download the app here
- Install the app by running the following adb command:
adb install -r -d pt-statue-viewer.apk
What is Pixel Tours?
Pixel Tours is a product design and strategy consultancy based in Toronto. We bring sensible UX and technology intelligence to complex digital products from web to mobile to virtual/mixed reality.
Interested in working with us? Say hello!
