Immersive Media Design

AR Book Map Final

Full Documentation

Background

Project Idea & Problem I’m Solving

For this project I wanted to continue my mapping world literature project that I started in the fall. I created my original Book Map site as a way for readers to expand their horizons and build empathy through literature from all over the world. My thought was that building the book map in an augmented reality format would make it more fun and interactive, and would allow the map feature to be a realistic globe that could be rotated and scaled.

Inspiration

Civilizations AR by the BBC

I was inspired by the Civilizations AR app created for the BBC by Nexus Studios. When you open the app, you move your phone to scan a plane, and when the plane is detected a globe pops into the scene. The globe includes interactive data points at geographic locations, that, when clicked, open an AR projection of a scanned artifact from that location. I envisioned my app working basically the same way. An AR globe would include interactive data points that when clicked would display the book information in some way. I had the data set ready to go in Mapbox, and only needed to build out the functionality in Unity.

Process

Mapbox Unity SDK

I spent the first part of this project exploring how to build a globe in Unity, but the project really took off when I discovered and imported the Mapbox Unity SDK.

Install the Maps SDK for Unity

The SDK includes several example scenes, one of which was a globe which seemed to be perfect for my purpose. The example globe opened with a sample geographic style and three custom markers.

Importing custom styles and data

My next task was to replace the default Mapbox styles with my own style sheet and tileset from my Mapbox account. The below instructions in the SDK documentation helped me figure out how to do this.

Custom maps

First, I had to go into my Mapbox account and copy the style URL and paste it into the the Image box in the Abstract Map section of the Map game object.

Next, I had to copy the tileset URL from Mapbox and paste it into the Tileset ID box in the Map Layers section of the Map game object.

Importing my style sheet and tileset made the globe in my Unity scene show up like this (below). However, the data points from my tileset showed up on the globe only as an image. They were not interactive in any way that I could discover.

Making the app AR

My next hurdle was turning the app into an AR app. I followed these steps:

1. Go to Window →Package Manager and add the AR Foundation and ARKit XR Plugin packages.
2. Add AR Session Origin and AR Session to scene.
3. Set AR Camera to MainCamera tag. Delete original camera.
4. Place the Map object into an empty GameObject and scale down the game object so that it fits into the tiny AR camera. I tried 0.0005 for X, Y, and Z first, but ended up liking 0.0003 better.
5. Change the Player Settings (set the bundle identifier, set the camera usage description, make sure target minimum iOS version says 13.0, set the architecture to ARM64)
6. Build & Run

Solving the Xcode error

However, when I tried to build my app in Xcode, I was continually getting an error that caused the build to fail. This error appeared to be just a glitch in the system, causing Xcode to be unable to locate folders that were right where they were supposed to be.

RJ helped me solve this problem by combing two different workarounds.

First, I had to direct Xcode to the folders manually by following these two steps (this tip came from this link):

1. Open the Xcode project and go to the Build Settings tab and search for Header Search Paths. Double click on the Header Search Paths row and press the + button to add a line and paste this into it $(SRCROOT)/Libraries/Mapbox/Core/Plugins/iOS/MapboxMobileEvents/include/
2. Manually link the libz library to the project. That can be done by going to the Build Phases tab and expanding Link Binary With Libraries. Click the + button and search for libz.tbd and press Add.

We thought the first step alone would solve it, but the builds still failed until I added in RJ’s original solution of for the error, which was to change the following lines of code in Xcode:

1. Line 1 in MapboxMetricsClient.m

#import “MapboxMobileEvents.h”

2. Lines 11–14 in MapboxMobileEvents.h

#import “MMEConstants.h”

#import “MMEEvent.h”

#import “MMETypes.h”

#import “MMEEventsManager.h”

When combined, the two steps above fixed the error and the builds succeeded.

Building interactivity

Other than the export error, I struggled the most with trying to figure out how to make the globe interactive.

At first I assumed it would be fairly straightforward to make my tileset data points interactive. I am sure that there is a way to do that somehow, but I could not for the life of me figure it out.

My next thought was to utilize the custom markers, like those that came with the default globe. I figured out how to manually place custom markers at longitude and latitude points on the globe, but I couldn’t figure out how to make them spawn across all the points in my data set. So I thought, maybe I could just place a few example markers and make those few interactive. I wanted the markers being clicked to cause a UI window to open which would display book information. I watched many unity UI interactivity tutorials, and when I tried out different interactive ideas they worked, but as soon as I tried to make those elements the markers on the globe, the interactivity broke. As a side note, I later realized that the markers on the globe did not scale along with the globe when viewing the globe in AR on my phone.

Finally, when it was getting down to the wire and my interactive markers weren’t panning out, I decided to do my last workaround and make clickable buttons in the corners of the screen that cause book information to pop up. The buttons impersonated what I wanted to happen when clicking on spots on the globe. It ended up looking like this:

Scaling and rotating the globe

The default Mapbox globe came with a built in rotation ability, but it was quite limited. You could only rotate the globe on one axis, and you couldn’t scale it at all. To add more interaction to the globe, I imported the Lean Touch plugin from the asset store and added several Lean Touch scripts to my globe (the three below). I learned about the Lean Touch plugin from this YouTube video.

After adding these interaction components, the globe worked like this:

Final Thoughts

Major takeaways

This whole project was a lesson in problem solving. It was often frustrating because I sometimes felt that the even if I put in tons of time, it was still not guaranteed that I would figure anything out. As a beginner in Unity and C# scripting (and coding in general) it felt like some of the problems I was facing were impossible to solve without help, and even if I found helpful information online, I didn’t always understand what it was directing me to do. However, in the end it was really satisfying to be able to face those problems, and realize that in the time frame I wasn’t going to get past them, and then find a simpler way to show what I wanted to show without needing to do it in the highest fidelity way. It reminded me of the prototyping process that we’ve been talking about all year in Jesse’s class. Sometimes there is an extremely simple way to prototype something that take a fraction of the time and effort, but will still give you a good idea of the what the final product might be.

Next steps

I would like to continue with this project in the original web version and the AR version, enough so that I can put it into my portfolio. I actually just received an email back from the Mapbox support people that I emailed a couple weeks ago. The Mapbox guy wrote this:

“You can set up the features on the AbstractMap using your custom data source to visualize your points. You can use a PrefabModifier to specify the 3D objects you want at the given coordinates.

Those prefabs can have behavior modifiers to interact and show data. I would have a look to see how it’s being done in our Data Explorer example. It should give you some ideas on how to do this and maybe even some scripts to repurpose.”

I haven’t had time yet to dig into his suggestions and try them out, but I would like to!

Previous Medium Documentation

AR Book Map

AR Book Map Part 2

AR Book Map Part 3

AR Book Map Part 4