ExplorAR: A Collaborative Augmented Reality Game Platform for Museums — Development Phase 1
You’re not born too late to explore the Earth, you’re just in time to re-explore the earth in ExplorAR!
Project Overview
ExplorAR is an augmented reality game platform for museums that provides a new experience to learn the world of the past by exploring augmented reality with your phone both inside and outside museums. In this interactive experience, players collect 3D-scanned real-life artifacts such as fossils, sculptures, & paintings. Players will learn how to preserve historical objects by excavating fragments of artifacts from rocks, collaborate by combining fragments of incomplete artifacts, & express their creativity by designing their own virtual gallery.
Background Research
Museums are places where both adults and children can spend their leisure time. They provide reliable, authentic, and understandable information where the public can find meaning and connection. In this era of digital information where there are more alternatives to leisure activities, museums have a decline in interest as a result from lack of social participation from digital natives. As reported by the New Media Consortium, major museums & institutions start digitizing their collection by 3D scanning and exporting them into 3D models which helps museums expand their audience. There is a need for museums to engage their visitors, increase participatory experience through new media, and create a crowdsourcing environment. Vincent Rossi of Smithsonian Digitization Program mentioned the need of utilizing 3D scans for educational use in a novel & meaningful way.
“We are exploring how we can make our digitized collections meaningful, inspirational, and significant: in other words, digitizing for impact.” — Smithsonian Digitization Program
ExplorAR, a collaborative artifact-based mixed reality exploration game can offer a new learning experience.
The goal of this project is to create an augmented reality game platform for museums that demonstrates a new gamified learning experience based on exploration which supports public learning and empowers the engagement between users and museums. This project will prove a museum-based game platform for both inside and outside museum which includes augmented reality treasure hunting, multiplayer collaboration, virtual gallery creation, and crowdsourcing platform, which is a better alternative to engage learning and motivate younger generations compared to the previous existing media (AR Viewer, 3D viewer, 360 virtual tours, etc.). This project will be used by people who provide educational materials such as Museum Curators and people who use the game platform such as Students (Elementary-College), Tourists, & Parents with their kids, which were already tested in a prior user research.
Diana Marques of The Smithsonian Institution has addressed about the current concerns and challenges of AR experiences in museums which discusses gimmickry, detraction, replacement, onboarding, & duration of content in augmented reality experiences in her paper: Concerns & Challenges of AR Experiences in Museums (Marques, D. & Costello, R., 2018). Her paper is based on the Skin and Bones augmented reality experience where users can learn how species look when it was still alive and how their skeletons work by scanning the bones in The Smithsonian’s National Museum of Natural History’s Bone Hall.
Based on the findings, the application has eliminated a handful of addressed concerns, although known issues still exist. To read the information of each bones, visitors have to scan the bones one by one and it could only be accessed with an iOS device. Indoor exhibition constraints also exist, meaning that it can only be used inside the exhibition which might also interfere with the existing museum experience. The short duration of the content only lasts during the visit inside the exhibition, which makes the application having a short application lifetime in the visitors’ phones. To create a meaningful augmented reality experience, technical and social aspects are needed to be considered.
ExplorAR takes a lot of useful lessons from the paper. ExplorAR integrates both inside & outside museum experience, meaning that it extends the application lifetime. There will be monthly updates of new content and special events for different museums to keep users engaged with the application. With Niantic’s Real World Platform, the application can be used for both Android and iOS devices. And lastly the emulation of meaningful and interactive experiences would make the application less gimmicky. ExplorAR emulates the experience of being an archaeologist or a paleontologist when exploring the world, excavating artifacts, and collaborating with other players and the experience of being a museum curator when decorating their virtual gallery.
Background Narrative
In a parallel universe, aliens have attacked the earth by stealing their historical archive and encapsulated them in space rocks. Join forces with other explorers to restore humanity’s knowledge!
Initial Design
Game Loop
Features
Map
This is the screenshot of the Main Menu which includes the map. The explorer avatar is located at the center of the map while interactive objects are scattered around the map. The main menu consists of Explore, E-Pedia, Gallery, Settings, & User Profile which will be explained later in this article.
Collectibles
Artifacts
Artifacts are the main collectable items in the game. These artifacts are based on real life 3D scans from museums which include: The Smithsonian Institution, Matson Museum of Anthropology, & Penn State College of Earth & Mineral Science.
Some artifacts that are encapsulated in rocks are fragmented and some of them are already completed. Each fragment contains a snippet of information related to the artifact. By combining fragments into a complete artifact, explorers can learn the whole information regarding to the completed artifact in their encyclopedia.
Explorers can retrieve fragments of artifacts in pile of rocks that appear in the map. The fragments are generated based on the artifacts of the closest museum to the explorers’ current location. These fragments spawn evenly distributed in a controlled space. Later in the future, different museums will feature their own featured artifacts.
Sensor Drives
Sensor Drives are non-reusable collectibles that can be used with the compass which can help explorers to find a specific category of an artifact a lot easier, such as Mythology, Fossils, or Objects. These drives are based on different categories of artifacts. In a short period of time, selected category of artifacts will spawn around the user.
Batteries
Space rocks are difficult to excavate, which is why explorers need battery-powered Laser Pickaxes. Excavation depletes batteries, explorers have to recollect batteries to regain energy for their next expedition.
Excavation Mode
Excavation Mode is where explorers extract artifacts from rocks. Tap on the rock to excavate a selected surface of the rock. Tapping the rock decreases battery power and increases progress percentage. Taps artifact when progress percentage has reached 100%, obtained artifact fragment and goes back to the Map scene. Explorers need to be careful when excavating, if a user taps artifact when progress percentage hasn’t reached 100%, the artifact health bar lowers. If the artifact health bar reaches 0, the excavated artifact is broken and the user goes back to the map. Explorers can’t excavate rocks if they don’t have battery power left, there will be a choice to get the rock or leave the rock if this happened.
Combine Mode
In creating an engaging learning experience, explorers can collaborate with each other. By having a common goal to collect and combine artifact fragments, explorers will collaborate with each other where they can combine missing fragments into a complete artifact.
To combine fragments into an artifact, explorers can select which fragment in their inventory they want to combine and simply drag and drop the piece into the other fragment in augmented reality. Both explorers acquire the completed artifact which they can see the artifact in the encyclopedia and use the artifact to decorate their gallery. This feature hopes to increase social engagement, learning motivation and healthy competition between others.
Encyclopedia
Explorers can see all of their completed artifacts on the encyclopedia. Their phone acts as an adventurer’s log book & mini encyclopedia that tracks all of their discovered artifacts. The encyclopedia can be sorted into different museums where explorers can read interesting facts about their findings and observe the 3D model by rotating and zooming the object. After the completion of an artifact, it will show the whole information about the artifact including facts and its history. If an artifact was completed through collaboration, the completion date and collaborators are shown.
Gallery Mode
The Gallery Mode is a feature where explorers can decorate their own gallery based on their collected artifacts. Simply place a portal to the gallery on the tracked surface.
Edit Mode
Explorers can edit their gallery by selecting the Edit button after they placed their portal on the tracked surface. There are 5 slots in the gallery which explorers can decorate based on their choice. After decorating their gallery, specific gallery visitors will visit their gallery based on the category of the artifacts.
Gallery Visitors
In the decorated virtual gallery, special guests will also visit the gallery. They have different interests and personalities. Intrigue them by decorating your gallery with different kinds of artifacts in your gallery. They are more than just fictional characters, they need to be understood too just like human beings. Explorers can interact with them and learn more about their backstories after they become recent visitors in the Guest Book. Once explorers have accumulated a lot of visits from other explorers and special characters, they can unlock new features for the gallery.
Portals
After the alien attacks, their spaceships have broken the time fabric, leading to the creation of portals where explorers encounter on the map. They are panoramas of the pinpointed location’s past. Inside these portals, explorers might encounter collectibles such as batteries, sensor drives, or a rock of a rare artifact. The same portal can be accessed once in every 15 minutes.
Development Phase 1
When I was working on this project alone during my internship at the Smithsonian, I stumbled upon Niantic’s announcement on Twitter about their developer competition. What surprised me is the Real World Platform SDK enables shared experiences for both iOS and Android devices which has never been done before by ARCore, ARKit, or even ARFoundation.
I applied ExplorAR for Niantic’s Developer Competition around March. Another problem was the requirement of being in a team, which has to be at least 3 people per team who have prior experience with Unity and Java server development. Being the sole developer of the project, I almost had to leave the chance to apply, but I am curious about their SDK and proceeded to find a way. Although I didn’t know anyone who has more experience in Unity from my previous university, I decided to invite my recently graduated friends who knew Java server development to join the project, Adam & Ralph. I had to try my best to be the main Unity developer while being the Game Design Lead and Art as well for the competition.
The announcement of the top 10 teams was in May which was very surprising for us because we didn’t expect our team could be selected to participate in the competition. After the announcement, we got invited for their initial bootcamp session in their office in San Francisco. Ralph and I went to the bootcamp session in June. We were surprised that the other teams are professional indie game studios that have at least 6 person working full time on a project. We got intimidated at first, but we decided to move forward and try our best.
For the timeframe of the competition, we decided to work on the Outdoor Experience. In the Outdoor Experience, explorers engage with the museums and with each other by collecting 3D-scanned real-life artifacts which include fossils, statues, and other historical objects from museums. Explorers will learn how to preserve historical objects by extracting fragments of artifacts, collaborate with each other by combining fragments of missing artifacts, and express their creativity by designing their own virtual gallery.
Niantic’s Real World Platform
ExplorAR is built using Niantic’s Real World Platform, which includes a world-scale persistent state engine, cross-platform multiplayer AR, high-speed networking, and geolocation data. It uses Unity as the client side and Java-based Gradle Server as the backend.
Milestone 1: AR Experience
Based on the previous prototype which was made using Google’s ARCore SDK on Unity, I could port the existing experience into Niantic’s RWP SDK. We took a week to learn the SDK and try to get the AR excavation mode working for the first milestone.
Milestone 2: Geospatial Experience
This milestone is when things are getting interesting. A lot of chaos happened. I just moved to New York from DC and I had an acid reflux attack which caused heart palpitations and being hospitalized for 2 nights. At this time we haven’t fully developed the geospatial experience.
The backend was a great challenge. We had some difficulties where we were confused how the server is not updated, but it all fixed after we created a new virtual machine for the server. This is something that kept the team off guard where we focused on getting the basic features working instead of creating meaningful content for the second milestone where it was considered incomplete.
At first, we didn’t have much content on the map since our initial idea is to make AR as the main and primary experience with minor support from the map. We wanted to find new solutions to solve a user experience problem with most of map-based games where people seem to be more focused on looking at the map, rather than exploring on their own without looking at the map too much. Since we are still a part of this competition, we decided to include more content and gameplay for the geospatial experience. This is when we added batteries and sensor drives to the map.
Milestone 3: Integration
Due to the time constraints, we decided to get an additional member for our team. From IGDA NY meetings, I met Joe, who works at Viacom and he seemed to be very interested to join the project. He has a lot more experience than us in Git and Unity and he has been a very great help to the project. After we got more familiar with the SDK, this is the milestone where the real development begins. We believe we achieved a lot of progress in this milestone.
One of the most important achievements is the artifact model optimization. Raw 3D scans use a lot of storage which slows down development time. I used Blender to reduce the size of the 3D models using the Decimate tool. From raw 3D scans which has approximately 200MB in average into 2MB in average while still looking feasible. This has dramatically increased development time when pushing and pulling the project from the cloud repository.
We finally integrated augmented reality features with the geospatial map. Although we encountered several minor bugs, we finished a lot of core features for the game. We need a few finishing touches to complete the game for the last milestone.
Milestone 4: Polishing & Bug Fixes
We finished several features that we had initially planned to be implemented in the previous milestone. We also added several polishing to user interface, explorer avatar model, and map textures.
Since this is an immersive augmented reality game, we don’t want the user interface to be distracting, so we utilized Google’s Material Design elements. Amber & Light Brown conveys the antiquity of fragments, while the sans-serif fonts & icons represent the modern age which fits with the theme of the game to rediscover the past with an emerging technology. ExplorAR’s logo is based on a pyramid and a T-rex.
Although we didn’t have the time to work on the shared experience for collaborative combine mode & shared virtual galleries, we solved all the known potentially game-breaking bugs that occurred in the previous milestone. After we submitted the project, we had to prepare ourselves for the project presentation and the demo session during the demo day.
Conclusion:
We finally submitted the project on time and we got invited to Niantic’s office at the Ferry Building in San Francisco during the demo day. Although we didn’t place as the top 3 finalists, we’re glad to be able to test Niantic’s Real World Platform SDK and be a part of the competition.
First of all, thanks to Niantic for selecting this project as one of the top 10 projects. Thanks for giving us a chance to use the SDK and it was great to see the first platform that enables shared experience for both Android and iOS. We believe our project shares the same vision to create adventurous augmented reality games that have positive social impact. Hopefully our feedback could improve the future updates of the Real World Platform SDK.
I’d like to thank the Smithsonian Institution, Matson Museum, & Penn State Earth & Mineral Sciences to provide 3D scans for the future release of the game platform. The game would never have any meaningful content without their 3D scans.
And most importantly, thanks to Adam Musciano, Ralph Chamberlain, & Joe Francia for helping me building this project during the competition. It was great to be able to work on a project with my friends for 4 months. Since all of us worked remotely and we have full-time jobs as well, we managed to work on the project during weekdays from the evening until late night and spend the whole day on weekends. Despite all the hardships we faced, we can submit the project within the milestone criteria. It’s been a month and I already miss the chaotic enjoyment of developing a large-scaled game in 4 months.
From this experience, I’ve learned a lot of lessons as a game design lead. I was unprepared how this project which started as a small personal project into a large-scoped project. I also could’ve estimated better of how many features that could be developed within 4 months with only 4 developers who are working part-time. Now I have to take classes for my master’s degree and have to work alone again. Nothing lasts forever anyways, but I still want to continue this project.
ExplorAR’s Future Plans
ExplorAR has finished its first initial steps, although it is still far from being complete for launch. We almost got everything done for the Outdoor Experience and missed some minor features due to time constraints. We built ExplorAR’s Outdoor Experience throughout the contest, but I expect to have the Indoor Experience on launching the game.
Shared Multiplayer
During the competition, I haven’t got the chance to take a deeper dive into the shared experience due to time constraints and other milestone requirements. I will finally try to learn deeper about building shared experiences with RWP’s toolkit, HLAPI. This will be used for the shared collaboration mode and shared virtual galleries in the future.
Crowdsourcing Mode
Based on the previous user research, museum curators mentioned they need a crowdsourcing platform that allows users to contribute knowledge of a lesser known artifact. Explorers may come from different cultural backgrounds which would be beneficial for the museum if they could provide information about the artifact. Museum Curators are free to provide lesser known artifacts in the game platform for their research. If the museum finds the information from explorers to be valid, they will update the information to the artifact’s encyclopedia entry and the user will receive a reward from the museum.
Indoor Experience
In Indoor Museum Experience, exclusive artifacts can be encountered inside the corresponding museum where explorers answer interactive quizzes to obtain them. Indoor Museum Experience artifacts are exclusive inside the corresponding museum. Explorers need to answer quizzes to obtain these exclusive artifacts in the museum. Clues about the artifact can be found throughout the museum exhibition.
There will be additional experiences in the museum where players can interact with the museum exhibition, such as Time Portals, Glyph Translator, Dino Revival, & Style Transfer Camera Filters. These features will be developed in the future after the outdoor experience is developed.
Other Challenges
For a non-profit game, it requires proper funding from educational institutions for future updates and maintenance. I also need to reach out to more museums that could provide 3D scans of artifacts for more variety of collectible content. User feedback from usability tests is still an important aspect of the development of the project as well to test new features before launch.
The development will be postponed for now for finding new team members and restructuring the next requirements of the game platform. Since I just got into NYU during the competition, I believe there are like-minded collaborators who would love to join this project. I am still intrigued in creating meaningful experiences for museums with emerging technologies. I am planning to keep in contact with Niantic and use this project for academic research and potentially my final project for master’s thesis. If you are a Unity developer who are interested to join the ExplorAR Team, feel free to contact me! :)
Thanks for reading this article, it’s been a long time since the last time I wrote articles on Medium! I haven’t got the time to write due to my schedule, but I’m glad I actually made it for this one. This one is a bit longer than I expected because I combined both of the project details and my personal experience. I hope you could learn something from this article and from my personal experience. Feel free to follow me on Twitter (@adlanarv) and check out my other projects on my web portfolio (adlanramly.com)!
