Development of an AR application for a museum
I would like to share my experience on creating AR App for the Museum in Rome. In the vibrant world of technology and cultural exploration, my journey into creating an augmented reality (AR) application for a museum in Rome has been an exhilarating endeavor. This article aims to share the insights, challenges, and triumphs encountered during the development process, shedding light on the transformative power of AR in enhancing the museum experience.
You can also check out the project on Behance for a visual journey through its design. Visit the link above for an interactive experience. Feel free to leave a comment if you have any questions or suggestions. Maybe we can continue to work on this project together or work on some other project.
The idea of this project is to develop an AR mobile application for the Sapienza Museum of Comparative Anatomy “Battista Grassi” visitors that will make the museum experience more engaging and informative.
We divided the project into five phases, and each phase had several tasks.
- Study Augmented Reality Technology
- Research Other Museums’ Experience
- Design and Prototype Development
- Application Development
- Testing and Iteration
Augmented Reality technologies
Augmented reality can be categorized in various ways, however, broadly speaking, the technology’s functionality can be classified into marker-based and markerless approaches.
Marker-Based Augmented Reality is when content is triggered when a specific marker comes into the camera’s field of view. Markers can include: images, QR codes, objects, faces, bodies, spaces.
Markerless applications operate without the need for marker recognition. For instance, Plane Detection involves scanning the surrounding surfaces within the application, onto which needed 3D models are overlaid.
Location-based augmented reality when a GPS position on a map is used to launch AR experiences.
AR tools for Artists
Snap Lens Studio and Adobe Aero are useful instruments for AR artists. These tools are primarily designed for AR experiences and interactive content rather than full-fledged mobile applications. So, these tools are not suitable for our purposes.
AR SDKs
ARKit, ARCore, Vuforia, and AR Foundation are robust frameworks for creating augmented reality (AR) applications. ARKit excels on iOS, offering advanced features, while ARCore, designed for Android, provides motion tracking and environmental understanding. Vuforia is known for its versatility, supporting both iOS and Android with marker-based and markerless tracking. AR Foundation unifies development on Unity, supporting ARKit and ARCore, ensuring a consistent AR experience across both platforms. Each framework brings distinct strengths, enabling developers to explore creative possibilities in AR.
I found a lot of information and video tutorials for Vuforia in connection with Unity. So, for augmentation I decided to use exactly Vuforia. I chose to develop the App on this stage only in Unity, as for me it is easier and for the first test version of the App it is more convenient tool for me to use.
At the same time as a future step I think that the approach of using React Native for most of the application and integrate scenes developed in Unity when it is necessary. This is an effective strategy that leverages the strengths of both development environments. This allows to take advantage of the flexibility and efficiency of React Native for the UI and general app features while using Unity for parts that require advanced graphics or AR, where Unity excels.
Devices
AR experiences can be enjoyed on smartphones, tablets, AR glasses, smart glasses, each offering unique interactions. Meta Quest 3, a standalone AR/VR device with built-in sensors, controllers, high-quality optics, and a vibrant developer community. Apple Vision Pro, a groundbreaking device, excels in entertainment, productivity, learning, and creativity with innovative features like eye tracking and hand gestures.
What problems can solve AR?
- Training and education (interactive and immersive training)
- Product design and prototyping (visualize and test)
- Remote collaboration and support (remote collaboration and support)
- Retail and marketing (allowing customers to see how products would look in their homes or on their bodies before making a purchase)
- Healthcare (training of medical professionals and assistance in surgical procedures)
Augmented Reality in the Museums
Examples of AR applications in Art Museums:
- Snapchat at Louvre (visitors can use their smartphone cameras to animate and interact with famous paintings and sculptures)
- The National Gallery in London utilized AR to showcase its collections beyond the museum walls
- Smartify (visitors can scan artworks using their smartphone cameras to access additional details)
Here are several ways in which AR can be applied in Historical Museums:
- Recreation of historical moments
- Interactive maps of historical events
- Visualization of artifacts
- Living stories from characters
- Virtual museum exhibitions
- Educational quests
- Enhanced information about historical figures
When I began studying the topic of augmented reality and exploring the applications already in use in museums, I decided to visit three museums in Rome: Circo Maximo Experience, Terme di Caracalla in 4D, Mausoleo di Cecilia Metella — Virtual Tour. In Terme di Caracalla they implemented App with virtual reality about 7 years ago. In Mausoleum of Cecilia there is App for Meta Quest 2 with virtual reality which was implemented one year ago, and the quality of the models is very high. Out of the three museums I visited, the most interesting implementation for me was at the Circus Maximus because they used AR (this project was done 5 years ago with Unity).
REVIVRE App
The Muséum national d’Histoire naturelle in Paris features the “REVIVRE” augmented reality app, providing an immersive encounter with life-sized 3D animated models of extinct animals. From insects to mammals, users can interact closely with the creatures, gaining educational insights into their behavior, ecosystems, and extinction reasons. This innovative use of augmented reality not only enhances the museum visit but also promotes biodiversity appreciation and nature conservation awareness.
Skin and Bones App
When the idea to develop an AR application for the Sapienza museum struck, I discovered a similar app already in use at The Smithsonian Institution. Called Skin and Bone, this AR app transformed the museum’s Bone Hall exhibit, allowing users to frame skeletons with their devices to see reconstructed animals in the same position and dimensions. Skin and Bone featured 13 skeletons, offering an immersive experience that depicted how skin, muscle, and movement would have looked. My test of the app on my iPhone revealed some limitations, including the need to locate animals on the museum map or list, a small selection of 13 animals, and the absence of a search function. Despite these challenges, the app provided a unique glimpse into exhibit history, showcasing the potential of AR technology in museums.
The Museum of Comparative Anatomy “Battista Grassi” in Rome features an extensive collection of anatomical exhibits. The museum aims to provide educational resources, allowing visitors to study animal structures and understand their biology and classification. In a bid to innovate, the museum is considering integrating augmented reality (AR) technology for interactive exhibits. An AR application could offer virtual dissection and detailed exploration of anatomical structures, enhancing the educational experience.
After UX research I identified Key Features for AR Museum App:
- Skeleton Recognition
- Rich 3D Models
- Audio Integration
- Interactive Manipulation
- Educational Information
- User-Friendly Interface
- Compatibility Across Devices
- Guided Tours (optional)
- Feedback Mechanism
In excalidraw.com I created a low fidelity prototype to test the design ideas.
I chose three main colors — Oro, Greige, and Rosso — for the application interface, drawn from the Sapienza and museum logos for a strong visual connection. Meticulously crafted icons and buttons ensure a simple and consistent user experience within a visually simple layout. The application logo features a whale, the largest exhibit in the museum.
For testing purposes, a prototype in Figma was developed for two flows.
Flow 1: Select Animal and Market Scanning (Image Target):
Users choose an animal upon entering a room. They then scan the market using image target technology. This sequential flow allows focused exploration, emphasizing one animal at a time.
Flow 2: Immediate Marker Scanning (Multi-Image/Area Target):
Users can initiate marker scanning upon entering a room. This flow caters to dynamic exploration, accommodating multiple markers or area targets for a swift and immersive experience.
Implementation
Setting up and configuring Unity
- installed the latest stable Unity version
- created new 3D project
- imported in created project Vuforia Engine from Unity Asset Store with Package Manager
Work with the Vuforia SDK
- registered on the Vuforia portal and created a license key
- created Image Targets Database for tests
- for Model Targets I used Vuforia Model Target Generator App
- to generate an Area Targets, I purchased an iPhone 15 Max Pro since Vuforia Creator App (which allows the generation of Area Targets) is exclusive to devices with built-in LiDAR sensors like iPads and iPhones
Development of the App
- added Vuforia Camera and configured it
- added AR Targets
- added Augmented Reality Object to Targets
- created 3 different menus on different separate scenes in Unity
To build the Application for Android I configured the project in the appropriate way. Configured the Scene Order, modified Player Settings adding a series of icons in various sizes was specifically created using the application logo.
To promote the AR experience in the Museum I created Website (desktop and mobile versions).
In summary, the Museum of Comparative Anatomy AR project with Unity and Vuforia revealed insights into optimizing user experiences, integrating AR with traditional exhibits, and leveraging technology for educational enrichment. The discoveries made during this project lay the groundwork for future developments in the intersection of augmented reality and museum engagement.
The future development plan for the AR mobile application for Museum of Comparative Anatomy involves strategic enhancements and expansions:
- Content Enrichment (increased number of recognized skeletons, add more 3D models)
- Add audio (sounds of animals)
- Add other options for interactions with 3D models
- Add Educational Information
- Test and improve Interface
- Create Apps for other devices
In a subsequent phase, as a second step, interactive interaction with the animal could be added. By answering a question, the animal might follow the person. The more correct answers obtained, the more animals will follow the person. At the end of the museum visit, it will be possible to get a photo with all the animals that have followed them.
