Cases and Tips for the “NHK AR Time Warp” App for Exploring the History of Asakusa (Android/iOS)
CONTENTS
Introduction
In this article, we would like to introduce and provide a technical explanation of the “NHK AR Time Warp” app for which our company was responsible for system development and design.
“NHK AR Time Warp” is an app that allows users to explore Asakusa by superimposing the townscape with various historical images, enabling users to explore Asakusa’s past and present. The app was used in NHK’s city exploration television program “Commemorative Photos Across Time — MACHIKODO Time Warp (まちかどタイムワープ)” which was premiered on July 17th, 2023.
Just like the elementary school students who participated in an experience event for the app, anyone is able to install it on a compatible smartphone device. Please note that the app will be operational from July 2023 until the end of August 2023.
■ Download the app for the historical AR experience in Asakusa
■ Short Trailer
■ CM
Development Background
The development of this app was inspired by a tweet from our engineer, Matt (@mechpil0t), who regularly conducts Research and Development (R&D) for various apps.
In Designium, it is a regular occurrence to witness engineers’s R&D cases, which subsequently acts as a catalyst for the creation of comprehensive services and product. This case is an example.
In 2020, Matt was conducting experiments with an app to verify various technologies and ideas developed through R&D. He had a personal interest in watching old footage from Japan. During this time, he noticed that many landmarks like Sensoji Temple’s Kaminarimon Gate are still well preserved today. This sparked the idea of using digital technology to overlay old footage onto the current environment, creating an intriguing concept for an R&D app.
Among the many old videos, Matt became particularly interested in the historical footage of Asakusa. The footage showed people passing through various gates in Asakusa, and he noticed that even in the present day, people were still passing through the same gates. He saw this as a special connection that transcends time, bridging the present and the past. This discovery became the catalyst for the tweet that led to the “NHK AR Time Warp” app’s development.
Apart from Asakusa, there are various old videos available from other locations as well. Matt has been conducting similar experiments in places like Kobe, Kyoto, Osaka, Nikko, Kamakura, and more.
Core Functions
■ Past videos are overlaid at the same location in the venue
When pointed at the Kaminarimon Gate or the main hall, footage from the past is displayed through Augmented Reality at their respective locations. Simply aiming the camera at the target building and pressing a button will reveal relevant footage on the screen.
■ AR Video Function that allows you to shoot as if you were in the past
When you capture a photo together with the AR video, it will be processed to have an old film-like effect, allowing you to take photos similar to the ones in this article. You can experience a time-warping-like effect, as if you have traveled back in time!
■ Video Commentaries
The app contains approximately 50 videos. Each video is accompanied by explanatory text, allowing users to understand the historical changes and transitions of each era.
When visiting Asakusa for sightseeing or other purposes, we encourage you to experience the near futuristic AR technology, as demonstrated in the videos below. The videos showcase how different footage is displayed based on your location. Please give it a try and enjoy this unique AR experience!
Technology and Ingenuity
■ About the function of “overlaying past videos in the same place”
To achieve the location-specific video display feature, we utilized IMMERSAL VPS (※1) — part of Hexagon. IMMERSAL VPS system is featured for large objects such as buildings.
(※1) VPS stands for Visual Positioning System, and it refers to a functionality that calculates feature points obtained from camera images and matches them with a set of feature points (Map) from previously captured images to identify similar locations. In simple terms, it is a capability that uses visual information to determine the current position, essentially pinpointing the current location based on what is seen through the camera.IMMERSAL provides an SDK (Software Development Kit) and has an official server, making it very user-friendly. However, in this app, we have made some adjustments to utilize it in a unique way.
1. Limiting the Range of the VPS Map
The created app covers a very wide area. Theoretically, it is possible to create one set of feature points (Map) covering the entire route from Kaminarimon to the main hall. However, we have divided it into multiple Maps. This allows us to easily update Maps when real decorations change and simplifies the process of creating Map data.
Creating Map data requires capturing the target building from various angles. In this app, we use approximately 20 Map datasets. If we were to combine all of them into one Map, even a single change in the decoration of the main hall would require redoing the entire shooting process, including Kaminarimon. That would be a lot of work to do it all over again.
2. Limiting the number of VPS Maps used for calculations
In this app, we perform the calculation of IMMERSAL’s feature points on the server. Although it is possible to align the positions using all Maps, we have not adopted this method. This is because there were critical issues when including all maps:
- It took several dozens of seconds or more for the calculations to complete.
- The server returned incorrect positions more frequently.
The 1st problem arose when the number of Maps increased to a dozen or more. The second problem occurred when there were multiple Maps with similar features. In places like Asakusa’s Nakamise shopping street, multiple locations had near identical decorations. As a result, very similar feature points were generated, leading to inaccurate positions.
To avoid these problems, we utilize the smartphone’s location information to limit the number of Maps used for calculations to those within a certain distance. Thanks to this approach, even if the total number of Maps increases, we can reduce the number of Maps used for a single positioning process to just a few, ensuring stable positioning.
3. Creating Map/Video Alignment Scenes
To ensure a seamless and accurate overlay of historical videos onto the current location, we have incorporated scenes dedicated to aligning the Maps with the videos. These alignment scenes allow users to fine-tune the positioning of the historical videos over the real-world surroundings. By utilizing this feature, users can precisely adjust the alignment between the historical content and the present environment, ensuring a more realistic and immersive experience while exploring the past and present of Asakusa.
4. Using a dedicated server for IMMERSAL
From a security perspective, we have set up a separate server for the app, distinct from the official IMMERSAL server. By using a dedicated server, we avoid any potential impact from sudden spikes in activity or high loads caused by other apps running on the same server. This ensures the app’s stability and enhances its security measures.
■ About the function of “shooting as if you were in the past”
The app applies special effects resembling old photos only to the part of the video that overlaps with the AR or full-screen view. To achieve this, it is necessary to crop the person’s portion from the captured image.
If the app only supported iOS and iPadOS, the implementation could be done using ARKit’s People Occlusion. However, this app targets both iOS, iPadOS, and Android. Unlike ARKit, ARCore for Android does not have a people occlusion feature. We considered other alternatives, but encountered issues with compatibility between them and the challenge of real-time processing on a wide range of devices, from low-end to high-end ones.
In the end, we decided to use Kaleido’s remove.bg service to achieve the people occlusion effect all platforms
This service’s API allows us to send photo data and adjustment parameters, and it returns an image with only the person cut out. This approach allows for a relatively simple implementation and ensures that the cutout accuracy is not dependent on the platform. It enables consistent results across different types of devices, making the shooting experience almost the same on all supported platforms.
ACKNOWLEDGEMENT
The story was also published on Antaeus AR✨
EDITORIAL NOTE
Hello! I’m Sakuma(@ppengotsu), an engineer at Designium. I hope the story have been clear and helpful for you in understanding the app development process. In addition to Matt’s initial tweet that sparked the idea, Designium are constantly developing various prototypes. We share these developments on our blog and the official Designium Twitter account, so please consider following us for updates and more exciting projects. Thank you!
This is Macao(@Macao_Cacao) from Public Relations at Designium. I asked Mr. Sakuma to explain the technical part in an easy-to-understand way for beginners, and he wrote an article that even a non-engineer like me could easily understand. I would be happy if you could enjoy the app and the story behind its development. Additionally, we hope that other engineers will use our experience to develop more interesting applications.
