Enhancing a Palace using 5G and AR
Through a collaboration with SK Telecom, Google, and the Cultural Heritage Administration, we have transformed the UNESCO world heritage site, Changdeok Palace (South Korea), into a 5G enabled, AR-enhanced cultural location.
I’d like to consider Changdeok Palace our second ‘Enhanced Location’ after the AT&T Stadium in Dallas. This time instead of sports we were able to demonstrate the potential impact of the ARCloud concept on products designed for the museum and cultural industries.
Changdeok Palace is the first 5G AR-enhanced Tour Experience.
Changdeok Palace is set within a large park in Jongno-gu, Seoul, South Korea. The world-heritage site has 13 buildings and 28 pavilions in the gardens, occupying 110 acres (45 hectares) in all.
On arrival, visitors are guided around the palace grounds by an enigmatic mythical creature — Haechi, who not only wayfinds but imparts the stories of the Joseon dynasty. Walking between a variety of locations on a digitalized ancient map, visitors can interact with members of the Joseon court and discover ancient artifacts as if they were still there.
The app runs on the following 5G phones: Samsung S10 5G, Galaxy Note 10 5G, LG V50 5G. It’s using Google’s Cloud Anchors and ARCORE and was built using the Unity Game Engine. There are more details on the project and the 12 unique experiences here.
This post is kind of a follow up to a previous one I wrote about ‘enhanced locations’ using hyper-accurate AR and 5G. You can read that post here.
Meet the Haechi
A mystical creature and a tour guide, the Haechi will direct you around the grounds of the palace and provide contextual commentary based on your location.
Haechi was a dream product feature; a digital tour-guide with a contextual understanding of its own location. It was also an opportunity to build upon HotStepper, which was the first AR wayfinding app, released back in 2017. I have loads of love for HotStepper and I hope perhaps in the future we can upgrade him with these new superpowers — but for now, we have Haechi.
HopStepper used GPS to navigate the world (and a few smart hacks here and there). That was ultimately the limitation of the product — a guide needs to be able to direct you precisely and take in other information in real-time that might be useful i.e. crossing a road or traffic light. HotStepper would often walk into a road (with cars) only to turn right five meters too late — it was hard to keep him on the sidewalk/pavement.
Just like our work at the AT&T Stadium, Changeok Palace is using a powerful visual positioning system, this time it’s Google’s Cloud Anchors. You can now think of Changeok Palace as having its own private ARCloud — content can be placed anywhere with supreme accuracy but it is also persistent and shared between users.
For Haechi (or HotStepper in different clothing) that’s a big upgrade, it’s as if he now has eyes and can see.
This was truly ground-breaking. It allows visitors to be guided all over the 110 acres site by a virtual character; leading them precisely to their destination as well as providing contextually accurate commentary (voice-over) about the things they can see right now.
The guided experience is also on-demand. You can look at the ancient map on your phone, tap any one of the AR-enhanced destinations and Haechi will immediately take you there.
One challenge with AR wayfinding is accounting for changes in elevation i.e. going up some stairs. Let’s start with the visual problem. I still think the best AR wayfinding character is a bird, simply because its feet don’t need to touch the ground. Even with a highly accurate mesh of some steps, it’s going to be nearly impossible to make the feet look like they are walking up the steps — in reality, they’ll either float above the steps or pass right through them. So the team developed a creative solution and I think it’s beautiful — he jumps up them.
So let’s address the technical problem of changing elevation — perhaps you noticed it already? ARCore is responsible for this one. The tracking gets confused as you begin to walk up the steps (where’s my flat surface!) which can result in Haechi either changing size or appearing to float. There’s not a lot we can do about this but of course, it will get better with time.
To wrap up Haechi I want to quickly talk about the speed of relocalization using Google Cloud Anchors. The following clip shows the moment Haechi walks you to the entrance to Injeongjeon Hall to meet the King and Queen. It is so exciting that we now have the accuracy to make an experience like this so seamless. The camera probably begins to detect the feature points of the hall around 6 seconds, and by the 8 seconds, the voice-over triggers their arrival. 🤯
Holograms
Through-out the palace you’ll meet kings, queens, princes to play archery with, dancers and more — all streamed over 5G using our proprietary compression and streaming solution.
For the Dallas Cowboys, we got to render 80ft holograms to emphasize their heroic status as the team's star football players. Changdeok was almost the identical opposite. The objective was to reimagine with authenticity the key characters in the Joseon dynasty.
The passion and care by all the teams to maintain the level of authenticity while still pushing the boundaries of resolution and capture is clearly shown in the Dance of the Spring Nightingale. Ideally, if you want to get the best scan you’d dress your actors in a material that wraps the body — these types of costumes on paper should be the worst!
There’s still a lot more to tell on this one, but I hope I’ve been able to provide some early insights.
I’ll finish by saying this project was delivered under Covid19 conditions — a small miracle by all the teams involved. And we’re just finishing final touches on an at-home version too, so soon anyone anywhere can learn about the Changdeok Palace and the Joseon Dynasty.
