Customer Information of the Future With Augmented Reality
A Report Straight From the Workshop
Augmented reality features that go beyond gimmicks. The combination of precise localisation and real-time data offers enormous potential in terms of customer information at train stations.
At the beginning of April, we took the plunge and made a new preview app ready for download in the Google Play Store. Named “SBB AR”, the app offers interested users the chance to test out and give their feedback on AR features. The idea behind this preview app is to validate use cases and have the opportunity to test out the usability of the developed features at length before they are then integrated into one of the existing SBB apps in a fully developed state.
Two features already live
The first feature can be used to scan the platform display boards. The detailed, interactive information on the respective service then appears below — virtually integrated into the real image — stops, delays or disruptions, as well as a forecast of the occupancy rate of the individual coaches.
The second feature involves the user looking around the station forecourt and directly viewing the individual stops of the local public services. The app also displays the upcoming services and departure times along with their stops, as desired and interactively.
So, How Did We Do It?
In the beginning, we asked ourselves — which technology are we going to use to develop the app?
In the past, people had to rely on libraries when developing an AR app, which worked well in some cases, and not so well in others. But nowadays, both iOS and Android devices innately support AR. On account of our expertise and our partnership with Google, we decided to focus on Android from the outset when developing the app.
This worked extremely well for initial tests but tied us closely to the platform.
In order to have as much future freedom as possible in this regard, we ultimately opted for a hybrid approach — native Kotlin code just for the user interface and Unity for the nitty-gritty aspects. The introduction, main menu, and much more had to be reimplemented for every platform. Thanks to Unity, the complex code remains independent of any of the platforms.
In practice, however, this approach posed a series of challenges. Although Unity runs on all possible devices, it also specifies the build process. In other words, it was difficult for us to infiltrate our UI code.
In the end, we settled on a multi-tiered solution:
1. Unity exports an Android project.
2. A script overwrites a range of files with the libraries and settings that we need.
3. Our UI code is imported into the project as a module.
4. The project is set up using Gradle and can be installed.
But that’s not all — there are a range of further problems that are waiting to be solved. This includes the fastest possible start, conflicting dependencies, hardware acceleration, and 64-bit support. (TK — are a range is BE so I left it along with the BE spelling.)
Challenge: Remote Testing
Our SBB Information Technology offices are in Bern, but the app was designed for the Zurich main station. This difference in location presented a challenge for the testing process.
The way in which we tested the app changed at various points during the development process.
Phase 1: In the beginning, we worked exclusively with the data that we had in Bern. Unfortunately, there’s not as much going on in the quaint town of Worblaufen as there is in the centre of Zurich, so this approach wasn’t sustainable.
Phase 2: Even after a relatively short period of time, we abstracted from our position to such an extent that we were able to hop back and forth on the (virtual) map and lead the app to believe that we were in Zurich all the time.
Phase 3: As soon as we started to recognise objects in the image through machine learning, this approach no longer sufficed. In a last-ditch attempt, we removed a few display boards from the station and installed them at our office, purely to test whether or not our app could correctly detect them.
That’s why we created a separate app that allowed us, for example, to take simple shots of the camera image and position and save them as binary files. Thanks to the existing abstraction, we were able to upload this data to the programming environment without great expense or time, and could then play out interesting situations from the office at any time.
Precise Localisation Is a Major Challenge
Precise location detection is the greatest challenge in the mostly closed, multi-storey station buildings. Google has supported SBB with localisation — the technology is based on ARCore, and the SBB project team has been in contact with Google’s development team.
In order to precisely position the display boards — where it comes down to a matter of centimetres — a combination of normal computer vision and machine learning was also used.
For us, however, this meant taking, intensively searching through, and annotating piles and piles of photos. After a while, we’d trained a MobileNetV2 with SSD to such an extent that it could successfully recognise indicators.
But this was not the end of the integration into our app — especially Unity. The chosen approach was to ultimately carry out the recognition in the native code (where the user interface lies dormant) and to communicate with Unity via JSON packets. In order to keep the latency between the worlds low, the two share a space in the memory — Unity inserts the last camera image and Android takes care of the content recognition.
However, the recognition only provides rectangles around the indicators and their numbers, which is not sufficient to place a 3D object in the right position. We, therefore, refined the result in several steps by filtering out bad results using the Point Cloud and analysing the image with OpenCV.
The entire app is based on observable streams. The image shows how the different processes are connected by streams and how data is passed on at each stage. If a new function is to be implemented now, you can simply latch it into any of the steps and refer to data.
Of Course, the App Needs to Be User-Friendly and Generate Added Value
In terms of usability and design, there are still few standards for AR. It’s clear that users shouldn’t be walking around with their smartphones in front of them. Not only would this be dangerous, it’s also not particularly user-friendly. That’s why the screen turns black today when the user starts walking around — but it shouldn’t do this when the user only moves slightly or just turns around. Ensuring this is a further challenge.
We are currently gathering customer feedback on how we should place the information. How should the presentation of the interactive information layer look in relation to the real environment on the screen? Should the information be displayed as if it were growing out of the platform display board, for example, or should it always depend on the user’s position? We are still trying to find the best balance here so that the information is easy to read and can still be assigned.
There are still many questions, but also bags of potential. We’re particularly seeing interesting applications in the areas of route guidance and orientation — where should the customer wait on the platform? Which exit should they take? What service is provided at the station?
Further features are in the pipeline — it’s not yet clear which ones will make it into the app. All features are set to be tested this year. We’re curious to see how it all works out with the AR app.