Engineering an audio delivery system for a haunted drive-thru attraction using AWS and LPR
This year has seen the rapid evolution of contactless answers to a worldwide epidemic, and the attraction industry is not standing on the sidelines.
This is the story of how I was able to create an innovative solution for a local haunt using cloud computing, license plate recognition, local web caches (PWA), hot glue, and zip ties.
Introduction
The owner and operator of local haunt attraction Legends of the Fog approached me in April to inform me that they wanted to do a drive-thru haunt this year to hedge against the risk of opening during a pandemic. A drive-thru haunt? I’d never heard of such a thing.
They needed someone to design the audio system and have it synchronized, personalized, and delivered to customers’ cars for this year’s Legends of the Fog attraction. I’d done something similar years earlier with the hayride attraction so surely this was similar enough, amirite? Luckily, a pandemic gives me lots of free time.
I knew it would be a miracle to deliver something with only 4–5 months to be ready for the public, but feigned confidence as best as possible. I’ve delivered many successful projects to the haunt over the last several years but this was at a whole new scale, timeframe, and on a haunt attraction budget. In the past I used IR signals to trigger audio tracks on individual wagons, but this was delivering synchronized sound effects to hundreds of unmodified cars over customer owned devices as they drove through the attraction. Challenge accepted.
Vehicle Tracking
After brainstorming sessions and prototyping, we settled on using LPR (license plate recognition aka ANPR) for the solution to be used for tracking a car’s progress through the attraction. Other solutions considered included beacons, GPS, and QR codes. License plates however offered a perfect solution. All cars are required to have them clearly displayed in a known location, format, with a reflective coating. We wouldn’t have to touch customers’ cars, and in addition technology has advanced to the point where LPR is quite affordable and accurate.
Prototypes of plate reading included some Raspberry Pi + OpenALPR/Watchman trials, some other cloud solutions, and NVIDIA Jetson Nano Dev Kits + onboard LPR driven by Plate Recognizer (a subsidiary of ParkPow, Inc). One of the key requirements of this system was that all LPR had to be done on device. The property used for the attraction is very large so local bandwidth and even Internet connectivity was extremely limited and fragile. We could not stream high quality video to be processed off-site because of latency and bandwidth.
Plate Recognizer was in the process of developing a Jetson Nano Docker based real-time stream LPR system and I was able to hop in a beta/trial run of this. I was very happy with the results. It gave me amazingly accurate results with a real-time integration point via a web-hook, and ran on device in a docker container for simple setup, configuration, and support. It did not stream the video anywhere — all processing was done on device. It was decided that the LPR part of this project would be handled completely by Plate Recognizer’s Stream product running on NVIDIA Jetson Nanos.
The Jetson Nano devices are only $99 each and the stream software was licensed affordably per camera per month. Toss in some $60 cameras from Amazon, a big Li-ion battery to last an evening and it is still shockingly inexpensive (to hot glue, Velcro, and zip tie together). This freed me up to focus on my strong suit, the back end services and web delivery system.
With what I considered to be the most difficult and fragile part of the system (the LPR) figured out, I needed to create a customer facing solution that could be delivered to hundreds of cars simultaneously and thousands of cars nightly.
Audio Delivery
With only 4 months to go, we eliminated developing an iOS and Android app. Getting through the app store would take time and developing multiple solutions and putting them through testing was simply not feasible. So we settled on building a mobile friendly website.
Building the front-end website came with another challenging requirement: customer cellular connection on-site is horrible. Yet I’m supposed to deliver audio from a webpage... Luckily to solve this problem we can use PWA technology, specifically pre-caching through service workers. When a customer visits the attraction they’re directed to a webpage which takes a minute or two to load because it is caching all of the assets required up front. When it is done loading, it’s is done for good and all requests for the necessary assets (sound files primarily) will be directed to the cache.
Our application became 2 pages. One to wait for everything to load and the service worker to start, then another page that creates a connection with the server and plays sounds when told. We now needed to somehow link the car’s license plate to the mobile phone’s web browser.
Creating the relationship between the customer’s web browser and their license plate proved to be one of the most challenging problems to tackle. We iterated through several ideas including:
- Having the customer type in their license plate. This would result in everyone in line exiting their vehicle to look at their own license plate while creating a traffic jam (only to make typos anyhow).
- Having the customer simply press a button as they passed a sign with a reader just before it. For anyone that does work in UX you can understand that users will push the button when they want to push the button. All of my testing showed that people like buttons. They like to push them when not told to see what happens. They like to push them, then press back. Text explaining not to do this is read as much as a terms of service.
- Displaying a QR code on the customer’s phone which is an identifier that the server understands. The customer displays this QR code at a scanning station running a desktop application that sends the identifier to the server to link to the last plate read at the initial LPR device, which is located immediately preceding this station.
We chose to go with the last option which gave the added benefit of gating the entire web experience from anyone being impatient. No button for you. Customers are familiar with the ticketing-like flow of this experience. Given more time, I’d bake the entry ticketing right into this process.
This did additionally require writing a quick scanning application in .NET Framework to run on a Windows laptop and to procure hardware QR scanners. This desktop application simply uses a single method which tells the server to link the device to the plate. Immediately after doing so, our system begins the guest’s experience.
Other challenges included detecting a page refresh to handle a “resume” and how to get around mobile sound restrictions surrounding requiring UI events to play sounds. I’ll go over these in details in a future article if I find free time.
Backend Software System
The design of the backend system would be a web server component which
- Served up our pages to the customers
- Managed active customer connections/sessions
- Created/received messages about linking or sound playback
- Provided a crude admin view of the system
- Handled incoming web-hooks from the cameras themselves
I also decided to make a “controller” back-end application completely independent of the website which would manage all of the logic about who gets what sound, when, and various events that take place when a camera sees a plate. In testing this system, the number of exception cases which need to be handled quickly becomes realized, the simplest of which is a plate reader missing a plate. Our controller will be in charge of figuring out what to do in all these scenarios leaving our little web servers off the hook.
Because my bread and butter is as a C# dev, I created the back-end web server using free tools: Visual Studio, .NET Core 3.1, and the SignalR library. This solution can run in docker containers (perfect for cloud based deployment and scaling) and SignalR is a websocket based eventing and two-way communication framework that gives me quick and easy real-time communication between the server and the JavaScript client with built-in retry and fallback logic.
The controller was similarly written using C# and was able to also use SignalR for communication out to clients since we also used Redis as the backing/eventing store. To say that another way, the controller component, running in a different instance from the web component, could create a SignalR Hub and trigger messages to clients even though it maintained no connection to clients. Note: this is due to SignalR using Redis and is a weird configuration worthy of an article by itself.
I needed a way to keep track of where cars were, what variants of what plates had been seen, and communicate between all components reliably. In memory caching was considered but eliminated due to the desire for high availability — if a controller fails I can’t simply start over with 50 cars in the middle of the attraction.
Redis allows for fast data storage and retrieval and is ready out of the box to integrate as a session store with .NET, a messaging bus for SignalR, and arbitrary data storage. Redis also comes with efficient sorted set combine logic directly in the cache layer which I was able to leverage for license plate match scoring. I’m effectively using Redis as a short term cache and data storage since I do not need any data to be persisted beyond the scope of a single drive through. We do NOT store plates, names, PII, images, etc. We store nothing. When the last customer of the night makes their way through the exit I don’t stop with deleting the data — I terminate our cloud infrastructure and all data that was processed by it. In this day and age I’m a little bit proud of that.
To add just a bit of fancy to the audio delivery I wanted to make some sounds dynamically generated using the customer’s name. This could have been implemented at the .NET core application layer, but to be fancy I stuck it in a standalone lambda. This calls at most two dozen lines of NodeJS code including the error checking which calls AWS Polly to instantly generate sound tracks which use the customer’s real name in playback; upping the ante on the “wow” factor we’re providing to guests.
The audio generated and the names of customers are not stored. It is streamed to the phone and then discarded.
Server Infrastructure
So now I had a solution developed but needed to run this 6 hours at a time and only a couple of days per week (attraction evenings). Ten years ago this would have meant securing a couple of servers in a warehouse somewhere connected by an internet pipe. We’d be looking at easily $20k in server hardware and at least $1k per month in co-locating costs. This would have made attempting this to be uneconomical for a small business and would leave developers such as myself being shackled to large corporations who could afford to bring our software to life at large scales.
This is where cloud computing is changing the industry. I could develop a solution that temporarily created the infrastructure needed to run my system for fractions of a dollar per hour with zero upfront costs.
Our final hardware solution was created on Amazon Web Services and consisted of:
- Elasticache Redis cluster for low latency, transient data caching
- EC2 for server instances
- Autoscaling group for dynamically adding server hardware on demand
- Elastic Container Service to run docker containers as services, scaling the website across a cluster of server hardware.
- Load balancer for routing incoming requests to docker containers within the cluster.
- S3 for storing and serving static assets
- Polly/Lambda@Edge for personalized voice track generation
- Cloudfront for CDN edge caching
To spin up this infrastructure and run it for 3 months, 24/7 would be somewhat costly and completely unnecessary. Using Terraform I was able to describe the infrastructure and have it available on demand. Since no data is stored within the system, I can completely tear it down at the end of the night. Terraform also allowed me to introduce variables so I could have modes of operation. This included things such as standby mode (all requests go to “Offline page”), all the way up to high availability mode which set clusters, scaling modes, and instance counts to create a fault tolerant solution for nights that the system is active. Terraform is available as a free cloud service and integrates with CI in GitLab or Github to automatically run your scripts when changes are pushed.
In the end, I have a cutting edge software solution with a backend hardware runtime cost of under $0.30 per hour. That’s not a typo.
Summary
By combining the power of NVIDIA’s small device graphics processing, Amazon’s AWS cloud computing, open source software, and new technologies defining PWA websites: this haunt will be able to stand out in these strange times of social distancing and provide fun, safe memories to our visitors.
For more information about this Baltimore area haunt, and to order your tickets today, check out Legends of the Fog.
I talked a lot about my role in the project but this would not have been possible without the help from the team at Legends of the Fog and most directly my fellow coding nerd Bob Gates and audio engineer Alicia Crowl. This was a team effort only possible collectively. This article didn’t even touch on the months spent doing audio editing, voice acting, wifi mesh network setup, desktop installs, hardware procurement, bill paying, help with coding, scripting, device building, hot glue application, and zip ties (thanks Matt!) by dozens of others. This article isn’t meant to cast a shadow on their roles in any way.
All of this was just for delivering the audio — a tiny portion of the complete show taking place this month at Legends of the Fog. Hundreds of actors come together to bring this drive-thru attraction to life.
It was quite the project, and I’m happy with how it turned out. I’m sure it won’t be without its bumps along the road but it was a fantastic project spanning several of my favorite engineering tools. I’ll give an update after the season is over complete with a lessons learned.
*Patent Pending
Interested in more detail? Please let me know. I’m passionate about collaborating with others on uses for technology and I’d be happy to follow up with more detail including but not limited to:
- AWS centric version of this article
- Details on LPR device creation/setup
- Developer focused version of this article where we take a deep dive into PWA, source control, docker, Terraform, CI, and Redis. Ok, maybe a series of articles.
No words or links on this page are paid endorsements. I’m just passionate and happy about most of them so I’m sharing freely.
