Automating Video Transcoding with AWS Lambda & Serverless
As a healthcare innovation company reinventing the point of care, Outcome Health is committed to delivering a high-quality patient experience via our devices. This means we are constantly improving Outcome Health’s technology platform, which involves managing hundreds of thousands of TV screens, handheld tablets, and digital wallboards playing static and video content tailored for audiences at the point of care. We deliver a massive amount of content, which comes from multiple vendors and sources — some in-house, some by our partners and sponsors.
The Pain Point
A major pain point that arose from working with so many sources of content was the array of video formats we were forced to work with. Our broadcasting standards require all videos be in a single format according to extremely specific requirements — content that doesn’t fit these specifications can cause our devices to crash. To prevent this, we use video transcoding and audio normalization to standardize all our videos.
For those who don’t know, video transcoding is the conversion from one video format to another. This can mean changing how a video is compressed (the “codec” it uses) from DivX to H.264, changing the container format from .wmv to .mp4, or just adjusting the number of frames per second the video plays at. Audio normalization can be thought of as standardizing how loud videos are, so none are much louder or softer than any of the rest.
The solution we had on hand through our chosen Digital Asset Manager (DAM — think of this as a library of digital content) service offered only video transcoding and not audio normalization. This was unacceptable. After a discussion with our DAM provider’s product teams, we determined that for them to create something usable that met all of the requirements would take more time than we had — the business couldn’t stop delivering content for such a long period.
The MVP Solution
After further discussions with our DAM vendor, we investigated their recommended solution, AWS Elastic Transcoder. As we navigated the main page for Elastic Transcoder, we saw an ad for AWS Elemental Media Convert. A click and a bit of research later, we realized Media Convert could handle all of our requirements. So, thanks, AWS ad team!
We took off running and began working on creating presets, job templates, and test jobs to tweak configurations until we were satisfied with the outcome and performance of our audio normalization process. The service was cloud-based (so, easier to manage), provided validation of each job (automated QA), and moved us away from our old solution that relied on outdated ffmpeg libraries that were no longer supported.
Failed Iterations of Our Solution
The lone drawback of this MVP solution was that the task still required manual intervention — someone had to push the buttons and add the configurations. To automate this process, we generated a quick Bash script that would extract most of those button pushes, however this still required a person to run the script at the appropriate time. The Bash script also required many permissions we did not feel safe handing out to less technical users.
A Cloud Solution to the Rescue
After taking a step back, we realized the cloud had gotten us this far — why not ask it to carry us the rest of the way? So after several whiteboarding sessions, we settled on a new workflow that triggered a custom cloud service we built using AWS Lambda, S3 Buckets, and Cloudwatch Event Triggers.
The new flow is simple:
- The content team drops the video file into the initial S3 bucket.
- This triggers a Lambda function (that only has permissions to create Media Convert jobs) and sends an alert to Slack notifying users that a job has begun for the newly added file.
- Once the job is complete, it fires an event to CloudWatch.
- The CloudWatch event triggers a secondary Lambda that reports the job’s status in Slack, and moves the initial file to either an “Error” or “Normalized” folder depending on the status of the job.
- The user then can grab the file from S3 if successfully completed, or find the original file that threw an error in an error directory within the initial bucket.
This new flow has a host of advantages. First, it removes all human permissions apart from uploading and retrieving media files to and from the respective S3 buckets, dramatically reducing security risk. Second, the service was built in the cloud, providing reliability and scalability — no matter how many files are added, AWS can scale as needed. Finally, reporting on the status of the service became easy with cloud-integrations — setting up a Slack alert to provide job status directly to users was a breeze.
Automating Our Automated Service
We had built a working, automated solution to a business problem. We could have stopped there, but chose to go even further by automating our entire testing and deployment process.
Using the Serverless framework, we were able to package the AWS services and infrastructure we were using into replicable code. This Infrastructure-as-Code concept made replicating the service and all the infrastructure it required as easy as a few short commands — dramatically shortening the amount of time it took to deploy to our Pre-Staging, Staging, and Production environments.
With Serverless, we were able to replicate and package the code, allowing for quick testing and deployment against each of our 3 environments (Pre-Staging, Staging, Production). By using custom stages within the serverless.yml file, we were able to apply environment-specific attributes.
Serverless isn’t perfect and wasn’t able to cover certain infrastructure resources. So, to compensate, we used Terraform — another Infrastructure-as-Code framework. Terraform created the missing resources Serverless was not able to create and glued everything together.
The final outcome of this effort was a simple configuration file that could be used to set up all the infrastructure required for this service. Whether we want to replicate this infrastructure on a new server, tweak it slightly and deploy it to a new instance, or just redeploy after a failure, we can do so with minimal effort. Without these new tools in our toolbox, we would have to rely on packaging our own code and creating our own environments manually through the old click-and-point method.
Outcomes & Results
The business value from our efforts was obvious to the content team — their new audio normalization service requires no human intervention, saving dozens of man-hours each week and reducing human error. By creating scripts to hold Infrastructure-as-Code, the DevOps team also saw massive reductions in the man-hours required to test, tweak, and maintain the service.
It is improvements like these that ensures Outcome Health’s network is growing and improving.
Lessons Learned & Looking Forward
Through the experience of improving our processes with new technology, we’ve learned a few things we thought we could share:
- Event-driven processes can be much better for certain use cases. The event-driven process we created with AWS Lambda enabled us to replace our current process. Chron-triggered processes would have been unable to do this.
- Infrastructure-as-Code is critical to developing services quickly. Without the rapid iteration time deployments via Serverless provided, developing our service would have taken at least twice as long.
- Iterate quickly to get feedback quickly. The main engineer on this project iterated through at least 4 prototypes before settling on the end service explained here. He wouldn’t have achieved the excellent result he did without those initial iterations providing him feedback on what was wrong with his product. Faster iteration time is better.
We’ll certainly be keeping these learnings in mind as we continue to build novel systems that help us improve patient outcomes everywhere.
