Dec 17, 2022

9 min read

How to Deploy a Video Generator in AWS with Terraform

Overview

In this tutorial, we will learn how to spin up an infrastructure to deploy a highly available, multi-zone resilient web application using Terraform. Terraform is an open-source Infrastructure-as-Code (IaC) tool that automates many laborious tasks on the cloud such as creating, modifying, and provisioning the infrastructure. You can also work with multiple clouds using Terraform! Here, we are mainly focusing on AWS Cloud.

A bit more context before diving into the architecture:

The web application we want to deploy is a video generator, where users can log in, and upload a photo of a character, and a script. The video generator will then produce a short video describing the story of a character (sounds like the film/anime companies are losing their jobs!). Of course, this money-making secret video-generating algorithm is our top secret, so we want to ensure no users can access it. At the same time, we want the application to have high availability, scalability, and fault tolerance to be ready for a crazy amount of visits 🔥

Prerequisites

AWS account and credentials
AWS CLI installed and configured on your IDE
Terraform installed on your IDE

This article aims to only use available services in AWS free tier. Be sure to follow all steps in this tutorial to destroy the up-and-running services and instances to avoid costs. See more prerequisites help guides/documentation/commands output in the Code section below.

Architecture Explained

High-level Application Component Architecture

Figure 1. Web application components architecture.

We will walk through the major components, their functionalities, and why they are included, from top to bottom (feel free to navigate through the list of components at your own pace 😎).

Domain Name System (DNS), like a phonebook of the Internet. After users type in ‘www.videogenerator.com’ (for example) into the web browser, DNS converts the domain names into IP addresses and loads our webpage. We do not go into specifics about networking in this tutorial (link for more info).

Content Delivery Network (CDN) is preferred in this scenario to cache the web server content to reduce the latency in rendering the content (especially since our users might access our web application from all across the world). The static or dynamic content is stored in the cloud (AWS CloudFront for example).

Load balancer acts like a proxy and automatically distributes the incoming application traffic, contributing to the high availability of the working nodes.

Multiple instances of the Web Server and Video Generator will be deployed using EC2 and the Elastic Load Balancing helps to route the incoming requests optimally so that no individual instance is overwhelmed.
The load balancer will also be attached to the auto-scaling of cloud containers, so that the controller scales up or down facing different amounts of requests.

Web server is the front-end interface exposed to users, which enables the user to upload character photos and scripts and click the button to start video generation. The file data will be directly transmitted to object storage. The IDs and the auto-computed hash value will be stored in a relational database for queries (to avoid the same video being generated twice). After successfully uploading the required materials and selecting `Start`, the request will be sent to Video Generators.

Figure 2. Ideated front-end interface.

Video Generators (not exposed to users) retrieve the file data from the storage and start processing. The generated video will also be stored using object storage (with IDs stored in the database to retrieve the correct one to display on the front end). When the generation is finished, it pops up in the embedded window on the interface for viewing.

AWS Cloud Architecture

Figure 3. AWS Cloud Services Diagram

To illustrate a more concrete deployment plan, I drew another diagram as shown above. Let’s see what AWS Cloud services are used here and their advantages:

Region: where the computing resources locate. For high availability and low latency, the video generator will be run in multiple availability zones within different regions, maintaining a multi-region Amazon EKS cluster (see Figure 4). In this way, our site stays up even if a data center in one of the AZs goes down!

Virtual Private Cloud (VPC): build a virtual private network to ensure that the web application is secured and isolated.

In this scenario, a VPC with a public subnet and a private subnet is recommended. Web Server goes into the public subnet for users to interact with it, whereas Video Generator will be in the private subnet so that they are not exposed to the rest of the world (remember our top secret huh?).
Internet Gateway attached to VPC, so the Web Server in the public subnet has access to the Internet and S3. NAT Gateway is configured to enable the private subnet to connect to the public subnet (one-way connection). Due to security reasons, we do not want the private subnet to go over the entire public subnet to connect to S3, so the VPC endpoint is used instead.

Elastic Kubernetes Service (EKS): EKS is selected to facilitate deploying, managing, and scaling containerized applications using Kubernetes on AWS. As mentioned above, the major advantages are: 1) by creating multiple replicas across multiple availability zones, the server never goes down; 2) enables on-demand auto-scaling, updates, and patching; 3) provides a managed control plane to monitor and provision nodes.

Figure 4. Abstract Amazon EKS multi-region deployment.

In Kubernetes, each pod runs a single instance of the given application, and multiple replica pods are maintained so there is no single point of failure.
A service defines a logical set of pods and the relevant policies to access or target specific pods.

Object Storage S3: Because we are dealing with large volumes of unstructured data including images, scripts, and videos, object storage is ideal, which can store large volumes of data, with each piece of data stored as an object, and the metadata provides a unique identifier for easier access.

Note that the S3 buckets are made publicly available so that users can access the generated videos with provided URLs. I imagined that the generated video would be of excellent quality and there will be numerous repeated views, so directly making S3 buckets public will reduce the intermediate networking complexities. However, this action of granting public access to any users without authentication also has some security risks (S3 security best practices guide).
When uploading the images and scripts, S3 uses a Checksum Retrieval function to compute a checksum for the file (MD5 or SHA-256), which can be used to compare upload file data to avoid generating the same video twice.

Amazon Relational Database (RDS): RDS is selected here since the data is structured and frequent queries are needed. The table includes 1-to-1 relationships (assume each entry is one image to one script) and two corresponding hash values:

When users upload a script and an image, the Web Server will send a query to the RDS: only if the new upload has at least one different hash value (users might use the same popular image), the request will be sent to generate a new video, to ensure that the same video is never made twice. Otherwise, it means that the video has already been generated based on the same materials, so the Web Server will directly retrieve the Video ID and display it on the front end.

AWS Batch: AWS Batch used a first-in, first-out (FIFO) strategy to control and run high-volume batch jobs in the compute environment, which can be organically integrated with Amazon EKS. Based on the queued jobs, AWS Batch then launches worker nodes in your cluster to process the jobs, enabling the system to have much better utilization and scalability.

Code

Pre-requisites Install & Set Up Guide

Install AWS CLI and configure credentials: check guide

Make sure AWS CLI is installed successfully: $ aws — version. Output will be:

Terraform installed: check Install Terraform

Verified terraform is successful install by using this command: $ terraform — version. Output will be:

I mainly used Terraform to set up the cloud architecture and I attached a GitHub Gist to each step. Terraform code is written in the HashiCorp Configuration Language (HCL) in files with the extension “.tf”. It use declarative language, meaning that we define what we finally want and Terraform will do all the details.

Step 1: Setting up AWS CLI

Configure an Amazon IAM to interact with AWS: check guide

$ aws configure — profile VideoGenerator

The profile will be used in terraform provider for authentication.

In the provider.tf, we configure the provider to be Amazon AWS, which allows us to interact with AWS resources.

Step 2: Network Infrastructure Setup

Create AWS VPC (Virtual Private Cloud).
Create two public and two private Subnets in different availability zones (Amazon EKS requires the subnets must be in at least two different availability zones) to ensure the high availability of our site.
Create Routing Tables and Security Groups and associate subnets with them. Add required routing rules. Notice that to I added separate security groups for public/private subnets, data plane, and control plane and defined traffic rules within each security group.
Create an Internet Gateway to provide internet access for services within VPC.
Create NAT Gateway in public subnets. It is used in private subnets to allow services to connect to the internet.

Step 3: EKS Set up

Create an EKS cluster named eks_cluster.
Set up the IAM roles and policies for EKS:
IAM Role: Create a Role with all needed permissions that Amazon EKS will use in managing Kubernetes and making calls to other Amazon services.
IAM Policy: Attach the trusted policy for EKS to use the IAM Role.

Step 4: Worker Node Set up

Create worker nodes to run the application workload.
Similarly, set up the IAM roles and policies for node groups:
IAM Role: Create a Role with all needed permissions for nodes to communicate with other Amazon services.
IAM Policy: Attach the trusted policy for the EC2 instances to use the Role and the AWS-managed permission policy.

Step 5: Create EC2 Instances

In this step, we create the EC2 instance that is launched into the public subnet. The EC2 instances have Amazon Linux AMI and instance type t2.micro.

Step 5: Define Variables, Data Sources & Output

As some environment variables are frequently referenced such as region, I defined them together in vars.tf. Create terraform.tfvars for Terraform to automatically load them to populate variables.

Other definitions include data source and outputs (required to connect with the EKS cluster).

Last Step: Build

Initialize config: terraform init