Terraform support for job templates and output presets of AWS Elemental MediaConvert

Published in

Craftsmen — Software Maestros

5 min readSep 11, 2020

AWS Elemental MediaConvert is a media transcoding service of AWS. For supporting vast VOD service, MediaConvert is very useful and reliable. Terraform is an Infrastructure as Code (IaC) tool which supports multiple Cloud platforms like AWS, Azure, and GCP. Unfortunately due to a lot of configurations of MediaConvert, terraform does not support MediaConvert resource and settings management yet. In this tutorial, we are going to take a look at how can we achieve this.

MediaConvert Job Templates

The job template is a part of MediaConvert. If we create some templates with input(s) and output(s) of transcoding, we can use that for every time to transcoding some files. Let’s say your platform needs 3 types of video outputs with different bitrate and resolution. So we can save that template and reuse for the next transcoding jobs. There is another section called Output presets in MediaConvert console. These preset store output configurations of transcoding jobs. And in the template, we can use one or more output presets.

Setting the Goal

So our plan is to manage MediaConvert templates with terraform as terraform does not support MediaConvert. So we will create templates and output presets at the time of Terraform apply command and destroy with Terraform destroy command.

MediaConvert presets and template

For the MediaConvert preset, we will use one preset for video and one preset for audio for simplicity. And we will use MS Smooth streaming protocol for transcoding.

A1 — Audio preset
P1 — Video preset

For the MediaConvert job template, we are going to use the following configuration. The presets A1 and P1 are mentioned in MS Smooth output settings.

Terraform Configs

For each AWS region, there is a separate MediaConvert endpoint URL. Our first goal is to grab that endpoint URL because to get create preset and template we need MediaConvert endpoint URL. We are going to use AWS CLI for that. Let’s install that from here. Then we will write a simple bash script to grab the MediaConvert endpoint URL. We need to format the output with jq — A JSON processor (will explain why after some moments). Let's install jq from here. Then our script will be like this in mediaconvert_url.sh file

# mediaconvert_url.sh#!/bin/bash
set -e

eval "$(jq -r '@sh "export profile=\(.profile) region=\(.region)"')"
aws mediaconvert describe-endpoints --profile "$profile" --region "$region" | jq -r .Endpoints[0]

Don’t forget to run

chmod a+x mediaconvert_url.sh

Now lets set AWS provider in provider.tf

# provider.tfprovider "aws" {
  region = "eu-west-1"
  profile = local.aws_profile
}

locals {
//  This profile name used in multiple places
//  So keeping it in a local var
  aws_profile = "your-aws-profile"
}

// For getting AWS region dynamically
data "aws_region" "current" {}

Now we have to pass the AWS profile set by us and region to mediaconvert_url.sh to get the MediaConvert endpoint URL. So we are going to use an external data source of Terraform and run the mediaconvert_url.sh. But terraform external data source only accepts JSON formatted output. That’s why we are using jq to format the output of the AWS MediaConvert describe-endpoints command. Let’s add an external terraform source in mediaconvert_url.tf

# mediaconvert_url.tflocals {
//  Keeping the mediaconvert endpoint URL to a local var
  mediaconvert_endpoint = data.external.mediaconvert_endpoint.result.Url
}

// Run external program using terraform's data external
data "external" "mediaconvert_endpoint" {
  program = ["/bin/bash", "-c", "${path.module}/mediaconvert_url.sh"]

  query = {
    profile = local.aws_profile
    region = data.aws_region.current.name
  }
}

// Terraform output - mediaconvert URL
output "mediaconvert_endpoint" {
  value = local.mediaconvert_endpoint
}

Now let’s keep our presets in presets directory and MediaConvert templates in templates directory like this

So we will use AWS CLI to create presets and job templates. But we will add this using a bash script. Also, our same bash script will be used to destroy. So a flag ACTION will be used to differentiate between creating and destroying. The script will look like this in mediaconvert.sh file.

# mediaconvert.sh#!/bin/bash

find presets/* -prune -type f | while IFS= read -r preset; do
  echo "$preset"
  PRESET_NAME=$(jq -r '.Name' "$preset")
  PRESET_SETTINGS=$(jq '.Settings' "$preset")

  if [ "$ACTION" = "destroy" ]; then
      aws mediaconvert delete-preset \
      --name "$PRESET_NAME" \
      --endpoint-url "$MEDIACONVERT_URL" \
      --profile "$AWS_PROFILE" \
      --region "$REGION"
  else
      aws mediaconvert create-preset \
      --name "$PRESET_NAME" \
      --settings "$PRESET_SETTINGS" \
      --endpoint-url "$MEDIACONVERT_URL" \
      --profile "$AWS_PROFILE" \
      --region "$REGION"
  fi
done

find templates/* -prune -type f | while IFS= read -r template; do
  echo "$template"
  TEMPLATE_NAME=$(jq -r '.Name' "$template")
  TEMPLATE_SETTINGS=$(jq '.Settings' "$template")

  if [ "$ACTION" = "destroy" ]; then
      aws mediaconvert delete-job-template \
      --name "$TEMPLATE_NAME" \
      --endpoint-url "$MEDIACONVERT_URL" \
      --profile "$AWS_PROFILE" \
      --region "$REGION"
  else
      aws mediaconvert create-job-template \
      --name "$TEMPLATE_NAME" \
      --settings "$TEMPLATE_SETTINGS" \
      --endpoint-url "$MEDIACONVERT_URL" \
      --profile "$AWS_PROFILE" \
      --region "$REGION"
  fi
done

Also, don’t forget to run

chmod a+x mediaconvert.sh

As we can see that we have to pass environment variables ACTION, MEDIACONVERT_URL, AWS_PROFILE, and REGION. To do this we are going to use a local-exec provisioner of Terraform in a null resource. We are going to add 2 provisioner — one to create presets and template and another for destroy. And we will pass environment vars of mediaconvert.sh from this provisioners. Let’s do it in mediaconvert.tf file.

# mediaconvert.tfresource "null_resource" "mediaconvert_creation" {
  //Provisioner for creation
  provisioner "local-exec" {
    when    = create
    command = "./${path.module}/mediaconvert.sh"

    environment = {
      ACTION           = "create"
      AWS_PROFILE      = local.aws_profile
      REGION           = data.aws_region.current.name
      MEDIACONVERT_URL = local.mediaconvert_endpoint
    }
  }

    // Provisioner for destroying
    provisioner "local-exec" {
      when = destroy
      command = "./${path.module}/mediaconvert.sh"

      environment = {
        ACTION = "destroy"
        AWS_PROFILE = local.aws_profile
        REGION = data.aws_region.current.name
        MEDIACONVERT_URL = local.mediaconvert_endpoint
      }
    }
}

Finally, our project will look like this

tree.
├── mediaconvert.sh
├── mediaconvert.tf
├── mediaconvert_url.sh
├── mediaconvert_url.tf
├── presets
│   ├── aws-mediaconvert-preset-a-1.json
│   └── aws-mediaconvert-preset-p-1.json
├── provider.tf
├── templates
   └── aws-mediaconvert-template.json

Time to test 😹

Now lets test with

terraform init   # for initialization terraform
terraform plan   # planning what to create and/or destroy
terraform apply -auto-approve     # create all
terraform destroy -auto-approve   # To destroy all

After terraform apply we can see our output presets and job template in the AWS console

Finally, you can run a MediaConvert job with the template to test it. Before that don’t forget to create an IAM role for MediaConvert with necessary S3 bucket permission.