Game Tech Tutorial
Published in

Game Tech Tutorial

How to create a Serverless Cron Job in AWS Batch?

AWS Batch is a managed service to allow you to run batch jobs on AWS like big data analyzing or updating data in MYSQL.

You don’t need to additionally provision or maintain an EC2 server to run cron jobs but simply set up Jobs, Compute Environment (EC2 or FARGAT), and Job Queue on AWS Batch.

You can combine with CloudWatch to periodically trigger Jobs of AWS Batch, changing the number of retry times, even sending an alarm via SNS while detecting any unexpected failures.

In this tutorial, we will explore

  • 3 Ways to Create Cron Jobs on AWS
  • Advantages of using AWS Batch
  • How to design Cron Job on AWS Batch and CloudWatch
  • How to setup Cron Job on AWS Batch

Before we dive deeper, let’s explore how to create cron jobs on AWS.

3 Ways to Create Cron Jobs on AWS

There are 3 ways to set up cron jobs on AWS.

1.EC2:

This is a common and traditional approach to creating jobs on Linux. Though it’s very easy to set up crontab on Linux, you need to spend more time maintaining the server. Also, it’s hard to achieve high availability.

2.CloudWatch and Lambda / ECS / EKS / Beanstalk:

2.1 CloudWatch and ECS/EKS/Beanstalk

  • CloudWatch aims to trigger lambda function or invoke API endpoints on ECS, EKS, or Beanstalk periodically.
  • This can easily achieve high availability and scalability. But ECS, EKS, or Beanstalk needs to share computing resources to run cron jobs. This might cause some bottlenecks in API’s performance.
  • To ensure high performance, you need additional work to separate cron job endpoints running in different EC2 nodes.

2.2 CloudWatch and Lambda

  • Lambda is a serverless service and can achieve high availability. However, Lambda has limited Memory, CPU resources, and the maximum execution time is only 15 mins.

3. CloudWatch and AWS Batch:

  • AWS Batch is a serverless service and can achieve high availability and scalability. You can increase Memory or CPU on daemon while cron jobs become slow. No restriction on the running time.
  • Your cron jobs are independent of EC2, ECS, or EKS. This can eliminate potential bottlenecks on your architecture while running big data analysis on the same computing resources.

The three approaches mentioned above have their tradeoffs. Choose one solution to fit your development stack and save your time.

Advantages of using AWS Batch

  • Cost:

AWS Batch is free and only charges the underlying resources you consume while running batch jobs. When your cron job is finished, the resources will be cleaned up on AWS. This can greatly save your costs.

  • Scalability:

You can set up EC2 Spot or FARGAT Spot to automatically scale on the daemon. If you need more resources to analyze big data, you can tweak CPU and Memory accordingly. To automatically scale out the computing nodes can reply on Fargate.

  • Availability:

AWS Batch is a Serverless service.

  • Maintenance:

No need to provision or maintain EC2 instances and worry about the availability of your server.

  • Log:

By default, you can easily check logs on CloudWatch to monitor your task is successful or failed.

Given the fact that AWS Batch is a cost-effective solution, let’s explore how to design and set up this server-less architecture for cron jobs.

How to Design Cron Job on AWS Batch and CloudWatch?

The flow is shown below.

CloudWatch --> AWS Batch --> Job Queue --> Run Go Application in Docker --> Download Batch Script from S3 --> Run Batch Script
  • CloudWatch periodically notifies AWS Batch Service to wake up cron jobs on Job Queue.
  • A cron job will be running in docker on AWS FARGATE or EC2 in the auto-scaling group.
  • Cron job starts to download all the data from S3 including shell scripts and data.
  • Then Cron job starts executing the bash script

You can build an application in Golang or other languages with Docker to run specific cron jobs.

Your docker will invoke a script file to download all the data from S3 where you can place any shell scripts or data when needed. This is very flexible to analyze big data on the daemon.

How to Setup Cron Job on AWS Batch?

Step1: Setup and Upload Scripts to S3

Before we get started, we need to prepare for the following.

  • Create an image repo for your application on ECR
  • Create an S3 Bucket
  • Create an IAM to manage S3 Bucket
  • Create fetch_and_run.sh
  • This script can download cronjob.sh from S3.
  • vim fetch_and_run.sh
#!/bin/bash
PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
BASENAME="${0##*/}"
BATCH_FILE_TYPE="script"
# Change cronjob sell script to be executed on S3
BATCH_FILE_S3_URL="s3://<Your S3 Bucket>/cronjob.sh"
# Standard function to print an error and exit with a failing return code
error_exit () {
echo "${BASENAME} - ${1}" >&2
exit 1
}
# Check what environment variables are set
if [ -z "${BATCH_FILE_TYPE}" ]; then
usage "BATCH_FILE_TYPE not set, unable to determine type (zip/script) of URL ${BATCH_FILE_S3_URL}"
fi
if [ -z "${BATCH_FILE_S3_URL}" ]; then
usage "BATCH_FILE_S3_URL not set. No object to download."
fi
scheme="$(echo "${BATCH_FILE_S3_URL}" | cut -d: -f1)"
if [ "${scheme}" != "s3" ]; then
usage "BATCH_FILE_S3_URL must be for an S3 object; expecting URL starting with s3://"
fi
# Check that necessary programs are available
which aws >/dev/null 2>&1 || error_exit "Unable to find AWS CLI executable."
which unzip >/dev/null 2>&1 || error_exit "Unable to find unzip executable."
# Create a temporary directory to hold the downloaded contents, and make sure
# it's removed later, unless the user set KEEP_BATCH_FILE_CONTENTS.
cleanup () {
if [ -z "${KEEP_BATCH_FILE_CONTENTS}" ] \
&& [ -n "${TMPDIR}" ] \
&& [ "${TMPDIR}" != "/" ]; then
rm -r "${TMPDIR}"
fi
}
trap 'cleanup' EXIT HUP INT QUIT TERM
# mktemp arguments are not very portable. We make a temporary directory with
# portable arguments, then use a consistent filename within.
TMPDIR="$(mktemp -d -t tmp.XXXXXXXXX)" || error_exit "Failed to create temp directory."
TMPFILE="${TMPDIR}/batch-file-temp"
install -m 0600 /dev/null "${TMPFILE}" || error_exit "Failed to create temp file."
# Fetch and run a script
fetch_and_run_script () {
# Create a temporary file and download the script
aws s3 cp "${BATCH_FILE_S3_URL}" - > "${TMPFILE}" || error_exit "Failed to download S3 script."
# Make the temporary file executable and run it with any given arguments
local script="./${1}"; shift
chmod u+x "${TMPFILE}" || error_exit "Failed to chmod script."
exec ${TMPFILE} "${@}" || error_exit "Failed to execute script."
}
# Main - dispatch user request to appropriate function
case ${BATCH_FILE_TYPE} in script)
fetch_and_run_script "${@}"
;;
*)
usage "Unsupported value for BATCH_FILE_TYPE. Expected (script)."
;;
esac
  • create cronjob.sh
  • This script can download all the files from S3 to docker and execute the cron job application.
  • vim cronjob.sh
#!/bin/bashS3_CRONJOB_BUCKET="s3://<Your Bucket>"echo "Args: $@"
env
echo "-- CGS Batch JOB! ---"
echo "jobId: $AWS_BATCH_JOB_ID"
# Copy all the files and data to docker application from S3
aws s3 cp "${S3_CRONJOB_BUCKET}/" . --recursive
# Execute your cron application in Docker
/app/myapp
if [ $? -eq 0 ]
then
echo "Successfully execute cronojb"
else
echo "Could not execute cronojb" >&2
echo "Failed to cronojb on $CURRENT_DATE"
# Optional: send an alarm to Slack to detect errors.
exit 1
fi
echo "Succeed to cronojb on $CURRENT_DATE"
  • Then upload cronjob.sh to S3://<your bucket>/cronjob.sh

Note:

  • Docker will invoke fetch_and_run.sh to download all the data from S3 Bucket and execute cronjob.sh.

Step2: Build Cron Job Application in Docker

Next, let’s start to build the cron job application in Golang with Docker.

# Goto your batch_job folder
# Place fetch_and_run.sh in batch_job
cd batch_job
  • Create the build script:
  • vim buildImageAndUpload.sh
#!/usr/bin/env bashif [ -d "tmp" ]
then
echo "tmp exist"
rm -rf ./tmp
else
echo "tmp not exist"
fi
# Create tmp folder
mkdir tmp
cp go.mod tmp/
cp go.sum tmp/
cp *.go tmp/
cp fetch_and_run.sh tmp/
cd tmp# Modify IAM to download files from S3
AWS_ACCESS_KEY_ID="<YOUR KEY ID>"
AWS_SECRET_ACCESS_KEY="<YOUR SECRET KEY>"
AWS_DEFAULT_REGION="ap-northeast-1"
# Setup the IMAGE URL for your cronjob on ECR
REMOTE_IMAGE="<YOUR IMAGE ON ECR>"
docker build --no-cache -t $REMOTE_IMAGE -f- . <<EOFFROM golang:1.14.15-alpine AS builderRUN apk add --no-cache --virtual .build-deps git musl-dev go && \
apk --update add ca-certificates && \
apk add --no-cache tzdata
# Change TimeZone
# ENV TZ Asia/Singapore
RUN mkdir /myapp
WORKDIR /myapp
ADD fetch_and_run.sh /myapp/fetch_and_run.shCOPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
# Build the binary.
# If you have libs dependent on C lib, set CGO_ENABLE=1 to compile and link C lib.
# RUN CGO_ENABLE=1 go build -a -ldflags "-linkmode external -extldflags '-static' -s -w" -o /go/bin/myapp myapp.go
# Otherwise, run
RUN CGO_ENABLE=0 go build -o /go/bin/myapp myapp.go
# Remove cache files and fodlers
RUN \
rm -rf /go/pkg/mod && \
rm -rf /usr/local && \
rm -rf /root/.cache && \
rm -rf /var/cache/apk/* \
apk del .build-deps
FROM alpine:3.14RUN mkdir /app
WORKDIR /app
# Change timezone
# ENV TZ=Asia/Singapore
ENV AWS_ACCESS_KEY_ID $AWS_ACCESS_KEY_ID
ENV AWS_SECRET_ACCESS_KEY $AWS_SECRET_ACCESS_KEY
ENV AWS_DEFAULT_REGION $AWS_DEFAULT_REGION
# Install bash
RUN apk add --no-cache bash
# Install aws-cli
RUN apk add --no-cache \
python3 \
py3-pip \
&& pip3 install --upgrade pip \
&& pip3 install \
awscli \
&& rm -rf /var/cache/apk/*
RUN aws --versionCOPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /go/bin/myapp .
COPY --from=builder /myapp/fetch_and_run.sh .
RUN chmod -R 765 /app/ENTRYPOINT ["/app/fetch_and_run.sh"]EOF# login to AWS and change Region
docker login -u AWS -p $(aws ecr get-login-password --region <Your Region>) $REMOTE_IMAGE
sleep 3# Upload to AWS
docker push $REMOTE_IMAGE

Build the Cron Job and upload to ECR:

  • sh buildImageAndUpload.sh

At this moment, you can run and test docker on local to validate cron jobs.

docker run -it <Docker Image url on ECR>

If it’s fine, let’s set it up in AWS Batch.

Step3: Create Job

Log in to AWS Console –> AWS Batch –> Job Definitions –> Click Create

Image: <Your image URL on ECR>

Enable Assign Public IP

Step4: Create Job Queue

Create a Job Queue

Step5: Create Compute Environment

Select managed and enable compute environment.

Select Fargate Spot and specify CPU and Memory.

Select network.

Create a job queue and set up compute environment.

Step6: Submit Jobs

Submit a job in AWS Batch to validate your cron job. Also, you can check logs on CloudWatch.

Conclusion

In this tutorial, you have learned different approaches and their tradeoffs to set up cron jobs on AWS — EC2, CloudWatch and ECS/EKS/Lambda/Beanstalk, CloudWatch, and AWS Batch.

It’s recommended not to use EC2 to run cron jobs because it may have a single point of failure problem and it’s hard to smoothly scale and maintain. In another approach like Lambda, you need to pay attention to the limits in execution time, CPU, and memory resources.

ECS/EKS/Beanstalk needs to share the resources with a cronjob. You need additional work to set up them properly to run cron jobs in different subnets or different EC2 nodes to ensure the same performance.

Therefore, AWS Batch might be the best solution to run cron jobs in terms of cost, performance, and availability. Lastly, we also provide a sample template to build a cron job in Golang which can improve your work productivity.

--

--

--

High-quality technical tutorials: AWS, GCP, Unity, SDK, Security, and more.

Recommended from Medium

Understanding Generics and Variance in Kotlin

CSS: Day 19

Illustration with scale text

Making a decision: Should I Automate? Or maybe not?

Innovative PointPay Token Sale Platform

How to embed a PWA inside an Android application

How to Open Sublime Text from Terminal in MacOS

What is Hacktoberfest and How can a beginner contribute?

Computer Science Courses @Kheyeh

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Eric Wei

Eric Wei

Senior Full Stack Engineer & Solution architecture | AWS, GCP | Cloud, Unity Game Development, SDK, DevOps, and more.

More from Medium

Relation Databases in AWS Cloud

Deploying to AWS Using Bitbucket Pipelines

Setup CI/CD for your AWS Lambda with Serverless framework and GitHub Actions

Working with AWS Lambda | Part-1