Backing up Route53 with Ruby and Lambda
Low cost, low maintenance backup of AWS Route53 using Ruby in AWS Lambda
For some time, I have had an item on my backlog. The high-priority Cloud SysAdmin items about keeping business data safe and securing systems all ticked off, same with infrastructure-as-code and continuous deployment.
However, one issue remained: An easy way of backing up the DNS records in AWS Route53.
Why backup Route53 at all?
AWS is responsible for- and does a great job keeping Route53 running. You will also have the power of AWS IAM at your fingertips to make sure that only the right people mess with your DNS records. That is what AWS provides.
AWS does not provide a way of getting back the DNS records (or even whole hosted zones) that you inadvertently delete or change. That is the responsibility of the customer.
Of course, you might not think that DNS data are as important as core business data — but losing your DNS zone will pretty quickly knock your business off the internet and make e-mails from both upset customers and angry bosses bounce.
Being able to restore a Route53 hosted zone also comes in handy if you ever have to move a hosted zone to another AWS account.
So you should make sure that your Route 53 hosted zones are backed up. Period.
Choosing a tool
Googling for “route53 backup scripts” should provide quite a lot of hits. Some of the most popular ones seem to be route53-transfer , cli53 and Roadworker. All of them are command line tools capable of generating a backup file of your Route53 hosted zone.
Choosing the right tool ultimately comes down to features, trust, and preferences. It goes without saying that you should never do backups without testing that you can restore them as well.
cli53: 800+ ⭐️ on GitHub, outputs BIND format files; written in Go
roadworker: 200+ ⭐️ on GitHub, outputs a DSL, written in Ruby
route53-transfer: 40+ ⭐️ on GitHub, outputs .csv; written in Python
cli53 can do a lot more than just backups, and the standard BIND format could come in handy for cross-cloud DNS disaster recovery. For Python programmers, route53-transfer might be the reassuring choice if the need to peek under the hood arises. However, in the end, it is the output of Roadworker that does it for me: A DSL for defining the state of Route53 — what is not to like about that?
S3 is the cheap, durable choice for storing files on AWS. S3 has versioning, access control, and lifecycle management, making it the oblivious choice. Encryption at rest for regulatory purposes? You can get that as well.
Now we just need to write a tiny bit of glue-code to run the script and then upload the files to S3. Oh, and then we need to decide on where to run the backup script.
You might already have an existing (tools/management) server where you could run it? It is no big deal, right? Just one more Ruby runtime to setup and manage, just one more IAM policy to attach to the server, just another cron job and just another set of log files that needs handling. Gaaaahhh… Death by a thousand cut!
We could off course set up a cheap t2 server to run nothing but the backup. However, by now everyone has probably learned that every server — no matter how tiny — needs to be monitored, patched and managed. Add to it that the server will be overwhelmingly idle for all but a few seconds every day. Certainly feels like a waste of both time and money.
So why not just let Amazon care about the VM? Let’s go serverless, let’s use AWS Lambda!
AWS Lambda and Ruby?
If you have made it this far, then you probably already know about AWS Lambda: Bundle up code in a zip file, hand it over to AWS with instructions about how- or when to run it. Watch the results and check the logs from the AWS Lambda console. Pay per fraction of a second instead of per hour.
Choosing Roadworker poses a challenge as AWS Lambda does not directly support Ruby. A popular choice for installing Ruby is RVM. The ruby .rvm installation directory for Ruby, Roadworker, and its dependencies clocks in at an astonishing 515 MB of disk space on an Amazon Linux — that, admittedly, includes source code.
Without source code, a zipped-up .rvm directory still clocks in at 90+ MB — significantly above AWS Lambda’s limit of a 50MB zip file.
A gentleman by the name of Attila Domokos has written a few blog posts about how to embed Ruby in Lambda, mentioning Phusions Traveling Ruby, but Roadworker requires a couple of native Gems that are not amongst the downloadable extra gems that Traveling Ruby provides. Traveling Ruby does, however, include helpful hints on how to minimize the size of Ruby installations.
If you are using any native binaries in your code, make sure they are compiled in this environment
AWS states that native binaries should be compiled in the Lambda environment, which Amazon provides as an AMI. The newly launched service AWS CodeBuild provides a set of Docker images that seems sufficiently compatible with the Lambda environment.
So let’s try to create a CodeBuild project that installs pre-requisite software, install Roadworker in an rvm environment, shave down the size using the hints from Traveling Ruby and then upload the resulting binary libraries to S3 as a zip file.
With AWS CodeBuild being a managed service, it seems like a better choice — compared to booting up a server from the Lambda environment AMI and building the Lambda code with an orchestration tool like Ansible.
Three years ago, Ansible was a hip, new choice for creating cloud infrastructure, and managing a growing sprawl of virtual machines. Now, Ansible will be a boringly safe choice, even though“boring” certainly has merit.
Also, to top things off, a Terraform script to install- and deploy the Lambda function. The script creates the backup bucket and CodeDeploy project, and then (in a second run) creates the Lambda function and schedules it for execution.
Installing the Lambda function
Download and copy terraform to a bin directory in your path. The version should be 0.7.13 or higher — as we are going to need support for Lambda Environment Variables. Also make sure that you have an updated version of the AWS CLI (version 1.11.28 or higher - more recent than re:Invent 2016 as we need support for CodeBuild).
Building the Lambda code
First, we need to create the Lambda zip file containing Roadworker installed in a Ruby environment by making a CodeBuild project and running a build.
Clone the GitHub repository dashsoftaps/lambda-roadworker
CodeBuild requires a few resources: An IAM Role that allows CodeBuild to output CloudWatch Logs and gives permission to write to an S3 bucket. Also: An S3 bucket for the Lambda zip file, which I will reuse for the backups.
I have turned on versioning and also lifecycle management on the bucket to purge old backups after a year and purge non-current versions after a month. It is not exactly your Grandfather-Father-Son backup tape rotation scheme, but it is close enough.
The bucket name is specified in the
tf/terraform.tfvars file. It should be changed from “hennings-roadworker-backup-bucket” to something that is appropriately and unique for your account.
Now create the necessary infrastructure with Terraform. Usually, I have already configured a profile with the AWS CLI and set it up in an
AWS_PROFILE environment variable, but there is now a wide range of options for safely managing and providing AWS credentials (AWS Vault, awsudo, instance profiles).
Change directory to tf to create the bucket and the policies. Cautious SysAdmins should probably run “terraform plan” to review which resource that Terraform is about to create before running the “terraform apply” command:
$ cd tf
$ terraform apply
Terraform will run for a while and create 10 resources. Terraform does not yet support creating CodeBuild projects natively (there is a work-in-progress pull request, so support should not be too far away). Until it is supported, the AWS CLI will have to be used to create the project and start the build:
$ aws codebuild create-project --cli-input-json file://codebuild-project.json
$ aws codebuild start-build --project-name lambda-roadworker
The software that is being build by the CodeBuild project is specified in the
buildspec.yml file in the root of the github repository:
Follow the build phase in the AWS Console. Here is how it might look while a build is in progress:
If everything goes well, then the status of the build should end as Succeeded and a zipfile named LambdaRoadworker.zip should exist in the bucket.
Deploying the Lambda function
Running the RoadWorker Lambda function also requires a few resources: Mainly a Lambda function, a scheduled cloudwatch event rule and a few resources to connect the function and the event rule and allow the event rule to call the Lambda function.
For the event rule, I’ve chosen a schedule that triggers the backup every 6 hours.
Compared to the previous run, an additional flag needs to be passed to deploy the Lambda function. Run a Terraform plan to review the resources that are to be created:
# terraform plan
Create the resources with:
# terraform apply
Be aware about the
terraform.tfstate file that has now appeared in the
tf directory. If you ever want to change or modify the infrastructure you just created, then you need to save- and manage this file.
Losing a Terraform state file is usually a pretty bad thing, but in this case, only a handful of resources is created. If you should lose the state file, then just delete the resources in the console (the Lambda function RoadworkerRoute53Backup, the IAM Roles prefixed with LambdaRoadworker and possibly the S3 bucket and CodeBuild project) and re-create them with Terraform.
Testing the Lambda function
A simple procedure to test the Lambda function is to go to the Lambda page in the AWS Console, find and select the function RoadworkerRoute53Backup, select Test function on the Actions drop-down. Choose Scheduled Event in the Sample event list and then Save and test — before verifying that the Execution result section shows a success.
You could also just use the AWS CLI to invoke the function:
$ aws lambda invoke -function-name RoadworkerRoute53Backup /dev/null
If the function runs successfully, then you should find a date-stamped backup file in the S3 bucket.
You can do backup validation and restores by installing Roadworker locally and download the backup file. See the Roadworker github page for additional documentation on which arguments to use to test- and restore backup files.
Ruby in AWS Lambda! Hooray!
Congratulations — you’re now running a Ruby command line tool in Lambda — and not just any pure-Ruby tool: It is actually a tool with dependencies to native gems in Lambda.
The Ruby environment is rvm, so there is no reason why other Ruby tools could not be deployed to Lambda functions using the same method.
Be aware that the resulting zip file is quite large, and that startup time might be quite long.
Cost of backing up Route53
Roadworker runs for just under 12 seconds when it backups my tiny Route53 zone. It runs four times per day which adds up to:
4 × 12 seconds × 31 days in a month × 0.00000208 USD/second
Compute cost equals 0.3 cents per month — but it gets slightly better: Lambda has 3.200.000 seconds of free tier monthly, and this function would use up only a tiny fraction of that. So if you are not currently using Lambda, then it will be free.
S3 storage cost does not amount to much either: The backup uses 4×30 + (336−30) = 456 files. Minimum billable file size on S3 is 128 kb, which adds up to 0.058368 GB × 0.023 USD/GB — or around 0.1 cents per month.
The total cost will end up being around half a cent per month for backing up my Route53 zone.
With backups of Route53 done and a lesson learned about running Ruby in Lambda, I now have one less item on my backlog to drag into 2017! Just in time for the holiday season.
Which long-overdue items do you still have on your list for 2016? Feel free to share below or 💙!