Serverless from the trenches 1: Protect your stack from deletion | Lumigo

It’s time to talk about the everyday challenges of serverless.

Whenever I scroll through the latest blog posts on serverless it feels like there are only two types of problems in the field: how do I get started and how do I architect my solution.

But what about all the day-to-day problems that developers and DevOps encounter when dealing with serverless? From simple deployment issues like protecting your stack from deletion to stress-testing your solution using serverless techniques.

That’s where Serverless from the Trenches comes in. Every other week these bite-sized blog posts will highlight a technique or tool to solve a real world problem that those of us on the frontline of serverless face on a daily basis.

Oops I deleted it again

Protecting your CloudFormation stacks from deletion is critical for production. So, in this post we’ll share an automated way to protect your CloudFormation stack using a simple bash script.

Each of our developers has their own AWS account (under one organization), because we believe that the best way to test your serverless code is in the cloud, not locally. Once in a while, the developer’s environment requires grooming, so we created a script that goes over all the stacks and deletes them, keeping the environment fresh, and avoiding the annoying sls remove.

The biggest implication for #serverless and CI/CD is that in the future, development is primarily going to be done on the cloud, and not on “my machine”

Why?

In addition to our personal AWS environments, we have a shared environment that is used as part of our CI/CD. Before being merged into master, each branch is deployed to a shared AWS environment on which integration tests are running. We run our grooming script on that environment as well, usually once per week, as we like to keep old environments in case debugging is required. On one of those sleepless nights while working on our deployment we accidentally ran the grooming script on our production.

Fortunately, our IAM role forbids stack deletion, but it made us think that we’d better add another layer of protection.

One of the ways that are promoted by AWS is to use termination protection, just click on your stack and choose Edit termination protection:

But how do you automate it?

Automate the unautomatable

We wanted to automate it during the deployment process, i.e. on sls deploy, but since we couldn’t find a Serverless plugin to do the trick for us, we improvised using bash and AWS CLI.

AWS CLI supports the update-termination-protection command which enables us to quickly turn on the protection for any installed stack. All we need is to grab the stack name.

The following bash script does the trick:

1. sls deploy -region ${1} -stage ${2}
2. stack_name=$(sls info -stage ${2} -region ${1}|grep stack:|awk ‘{print $2}’)
3. echo ${stack_name}
4. aws cloudformation update-termination-protection -region ${1} -enable-termination-protection -stack-name ${stack_name}

So, how do you make sure nothing is being deleted by accident? Do you use bash scripts or other tools? Share your techniques for making your environment safer.

Add this script to your deployment script and worry no more about accidental deletions.

Monitor & debug your serverless application effortlessly! Get alerted as soon as an issue occurs and instantly drill down to see a virtual stack trace & correlated logs.

Set up your free Lumigo account today & start fixing serverless issues in a fraction of the time! Find out more

Originally published at https://lumigo.io on April 25, 2019.