Build a custom CLI to AWS Cloudformation with Boto3

I will be honest. This isn’t my best guide. Its also not my worst. Think that lands it squarely in the “medium of medium” camp.

What it is: A simple implementation of a Boto3 stack create and delete call using that includes all the code on github. So maybe that is helpful. I like helping. That is what https://dronze.com is all about. Helping.

Cloudformation is a way to provision or deprovision groups of AWS assets.

So this guide is how we made our bare bones implementation.

Creating Stacks With Boto3

Creating stacks with boto is easy. I would say the hardest thing is deciding how to transform a CLI based interface into something that an API can call. The goal is to make a set of commands that are pretty simple and will translate to the boto3 create_stack call spec:

response = client.create_stack(
StackName='string',
TemplateBody='string',
TemplateURL='string',
Parameters=[
{
'ParameterKey': 'string',
'ParameterValue': 'string',
'UsePreviousValue': True|False
},
],
DisableRollback=True|False,
TimeoutInMinutes=123,
NotificationARNs=[
'string',
],
Capabilities=[
'CAPABILITY_IAM'|'CAPABILITY_NAMED_IAM',
],
ResourceTypes=[
'string',
],
RoleARN='string',
OnFailure='DO_NOTHING'|'ROLLBACK'|'DELETE',
StackPolicyBody='string',
StackPolicyURL='string',
Tags=[
{
'Key': 'string',
'Value': 'string'
},
]
)

Wow. That’s a lot of stuff. I don’t think we need all that for a “simple” implementation. So this was the bare minimum we chose:

$ python create-stack.py -h
usage: create-stack.py [-h] --config CONFIG --name NAME
--templateurl TEMPLATEURL --params PARAMS --topicarn TOPICARN
[--tags TAGS]
optional arguments:
-h, --help show this help message and exit
--config CONFIG the config file used for the application.
--name NAME the name of the stack to create.
--templateurl TEMPLATEURL
the url where the stack template can be fetched.
--params PARAMS the key value pairs for the parameters of the stack.
--topicarn TOPICARN the SNS topic arn for notifications to be sent to.
--tags TAGS the tags to attach to the stack.

And since the tags and params are pretty much the same:

{
'ParameterKey': 'string',
'ParameterValue': 'string',
'UsePreviousValue': True|False
}

for the parameters and for the tags used for metadata related to the stack:

{
'Key': 'string',
'Value': 'string'
}

Why don’t we normalize the implementation that allows name value pairs to be parsed using a query string since they are so close:

# Parameters=[
# {
# 'ParameterKey': 'string',
# 'ParameterValue': 'string',
# 'UsePreviousValue': False
# },
# ],
def make_kv_from_args(params_as_querystring, name_prefix="", use_previous=None):
nvs = parse_qs(params_as_querystring)
#{'i': ['main'], 'enc': [' Hello '], 'mode': ['front'], 'sid': ['12ab']}
kv_pairs = []
for key in nvs:
# print "key: %s , value: %s" % (key, nvs[key])
kv = {
"{0}Key".format(name_prefix):key,
"{0}Value".format(name_prefix):nvs[key][0],
}
if use_previous != None:
kv['UsePreviousValue'] = use_previous
kv_pairs.append(kv)
return kv_pairs

Believe it or not this is the hardest thing about the implementation because using boto3 is so easy.

We have a general way to create the boto3 client so we can choose to configure it, or use the fallback credentials approach that boto3 attempts natively:

def make_cloudformation_client(config=None):
"""
this method will attempt to make a boto3 client
it manages the choice for a custom config
"""
#load the app config
client = None
if config != None:
logging.info("using custom config.")
config = load_config(args.config)
client = boto3.client('cloudformation',
config["AWS_REGION_NAME"],
aws_access_key_id=config["AWS_ACCESS_KEY_ID"],
aws_secret_access_key=config["AWS_SECRET_ACCESS_KEY"])
else:
# we dont have a configuration, lets use the
# standard configuration fallback
logging.info("using default config.")
client = boto3.client('cloudformation')
if not client:
raise ValueError('Not able to initialize boto3 client with configuration.')
else:
return client

Then all you need to do is make the boto3 call:

# setup the model
template_object = get_json(args.templateurl)
params = make_kv_from_args(args.params, "Parameter", False)
tags = make_kv_from_args(args.tags)
response = client.create_stack(
StackName=args.name,
TemplateBody=json.dumps(template_object),
Parameters=params,
DisableRollback=False,
TimeoutInMinutes=2,
NotificationARNs=[args.topicarn],
Tags=tags
)

That is it. You will want to do some response handling (see code) but you are just about ready to try to create a stack. This can be called via our CLI app, create-stack:

python create-stack.py --name newstack01 --templateurl https://raw.githubusercontent.com/dronzebot/dronze-qlearn/master/cicd/cloudformation/ec2_instance_sg.json?token=AAY5LkQG0d0G4Uql5Y8T-74L2BuGjKfNks5Y7kTAwA%3D%3D --params "KeyName=dronze-oregon-dev&InstanceType=t2.small"   --tags "name=newstack01&roo=mar" --topicarn arn:aws:sns:us-west-2:705212546939:dronze-qlearn-cf
INFO       2017-04-05 08:17:07,009 make_cloudformation_client           50  : using default config.
INFO 2017-04-05 08:17:07,041 load 628 : Found credentials in shared credentials file: ~/.aws/credentials
INFO 2017-04-05 08:17:07,675 _new_conn 735 : Starting new HTTPS connection (1): cloudformation.us-west-2.amazonaws.com
INFO 2017-04-05 08:17:08,345 main 111 : succeed. response:{"StackId": "arn:aws:cloudformation:us-west-2:605211536939:stack/newstack01/fd7ccbf0-1a11-11e7-a878-503ac9316861", "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "RequestId": "eeef8f29-1a12-11e7-8b60-5b681d5e1677", "HTTPHeaders": {"x-amzn-requestid": "eeef8f29-1a12-11e7-8b60-5b681d5e1677", "date": "Wed, 05 Apr 2017 15:17:08 GMT", "content-length": "380", "content-type": "text/xml"}}}

When you watch what happens in EC2 you see something kindof like this:

These events are sent to the sns queue.

And there you have it. A simple custom client for cloudformation and boto3.

Deleting Stacks with Boto3

Deleting stacks is even easier. This implementation includes a simple parser to allow retained resources to be excluded as comma separated string.

The help:


$ python delete-stack.py -h
usage: delete-stack.py [-h] — name NAME [ — retain RETAIN] [ — log LOG] [ — config CONFIG]
arguments:
-h, — help show this help message and exit
— name NAME the name of the stack to create.
— retain RETAIN the names (comma separated) of the resources to retain.
— log LOG which log level. DEBUG, INFO, WARNING, CRITICAL
— config CONFIG the config file used for the application

There really isn’t much to the the implementation because we are using the same function to make the client:

#load the client using app config or default
client = make_cloudformation_client(args.config)
retained_resources = []
if args.retain and len(args.retain)>0:
retained_respources = args.retain.split(",")
response = client.delete_stack(
StackName=args.name,
RetainResources=retained_resources
)

Running delete stack is super easy:

$ python delete-stack.py --name newstack01
INFO 2017–04–04 18:03:17,576 make_cloudformation_client 50 : using default config.
INFO 2017–04–04 18:03:17,677 load 628 : Found credentials in shared credentials file: ~/.aws/credentials
INFO 2017–04–04 18:03:18,187 _new_conn 735 : Starting new HTTPS connection (1): cloudformation.us-west-2.amazonaws.com
CRITICAL 2017–04–04 18:03:18,961 main 58 : succeed. response: {“ResponseMetadata”: {“RetryAttempts”: 0, “HTTPStatusCode”: 200, “RequestId”: “a7e04f04–199b-11e7–8a78–11614ee92102”, “HTTPHeaders”: {“x-amzn-requestid”: “a7e04f04–199b-11e7–8a78–11614ee92102”, “date”: “Wed, 05 Apr 2017 01:03:18 GMT”, “content-length”: “212”, “content-type”: “text/xml”}}}

Using a Configuration File

I don’t pass configurations on the CLI, to me the args on the CLI are about runtime not config. We have a config file that has the static configs in it:

AWS_ACCESS_KEY_ID=[my_access_key]
AWS_SECRET_ACCESS_KEY=[my_secret_access]
AWS_REGION_NAME="us-west-2"
LOG_LEVEL="INFO"

Boto3 is capable of auto configuration, and it will behave like aws CLI and attempt to find configs from ~/.aws/credentials but if you want explicit configs that is available using the config option in the CLI. If you do this the debug level will default to INFO.

Conclusion

Making a cloudformation CLI is easy, three things that helped me to remember:

  • The hardest thing is managing and transforming arguments
  • Handling exceptions and messages is the common grunt work of CLI
  • Configuration is important, give yourself options.
One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.