A sensible DevOps Org level multi account AWS setup with Terraform

Disclaimer: This is something that I think makes a lot of sense, but may not for your use case. I’m discussing how to create a new org (Organisation) in AWS with permissions and users. I also expect that you have some familiarity with AWS, maybe you’ve launched an EC2 instance, or similar.

I wanted to document my experience setting up a new environment with Terraform as my frustration with getting straight answers online and tearing through documentation and blogs is very tiring. There’s so much incomplete blog knowledge that I struggled to put all the pieces together.

It’s June of the year 2020 at the time of writing and Terraform is version 0.12. AWS (Amazon Web Services) has some relatively new offerings such as Organisations, and a lot of old features. It still feels to me as less work to setup with AWS than Kubernetes (k8s), as it’s not so different from an ECS (Elastic Container Storage) cluster, plus it has many built in features.

I have a repo in GitHub here. I recommend checking it out and following along through the repo.

Principle

We want to create a “bastion” or jump account, from which we use to access other accounts. This means we can completely isolate our environments. I took inspiration from this blog and this blog and this blog.

Prior Knowledge

Why Terraform?

Declarative code is a common pattern these days, knowing what’s in your infrastructure at any time gives you great confidence in your setup and helps when you’re doing lots of iterative changes as you don’t need to worry about migrations, teardowns or replicating steps, such as for each environment.

What’s AWS Organisations?

If you’ve ever made a site that needs to cater to businesses, you’ll usually end up creating some kind of organisations model, as usually you have many users, who access many accounts, but you need to bill them once and they need access to the same resources. AWS’s implementation is as you would expect in this regard.

AWS Vault

AWS Vault is how we want to login, every time you want to access the AWS console, you just use the aws-vault login command. The reasoning is you never want to deal with passwords, even with a password manager, using a PGP key is much easier.

Getting Started

Download tools

Download aws, terraform, keybase and aws-vault using your preferred method, for me on macOS its:

brew install awscli terraform
brew cask install aws-vault, keybase

Create an account

If you haven’t created an account on AWS yet, you should do that (enable MFA, store passwords in a password manager, etc), and create an IAM user (I called it “terraform”) with programmatic access and setup aws cli with your new secret keys.

Checkout my repo of code

I’ve written all the code I used for setting up an org in this repo, feel free to clone and follow along there.

NOTE: This code will not work immediately for you, you will need to accept AWS permissions and comment out sections, and fill in names and credentials before it fully works for you.

Code Explain

Track your own account

Terraform reads all the files in your current working directory, so don’t worry too much about naming.

In org/main.tf

The provider is not strictly necessary and I’ll omit it later until it’s actually necessary, but it’s here to be explicit.

Here we’ve defined our new organisation and the account we just created (make sure to put your own account email and name in here). Initialise it with the following command.

# Do this often, a part of terraform life
terraform init

Because you didn’t create your account with Terraform, you need to import it, so run the following with your account id:

terraform import aws_organizations_account.team_name 111111111111

Commit the changes with terraform apply

After a short moment, you should have some terraform.tfstate in your local directory. For storing state in s3 so that you always have the latest changes (which you should do!), see this blog article and my code implementation in tf_backend. Note that again, because of the chicken and egg situation, you need to create the s3 buckets first before you can track them with Terraform.

Also I needed to accept an email from AWS at this point to verify my org so make sure you do that if you get it.

Create a new account

All you need to do to add a new account is append the following to main.tf

In org/main.tf

Running terraform apply at this point created a new account for me which was great!

Create new users, groups and policies

This is fairly repetitive, so we can create a module, which in Terraform sense, is just another folder of config that you can reference as a “module”.

Keep abstractions as simple as possible

The principle here is to keep the abstraction to actually what you want to use, you don’t want to handle every use case as that can only lead to confusion.

With policies, there are so many different ways to apply them, we’re gonna stick to applying them inline on groups (and later on roles) as that’s all we need.

For the time being, let’s make an admin group and add you as an IAM user. Note that I am using a module abstraction, but they should be self explanatory, and feel free to look at the underlying module code in modules/users and modules/groups.

In org/main.tf

Some style choices I’ve made here are the choice to use jsonencode, rather than the classic file or <<EOT because I’m preferring to keep the code local, no jumping around needed, and using the built in map syntax because it’s similar to json but has benefits of not needing to include commas and the ability to add comments.

Also of note is the specification of keybase_name, this is because Terraform has a recommended Keybase implementation.

Create Keybase account with PGP key

Basically create an account and create a PGP key, this is mostly clicking around the UI for a bit. Your public key will be made public and discoverable with Terraform.

Setup AWS Vault

I have this output in my setup, where I’ve printed out the instructions for each user.

Of note is step 2, where Keybase is used to decrypt using its PGP private key.

AWS Vault may ask you for a password, I just used my password manager to track a random one. By default it will pester you for this password all the time, but this is counter productive so (for macOS) you need to disable locking on the keychain and then it will stop asking for it except when adding new accounts.

AWS AssumeRole

Here’s where the magic happens, assume role is how a user can pretend to be a “role” and therefore do things in that capacity, including across accounts.

So first we need a new provider as we need new providers for every account.

In org/main.tf

Note that this uses the admin role specified in the account creation.

And then I have the following code in my module:

In modules/env_role/main.tf

The assume_role_policy determines who is allowed to assume this role, in this case I specify everyone from a given set of account ids. The root on the end of the “arn” means any (groups, users, roles) basically. This is because we’re gonna lock it down from our jump account, so there’s no point in double locking it down practically.

We also have a aws_iam_role_policy which we use to attach an inline policy, perhaps readonly or read write for example, use as you see fit. For more information on policies, I recommend watching this video or reading the documentation for it.

Setup ~/.aws/config

Currently there’s no way to do this at the project level so you can either run a script to handle aws commands, export an env var, or put it in your global ~/.aws/config . Here’s how I’ve setup mine:

[profile team-name]
region=eu-west-2
[profile team-name-dev-ro]
source_profile=team-name
role_arn=arn:aws:iam::111111111111:role/dev_ro
[profile test-name-dev-rw]
source_profile=team-name
role_arn=arn:aws:iam::111111111111:role/dev_rw

This means when I run aws-vault login team-name it creates some credentials and logs me in automatically. If I want to jump on dev, I just run aws-vault login team-name-dev-ro . This is an extremely low friction method of jumping between accounts as necessary. I recommend checking this in to your repo to share with your team.

Limit users to dev only

Most of the time, users should not be able to login to your jump account, it’s only meant to be a jumping off point, you don’t want to create many resources there or anything like that. So you can create a new group like the following:

in org/main.tf

This limits you to one permission, which is to sts:AssumeRole and I’ve setup a role which is read only on dev.

Summary Benefits

1. Split accounts under an org structure for each of your environments.
2. Track all infra changes in Terraform, feel free to create a new account with admin privileges just as a playpen for trying things out
3. No password login with aws-vault login <env> to any environment, no need to remember urls for each account etc.

Final Thoughts

There’s many ways of doing the same thing that will be different by corporate structure, this is one setup that I found extremely easy to reason about and easy to use. Let me know if there’s anything you’d like to know about or anything I need to correct in my article!