Manage your AWS resources with Puppet

If you are familiar with Puppet for configuration management, you will probably be wondering if you can use Puppet to manage AWS resources. The answer is Yes, but bad news is, it does not provide complete control as you might expect. Well, for many use-cases, in small to medium infra setup, Puppet probably is still good enough. We all know Puppet general use-case is to automate repetitive and manual tasks. Luckily, only a handful of activities are repetitive, those are what puppet can cover.

In this blog, I will go through creation of Public-Private VPC with NAT Server and Bastion host using Puppet. The configuration subsequently will be used to manage your subnets, routing table, security groups and EC2 instances. If you already have existing VPC, you still can get Puppet to manage it, but with extra efforts to translate your existing setup into Puppet code.

Why Puppet?

Before we go too far, to give you more motivation (or rather, if it doesn’t suit you, you won’t waste your time here), let me explain why you would use Puppet?

Puppet vs Manual

You definitely can go manual, login to AWS console and do everything by hand. But the more instances, subnets and security groups you manage, the less efficient will your process go. AWS console will far far slower than just updating a few lines in Puppet. How about someone mistakenly configure, or change your security groups, or routing tables? They easily mess up your whole infra, don’t they? Even if it did not do that level of damage, are you able to comfortably revert to original state? Have you made a copy of your security group entries somewhere?

Puppet vs CloudFormation

CloudFormation have more complete, solid and integrated solution that any Puppet can bring to the table. Why bother look into Puppet at the first place? There are a few reasons you might be better served by Puppet:

  • You already have Puppet in place, adding AWS into the picture make a lot of sense
  • You want to have flexibility to control by hand sometime, where CloudFormation will not allow, once you have decided to use it
  • You are more comfortable with easy to understand pp file format than CloudFormation json format
  • Get CloudFormation up and running overwhelmed you

Objective

Our minimum objectives here are:

  1. manage subnets from Puppet
  2. manage routing table & security groups from Puppet
  3. manage instances from Puppet

You would be tempting to add many more into the list, like managing auto scaling group or load balancers. Unfortunately, current Puppet AWS module is lack of features to control many important aspects of those resources.

Requirements

  • Linux environment
  • Puppet 3.4 or greater

Installation

If you do not have Puppet installation, follow this guide to help you set it up quickly: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-masterless-puppet-environment-on-ubuntu-14-04

Once you already have Puppet installed, you will need to install Puppet AWS module. Follow this guide to do installation: https://github.com/puppetlabs/puppetlabs-aws. For security reason, I would recommend to install Puppet in a protected instance: inside a private subnet, and can only be reached via Bastion host.

Overview of Steps

We will start with creating our own module. We will then creating a series of pp file inside the module, from VPC to EC2 instances. Each pp file will manage one specific resource category, eg. security groups, subnets or instances. We like to have public subnet and private subnet as separate pp file, due to private subnet dependency to NAT server, a resource created in the public subnet. We will do the following in order:

  1. Create Puppet module
  2. Create VPC
  3. Create Security Groups
  4. Create Public Subnet
  5. Create NAT server
  6. Create Private Subnet
  7. Create Instances

Create Puppet Module

Create a folder under /etc/puppet/modules.

cd /etc/puppet/modules
mkdir <my-module>
mkdir <my-module>/manifests

And write init.pp file under /etc/puppet/modules/<my-module>/manifests folder

class <my-module> {
}

Create a VPC

Create a file named vpc.pp under the same folder

class <my-module>::vpc {
ec2_vpc { ‘<my-vpc-name>’:
ensure => present,
region => ‘ap-southeast-1’,
cidr_block => ‘11.0.0.0/16’,
}
ec2_vpc_internet_gateway { ‘<my-vpc-name>-igw’:
ensure => present,
region => ‘ap-southeast-1’,
vpc => ‘<my-vpc-name>’,
}
}

Replace my-vpc-name, region and cidr block with with your own name, region, and cidr block. Above block will create a VPC and default Internet Gateway.

Create pp file that let you apply configuration from command line. Go to main manifests folder /etc/puppet/manifests , and create a pp file, called aws-vpc.pp

node default {
     include <my-module>::vpc
}

This basically tells Puppet to apply configuration in your module under vpc class to default node. At this point you can run puppet to create your VPC.

sudo puppet apply aws-vpc.pp

Go to AWS Console — Networking — VPC. It should be there, the new VPC you created through Puppet. Exciting, isn’t it? You have just created a VPC and an Internet Gateway using a few lines of Puppet code.

Create Security Groups

Let’s do some more interesting Puppet magic. Create another file, security.pp under the same folder as vpc.pp

class <my-module>::security {
ec2_securitygroup { ‘<my-module>-sg-bastion’:
ensure => present,
region => ‘ap-southeast-1’,
vpc => ‘<my-vpc-name>’,
description => ‘Security group for bastion’,
ingress => [ {
protocol => ‘tcp’,
port => 22,
cidr => ‘0.0.0.0/0’
}]
}
ec2_securitygroup { ‘<my-module>-sg-nat’:
ensure => present,
region => ‘ap-southeast-1’,
vpc => ‘<my-vpc-name>’,
description => ‘Security group for nat server’,
ingress => [{
protocol => ‘tcp’,
port => 22,
security_group => ‘<my-module>-sg-bastion’,
}]
}
}

This pp file will setup security groups for your VPC. The first one is for Bastion host or jump server, allow your connect to other servers in your VPC. The second one is for NAT server, it allows for ssh connection from Bastion host.

Similar to VPC, create pp file to let you apply configuration from command line. Go to main manifests folder /etc/puppet/manifests , and create a pp file, called aws-security.pp

node default {
  include <my-module>::security
}

Run the following command to apply configuration

sudo puppet apply aws-security.pp

Go to AWS Console — Networking —Security Groups. It should be there, the new Security Groups you created through Puppet.

Create Public Subnet

Create another file subnet-pub.pp under the same folder as your vpc.pp

class <my-module>::subnet-pub {
ec2_vpc_routetable { ‘<my-vpc-name>-rt-pub-1a’:
ensure => present,
region => ‘ap-southeast-1’,
vpc => ‘<my-vpc-name>’,
routes => [
{
destination_cidr_block => ‘11.0.0.0/16’,
gateway => ‘local’
},{
destination_cidr_block => ‘0.0.0.0/0’,
gateway => ‘<my-vpc-name>-igw’
},
],
}
ec2_vpc_subnet { ‘<my-vpc-name>-pub-1a’:
ensure => present,
region => ‘ap-southeast-1’,
vpc => ‘<my-vpc-name>’,
cidr_block => ‘11.0.10.0/24’,
availability_zone => ‘ap-southeast-1a’,
route_table => ‘<my-vpc-name>-rt-pub-1a’,
}
}

This file will do 2 things: create routing table for your public subnet, and the public subnet itself. Public subnet is where you will place your NAT server.

Similarly, create aws-subnet-pub.pp, and run the command to apply configuration

sudo puppet apply aws-subnet-pub.pp

Create NAT Server

Now it’s time to create NAT server. Before you create an instance, decide how you are going to access the instance later on. Usually we create one root key for instances in the same VPC. You may want to do the same. Go to AWS Console — EC2 — Keys, and Generate Key Pair with <my-vpc-name> as name. Once you do that, go on create pp file for NAT, nat.pp:

class <my-module>::nat {
ec2_instance { ‘<my-vpc-name>-nat-1a’:
ensure => present,
region => ‘ap-southeast-1’,
availability_zone => ‘ap-southeast-1a’,
image_id => ‘ami-b49dace6’,
instance_type => ‘t2.micro’,
monitoring => false,
key_name => ‘<my-vpc-name>’,
security_groups => [‘<my-vpc-name>-sg-nat’],
subnet => ‘prov-prod-pub-1a’,
associate_public_ip_address=> true,
tags => {
‘Project’ => ‘your project’,
‘Environment’ => ‘your environment’,
},
}
}

This configuration will create a NAT instance for you VPC. However, Puppet currently does not allow you to set source/dest. check setting that is required in a NAT instance. You have to do it manually via AWS console, to set instance source/dest. check to false. After creating aws-nat.pp file, run:

sudo puppet apply aws-nat.pp

Create Private Subnets

Now you are ready to create private subnets. Create a pp file, subnet-priv.pp:

class <my-module>::subnet-priv {
ec2_vpc_routetable { ‘<my-vpc-name>-rt-priv-1a’:
ensure => present,
region => ‘ap-southeast-1’,
vpc => ‘<my-vpc-name>’,
routes => [{
destination_cidr_block => ‘11.0.0.0/16’,
gateway => ‘local’
},{
destination_cidr_block => ‘0.0.0.0/0’,
gateway => ‘<my-vpc-name>-nat-1a’
},
]
}
ec2_vpc_subnet { ‘<my-vpc-name>-priv-1a’:
ensure => present,
region => ‘ap-southeast-1’,
vpc => ‘<my-vpc-name>’,
cidr_block => ‘11.0.20.0/24’,
availability_zone => ‘ap-southeast-1a’,
route_table => ‘<my-vpc-name>rt-priv-1a’,
}
}

It’s very similar to pp file for public subnet, the only different is in the private subnet you can’t directly connect to the internet, it has to go through NAT. The routing table ensure this. The gateway points to your NAT server you created previously, rather than an internet gateway. Go on create aws-subnet-priv.pp file, and run puppet apply

Create EC2 Instances

Finally we reach to the end of our tutorial. We will be creating a bastion host, and placed it inside public subnet. Create a pp file ec2.pp

class <my-module>::ec2 {
ec2_instance { ‘<my-vpc-name>-bastion’:
ensure => present,
region => ‘ap-southeast-1’,
availability_zone => ‘ap-southeast-1a’,
image_id => ‘ami-c28240a1’,
instance_type => ‘t2.micro’,
monitoring => false,
key_name => ‘<my-vpc-name>’,
security_groups => [‘<my-vpc-name>-sg-bastion’],
subnet => ‘<my-vpc-name>-pub-1a’,
tags => {
‘Project’ => ‘my-project’,
‘Environment’ => ‘my-env’,
}
}

Go on create aws-ec2.pp file, and run puppet apply

How do we go from here

With all the Puppet code in place, you are ready with any changes of infrastructure in the future. Checkin your Puppet code into code repository to keep your configuration in a safe place.

In future if you need to create an EC2 instance, you just need to modify ec2.pp file, and set security groups and subnets to your existing setup. If you need new security group or subnet, you just need to update respective pp file and run puppet apply. Not only it will make managing AWS resources more manageable, puppet code will also serve as documentation for your infrastructure.