Terraform + Diagrams: Provisioning and visualizing a simple environment on AWS

Photo by Chris Ried on Unsplash

When we are provisioning an environment in the Cloud or On-Premises, using Infrastructure as Code (IaC) concept, we already have the documentation of the project as a declarative form on the own code, however it’s become easier if we able to visualize the real state of the environment after the provisioning, do you agree? So, thinking about it I wrote this article just for show how provisioning a traditional environment by using Terraform tool and take the resources name in order to generate a final topology of the environment provisioned by using Diagrams (Diagram as Code) tool.

I created a repository¹ in my GitHub with all files used in this article.

Scope

The scope consists of provisioning an infrastructure with three instances (balanced) to receive an application in a VPC, where these instances will need to access one Database that is in another VPC.

Below we have brief topology:

           VPC1       VPC2
         +-----+    +-----+
  +------> EC2 |    |     |
LB|------> EC2 +--->+ RDS |
  +------> EC2 |    |     |
         +-----+    +-----+

AWS User and Group

First we need to create an user and associate it a group with sufficient administrative privileges in order to provision/create the required resources. So I created a user called “terraform” and I associate to “coder” group as we can see by using the command below.

aws iam list-groups-for-user --user-name <user>

Example:

[ebarros@eva ~]$ aws iam list-groups-for-user --user-name terraform 
{ 
    "Groups": [ 
        { 
            "Path": "/", 
            "GroupName": "coder", 
            "GroupId": "AGPA24FITNI6ERKVPJJ5B", 
            "Arn": "arn:aws:iam::747677903420:group/coder", 
            "CreateDate": "2019-11-28T01:09:25+00:00" 
        } 
    ] 
}

To know more about user, groups and roles on AWS, please refer the official documentation:

https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html

Directory Structure

diagrams-terrafom
├── aws_terraform
│   ├── ec2.tf
│   ├── provider.tf
│   ├── rds.tf
│   └── vpc.tf
├── diagrams_aws.py
├── requirements.txt
└── web_service.png

The “diagrams_aws.py” file, which contains the diagrama as code (DaC) content, is responsible for generate the “web_service.png” file. The “requirements.txt” file contains all dependencies required to run the python code. Inside the “aws_terraform” directory I created the .tf files according to each AWS service.
I built this directory structure because I felt more comfortable working with that. There is not a default, so feel free to create by your own way. Just make sure that the “.tf” files are in the same directory and the Terraform Engine will known how to correlate them.

Terraform

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.²

How do I install and configure it?

Please refer the Terraform official documentation:

https://www.terraform.io/docs/cli-index.html

Provider

We need to create a “provider.tf” file. There is no required arguments for this file, but it is a good pratice insert the region and the keys (or the path to them) that will allow the environment provisioning by Terraform inside the Cloud provider.

Below we have the file content:

provider "aws" {
  region     = "sa-east-1"
  access_key = "<Access_KEY>"
  secret_key = "<Secret_KEY>"
}

To know more about using AWS Provider as well its whole resources on Terraform, please refer the official documentation:

https://registry.terraform.io/providers/hashicorp/aws/latest/docs

VPC

The VPC is like a main base for all Infrastructure in Cloud, because act as a virtual Network layer.
I idealized this environment with two VPCs and Three Subnets, one VPC with one Subnet for APP and other VPC with two Subnets in different AZs for DB (required). Besides that we need to create an Internet Gateway in order to allow the traffic from APP to Internet and a VPC peering in order to have communication between VPCs. In addition to Security Groups and Route Tables.

Below we have the “vpc.tf” file content:

### VPC SECTION
## VPC for APP
resource "aws_vpc" "dac_app_vpc" {
  cidr_block           = "10.128.0.0/16"
  enable_dns_support   = "true"
  enable_dns_hostnames = "true"
tags = {
    Name = "dac_app_vpc"
  }
}
## VPC for DB
resource "aws_vpc" "dac_db_vpc" {
  cidr_block           = "10.240.0.0/16"
  enable_dns_support   = "true"
  enable_dns_hostnames = "true"
  
  tags = {
    Name = "dac_db_vpc"
  }
}
### SUBNET SECTION
## Subnet for APP
resource "aws_subnet" "dac_app_subnet" {
  vpc_id            = aws_vpc.dac_app_vpc.id
  cidr_block        = "10.128.0.0/24"
  availability_zone = "sa-east-1a"
tags = {
    Name = "dac_app_subnet"
  }
}
## Subnet for DB
resource "aws_subnet" "dac_db_subnet_1" {
  vpc_id            = aws_vpc.dac_db_vpc.id
  cidr_block        = "10.240.0.0/24"
  availability_zone = "sa-east-1b"
tags = {
    Name = "dac_db_subnet_1"
  }
}
resource "aws_subnet" "dac_db_subnet_2" {
  vpc_id            = aws_vpc.dac_db_vpc.id
  cidr_block        = "10.240.1.0/24"
  availability_zone = "sa-east-1c"
tags = {
    Name = "dac_db_subnet_2"
  }
}
### INTERNET GW SECTION
## Internet Gateway for APP
resource "aws_internet_gateway" "dac_app_igw" {
  vpc_id = aws_vpc.dac_app_vpc.id
tags = {
    Name = "dac_app_igw"
  }
}
### VPC PEERING SECTION
## Peering connection between dac_app_vpc and dac_db_vpc
resource "aws_vpc_peering_connection" "dac_app_db_peering" {
  peer_vpc_id   = aws_vpc.dac_db_vpc.id
  vpc_id        = aws_vpc.dac_app_vpc.id
  auto_accept   = true
tags = {
    Name = "dac_vpc_app_db_peering"
  }
}
### ROUTE TABLE SECTION
## Route for APP
resource "aws_route_table" "dac_app_rt" {
  vpc_id = aws_vpc.dac_app_vpc.id
route {
    cidr_block                = "10.240.0.0/16"
    vpc_peering_connection_id = aws_vpc_peering_connection.dac_app_db_peering.id
  }
route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.dac_app_igw.id
  }
tags = {
    Name = "dac_app_rt"
  }
}
## Route for DB
resource "aws_route_table" "dac_db_rt" {
  vpc_id = aws_vpc.dac_db_vpc.id
route {
    cidr_block                = "10.128.0.0/16"
    vpc_peering_connection_id = aws_vpc_peering_connection.dac_app_db_peering.id
  }
tags = {
    Name = "dac_db_rt"
  }
}
## Route Table - Subnet Associations
resource "aws_route_table_association" "dac_app_rta2" {
  subnet_id      = aws_subnet.dac_app_subnet.id
  route_table_id = aws_route_table.dac_app_rt.id
}
resource "aws_route_table_association" "dac_db_rta1" {
  subnet_id      = aws_subnet.dac_db_subnet_1.id
  route_table_id = aws_route_table.dac_db_rt.id
}
resource "aws_route_table_association" "dac_db_rta2" {
  subnet_id      = aws_subnet.dac_db_subnet_2.id
  route_table_id = aws_route_table.dac_db_rt.id
}
### SECURITY GROUPS SECTION
## SG for APP VPC
resource "aws_security_group" "dac_app_sg" {
  name = "dac_app_sg"
  description = "EC2 instances security group"
  vpc_id      = aws_vpc.dac_app_vpc.id
ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    description = "Allow SSH from my Public IP"
    cidr_blocks = ["<public_IP>/32"]
  }
ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    description = "Allow HTTP traffic"
    cidr_blocks = ["0.0.0.0/0"]
  }
ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    description = "Allow HTTPS traffic"
    cidr_blocks = ["0.0.0.0/0"]
  }
egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  
  tags = {
    Name = "dac_app_sg"
  }
}
## SG for DB VPC
resource "aws_security_group" "dac_db_sg" {
  name = "dac_db_sg"
  description = "EC2 instances security group"
  vpc_id      = aws_vpc.dac_db_vpc.id
ingress {
    from_port   = 3306
    to_port     = 3306
    protocol    = "tcp"
    description = "Allow traffic to MySQL"
    cidr_blocks = ["10.128.0.0/24"]
  }
egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
tags = {
    Name = "dac_db_sg"
  }
}

Summary of main resources created by the previous “.tf” file:

• dac_app_vpc: VPC for APP (CIDR: 10.128.0.0/16)
• dac_db_vpc: VPC for DB (CIDR: 10.240.0.0/16)
• dac_app_subnet: Subnet for APP (CIDR: 10.128.0.0/24)
• dac_db_subnet_1: Subnet 1 for DB (CIDR: 10.240.0.0/24)
• dac_db_subnet_2: Subnet 2 for DB (CIDR: 10.240.1.0/24)
• dac_app_igw: Allow the traffic from APP to Internet
• dac_vpc_app_db_peering: Allow the communication between VPCs
• dac_app_rt: Route table for APP
• dac_db_rt: Route table for DB
• dac_app_sg: Security group for APP VPC (Ingress: allow ports 22, 80 and 443. Egress: All networks)
• dac_db_sg: Security group for DB VPC (Ingress: allow only port 3306 from APP Subnet. Egress: All networks)

To know more about Amazon VPC, please refer the official documentation:

https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html

RDS

RDS is a fully managed resource database in AWS which allow to create a database instance without care about OS Patching and Server Provisioning, for instance. This resource supports the main DB engines such as SQL Server, MariaDB, ProstgreSQL, etc. In this case the file “rds.tf” will create a MySQL instance and a DB subnet group by using the DB Subnets previously declared in the “vpc.tf” file.

Below we have the “rds.tf” file content:

## DB Subent Group
resource "aws_db_subnet_group" "dac_db_subnet_group" {
  name       = "dac_db_subnet_group"
  subnet_ids = [aws_subnet.dac_db_subnet_1.id, aws_subnet.dac_db_subnet_2.id]
tags = {
    Name = "dac_db_subnet_group"
  }
}
## DB instance
resource "aws_db_instance" "dac_db" {
  allocated_storage       = 20
  storage_type            = "gp2"
  engine                  = "mysql"
  engine_version          = "8.0"
  instance_class          = "db.t2.micro"
  name                    = "mydb"
  identifier              = "dacdb"
  username                = "<db_user>"
  password                = "<db_password>"
  parameter_group_name    = "default.mysql8.0"
  db_subnet_group_name    = aws_db_subnet_group.dac_db_subnet_group.name
  vpc_security_group_ids  = [aws_security_group.dac_db_sg.id]
  skip_final_snapshot     = "true"
}

To know more about Amazon RDS, please refer the official documentation:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html

EC2

AWS EC2 is a computing layer service that basically allows to create/destroy Linux or Windows based instances (VMs), scale up or scale out according to demand. In this case the file “ec2.tf” will create three Linux instances (t2.micro) by using “Amazon Linux 2” image. This file will also create a NLB (Network Load Balancer) and its required components.

Below we have the “ec2.tf” file content:

## EC2 INSTANCES
resource "aws_instance" "dac_app" {
  count                         = 3
  ami                           = "ami-02898a1921d38a50b"
  instance_type                 = "t2.micro"
  key_name                      = "<KEY_name>"
  vpc_security_group_ids        = [aws_security_group.dac_app_sg.id]
  subnet_id                     = aws_subnet.dac_app_subnet.id
  associate_public_ip_address   = "true"
tags = {
    Name = "dac_app_${count.index}"
  }
}
## NLB
resource "aws_lb" "dac_app_lb" {
  name               = "dac-app-lb"
  internal           = false
  load_balancer_type = "network"
  subnets            = aws_subnet.dac_app_subnet.*.id
tags = {
    Environment = "dev"
  }
}
## LB Target Group
resource "aws_lb_target_group" "dac_app_tgp" {
  name     = "dac-app-tgp"
  port     = 80
  protocol = "TCP"
  vpc_id   = aws_vpc.dac_app_vpc.id
}
## LB Targets Registration
resource "aws_lb_target_group_attachment" "dac_app_tgpa" {
  count            = length(aws_instance.dac_app)
  target_group_arn = aws_lb_target_group.dac_app_tgp.arn
  target_id        = aws_instance.dac_app[count.index].id
  port             = 80
}
## LB Listener
resource "aws_lb_listener" "dac_app_lb_listener" {
  load_balancer_arn = aws_lb.dac_app_lb.arn
  port              = "80"
  protocol          = "TCP"
default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.dac_app_tgp.arn
  }
}

Summary of main resources created by the previous “.tf” file:

• dac_app: Three EC2 Instances (dac_app_0, dac_app_1 and dac_app_2)
• dac_app_lb: External Network Load Balancer
• dac_app_tgp: Target group to receive traffic from NLB
• dac_app_tgpa: Group Instances attached on dac_app_tgp
• dac_app_lb_listener: LB listener on port 80 (HTTP)

To know more about Amazon EC2, please refer the official documentation:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html

Planning and applying

There is three main commands in Terraform, which are:

• terraform plan: Create a plan, based on declared “.tf” files, of the environment that will be provisioned.
• terraform apply: Deploy the whole environment on Cloud provider according to the plan.
• terraform destroy: Destroy the whole environment previously created.

Below we can see a example of the “terraform plan” and “terraform apply” commands:

Environment provisioning process

Provisioning Results

On terminal below we can see the filtered outputs of the AWS CLI commands showing the resources that has been created by Terraform.

Showing the resources

If you want to destroy the whole environment, just type “terraform destroy”, then confirm the plan and all resources will be deleted.

Note: Keep the provisioned environment until the final of article, because we will use the “terraform.tfstate” file in the next topic.

Diagrams

Diagrams lets you draw the cloud system architecture in Python code. Diagrams currently supports six major providers: AWS, Azure, GCP, Kubernetes, Alibaba Cloud and Oracle Cloud. It now also supports On-Premise nodes as well as Programming Languages and Frameworks.³

How do I install, configure and use it?

Please refer the Diagrams official website and the project repository on GitHub:

https://diagrams.mingrammer.com/
https://github.com/mingrammer/diagrams

How it works?

This simple tool works over Python, so after a minimal configuration basically we just need write the “.py” file and run it.

Generating the Diagram

First of all, we need to import to the code all required classes according each AWS resource that will be displayed in the topology.
Then I opened the “terraform.tfstate” file and decoded it as a JSON in order to get all resources name that was created during the environment provisioning.
After that I built a list with each resource name and used each element position as names to my Nodes and Clusters.

Note: This process was made manually inside the code, but I intend to automate it by using Python CDK for Terraform and show in a future article.

The sintax to build the diagram scheme is explained with more detail in the official documentation, but basically we need to use the each AWS resource class to create the nodes, “<< >>” or “--” to connect the nodes and the Cluster class to create the “boxes”.

The “diagrams_aws.py” file shows the code below:

from diagrams import Cluster, Diagram
from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.aws.network import ELB, VPCPeering
import json
import os
from pprint import pprint
## Opening the .tfstate file
with open('aws_terraform/terraform.tfstate') as json_file:
    tf_data = json.load(json_file)
#pprint(data)
r_names = []
## looping over the names
for x in tf_data['resources']:
    r_names.append(x['name'])
print("\nResources Name List: \n")
pprint(r_names)
app_vpc_name = r_names[len(r_names)-3]
db_vpc_name = r_names[len(r_names)-2]
app_vpc_cidr = '10.128.0.0/16'
db_vpc_cidr = '10.240.0.0/16'
app_subnet_name = r_names[len(r_names)-6]
db_subnet1_name = r_names[len(r_names)-5]
db_subnet2_name = r_names[len(r_names)-4]
app_subnet_cidr = '10.128.0.0/24'
db_subnet1_cidr = '10.240.0.0/24'
db_subnet2_cidr = '10.240.1.0/24'
peering_name = r_names[len(r_names)-1]
app_name = r_names[2]
db_name = r_names[0]
tgp_name = r_names[6]
lb_name = r_names[4]
print("\nGenerating Diagram...")
with Diagram("Web Service", show=False):
load_balancer = ELB(lb_name)
with Cluster("VPC: "+app_vpc_name+"\nCIDR: "+app_vpc_cidr):
        with Cluster("Subnet: "+app_subnet_name+"\nCIDR: "+app_subnet_cidr):
            with Cluster("Instance Group: "+tgp_name):
                workers = [EC2(app_name+"_0"),
                            EC2(app_name+"_1"),
                            EC2(app_name+"_2")]
    
    vpc_peering = VPCPeering(peering_name)
with Cluster("VPC: "+db_vpc_name+"\nCIDR: "+db_vpc_cidr):
            with Cluster("Subnet 1: "+db_subnet1_name+"\nCIDR: "+db_subnet1_cidr+"\nSubnet 2: "+db_subnet2_name+"\nCIDR: "+db_subnet2_cidr):
                database1 = RDS(db_name)
load_balancer >> workers >> vpc_peering >> database1
print("Congratulations! The diagram has been succesfully generated in this path: "+os.getcwd()+"\n")

Finally, running the below command:

python3 diagrams_aws.py

As a result we have the complete and wonderful diagram below:

Final topology of the provisioned AWS resources

[1]: GitHub repository: terraform-diagrams.
https://github.com/ebarros29/terraform-diagrams

[2]: Terraform official website. Introduction to Terraform
https://www.terraform.io/intro/index.html

[3]: Diagrams: Diagram as Code. About Diagrams
https://diagrams.mingrammer.com/