Deploying your first micro-service in AWS using Pulumi
Written with the assistance of Jdcallet. Special acknowledgment for his help and support.
Infrastructure As Code (IaC) has become a key concept when working with Cloud providers and implementing DevOps, having IaC in your projects helps you manage, automate, test, and simplify deployment processes by building CI/CD pipelines that can integrate the application and the infrastructure code.
There are several IaC tools compatible with the most popular cloud providers. The most common tools are CloudFormation, which works with AWS specifically, and Terraform, which can be implemented in several cloud providers. When you want to explore the IaC world you may need to learn a new language like YAML for CloudFormation, or another DSL according to the tools that you choose. If you already have basic knowledge in programming languages you may want to apply some concepts like loops, exceptions, conditionals, etc. That can be difficult if you are working with CloudFormation or Terraform because these do not use real programming languages.
Pulumi is an IaC tool that utilizes popular programming languages to provision and manage your cloud resources. Pulumi is open-source and it supports Python, Golang, Node.js, and .Net. With Pulumi you can take advantage of loops, conditionals, and other options native to other programming languages, leveraging the power of these languages into your IaC scripts.
In this article, we are going to dive into a microservices architecture in which we will use Pulumi to deploy an ECS cluster to run a microservice built-in Python3.
Pulumi basic concepts
Before the hands-on, you should have some knowledge related to Pulumi, these are the three concepts that you will need before starting with Pulumi. These concepts can be reviewed in detail in the Pulumi Documentation:
Program: This is the code that you write to create your infrastructure, this can include multiple files in your programming language of choice, from Pulumi CLI you run Pulumi up command and Pulumi will execute it and determine the desired infrastructure state for all resources declared.
Project: This is like a directory that contains your program, this is defined by the Pulumi.yaml file that is used to define the runtime(Python, Golang, Node.js, or C+) that is used to customize configurations. Every folder containing a Pulumi.yaml can be considered a project
Stacks: Stacks are like units that you can use to divide your project into environments like test, production, or staging. If you have worked with Terraform, you can think of the stacks as Terraform workspaces. Each stack can contain variables that you can use in your code.
Getting started with Pulumi
In order to start with this project you will need the following tools:
- Cloud Provider Account: for this case, we will use AWS, you need an AWS account and an IAM user with programmatic access, set enough permissions to create the resources defined in the scripts.
- Runtime: Pulumi supports Golang, Python, Node.js, and C#, you will need to install the correct package to run code in the language that you want, for this case we will use Python 3.6.
- Pulumi Account: You need a Pulumi account to store the state of your infrastructure and manage your project, please follow this link for more information. https://www.Pulumi.com/docs/intro/console/accounts-and-organizations/accounts/
- Pulumi CLI: You need the Pulumi CLI to deploy the infrastructure changes in AWS. https://www.Pulumi.com/docs/get-started/install/
What are we going to deploy with Pulumi?
Pulumi can be used with many Cloud providers and resources. This architecture shows the resources to be deployed using Pulumi with Python3 in AWS. In the repository, we have defined the requirements to launch these resources(README.md).
This is a typical microservices architecture deployed using containers and orchestrated with AWS Fargate. This architecture provides high availability and scalability for a simple API built-in Flask.
This is the whole list of AWS resources to be deployed with Pulumi:
- 1 VPC
- 2 public and 2 private subnets
- NAT Gateway
- ECS cluster with one service running task running in Fargate
- Application Load Balancer
- Security groups for the ECS tasks and ALB
- DynamoDB table
- IAM role with the permission to make calls to DynamoDB from the ECS task
The code
In this case, we will use Python3 and POO(Object-Oriented Programming). In the GitHub Repository, you will find the infrastructure folder that contains the Python code built to deploy the AWS resources. This code uses Python local packages stored in the aws_components folder, that contains a package for each AWS resource required. Pulumi uses a Python virtual environment, the file requirements.txt specifies the packages that the code will use with the virtual environment.
ALB folder defines the aws_alb class and this contains the methods to create the components that AWS ALB needs to work, for example, listener or target group creation.
We are defining the __init__ method to initialize the attributes that we pass when we create an object from this class. The create_alb method creates an ALB by the Pulumi package for AWS imported into line 2. Each AWS resource has some parameters that you must specify for the creation. You can find more information in Pulumi Documentation for AWS.
As we mentioned earlier, using popular programming languages allows you to take advantage of the power that each language has, for instance, in the image above we are creating an AWS Networking, the code in the infra.pyx file shows how we are creating the object from the class VPC that we are defining in the code above (networking_class.pyx file). We are creating two Python lists with the CIDR blocks that we want to assign for each subnet, in the class definition we are using a for statement to read the list and create the subnets.
In the repository you will find a file named infra.py, this file creates each component using the classes defined.
Where is stored the infrastructure states?
Pulumi stores the state of the infrastructure in Pulumi Web by default, according to Pulumi documentation multiple snapshots are taken to reliably support the infrastructure states. However, you can store the states locally or use cloud services like AWS S3, Azure blob storage, or Google Cloud Storage.
In your Pulumi account, you can manage your projects, stacks, and states. Each stack contains information about the resources created, the activity, and the variables configured.
In this case, we are storing the state in Pulumi Web, and all the activity in the stack defined is tracked by Pulumi. Each project can have multiple users and all their actions are registered by Pulumi.
How to manage the outputs for each resource?
Each resource created by Pulumi has some outputs that you can pass to other resources, for example, when you create a VPC you need the VPC ID to create the subnets, the Pulumi documentation has more detail about the output for each AWS resource. Outputs are returned as properties on the instantiated resource object, think in the outputs like an object with parameters that are used to map dependencies between other resources. According to Pulumi SDK for Python Documentation “Output helps encode the relationship between Resources in a Pulumi application. Specifically an Output holds onto a piece of Data and the Resource it was generated from”. Keep in mind that you can not access the output directly, for instance, the method create_alb into the class aws_alb returns a Python dictionary with the ARN, DNS, and Zone ID for the ALB created, you can not print or manipulate these outputs directly because each element in the dictionary is a Pulumi Output object, if you try to print that, you will get the following output.
You can use two classes that Pulumi has defined to manipulate the outputs, for example, if you want to create a full URL for the ALB you can use the apply method. In the following code, we will create a variable ALB_URL to store the URL with the https protocol(line 10). apply() method takes a Pulumi Output and adds a string that you specify, but the output of the apply method is still an Ouput object, that maintains the metadata that is used to mapping dependencies between the resources.
Keep in mind that you can not convert the Outputs to String, if you want to know the value for a specific Output you could use an export function(line 12 in code above)that prints in the CLI the outputs of the Pulumi program that you are running, that will look like:
Testing
Testing is an important way to be sure that your code will work, however, how can you test the infrastructure code?. Here we have another advantage of using common programming languages in our infrastructure resources. You can take advantage of testing frameworks that have been defined for each programming language. In this case, we will use unittest framework for Python3.
Pulumi Testing Documentation has a great description of testing with examples that you can follow. For this case, we created two basic unit tests using mocks. These tests do not require that you launch the resources in the cloud provider.
The first step is to define the mocks that you will use, Pulumi provides a Mocks class in which you have two methods, new resource, and call methods. In the first method, you can define the outputs for the resources that you want to mock, for example, in the following code we created an if statement( line 12) to return an ARN if the resources that the code will try to create is an ALB listener. This mock will be used to test the methods that we defined in the aws_alb class.
Take a look at line 42, we are defining the test_listener_method in which we create an object from the aws_alb class, the idea is to test the method to create an ALB listener.
Testing is a topic that can involve several components and that is beyond the scope of this article. In future posts, we will try to cover this topic in detail.
Conclusion
We have seen how we can use Python3 to define the infrastructure in AWS for a simple microservice, we can take advantage of a real programing language and apply the knowledge that we already have. We are free to use external libraries and software practices that help us to build great code for our infrastructure.
