API services on cloud #2: Divide and rule
Note: this is the Part 2 of the “API services on cloud” series on running API Services on cloud and learning cloud infrastructure automation. Stay in the know. There’s more of this to come!
< < Start over at Part 1 | Next >
Intro
In the previous tutorial I wrote about creation of a simple infrastructure to run an API service. For that example the solution was good. But there were two disadvantages which I want to solve in this tutorial:
- Monolith template - it is hard to manage, the addition of new resources is complicated.
- Instances are completely isolated from the external world - this makes issues debugging process too complicated and ‘blind’. Yes, sometimes we still need to log into the instance to understand what’s wrong.
To solve the first problem I’m going to use the principle ‘divide and rule’. In this tutorial I will explain how I partitioned one massive CloudFormation template into smaller templates.
To solve the second problem I am going to add a Bastion box to the infrastructure. And allow SSH access only from that instance to the service instances.
The infrastructure diagram is almost the same except the Bastion host which settled into one of the public subnetworks. Thus I am not repeating it here. Please, see the Part 1 for the reference.
Prerequisites
- To be able to log into Bastion host and then from Bastion into a private instances you need an EC2 key pair. Please, follow the instructions to create the EC2 KeyPair. The key pair should never be committed to the public repositories.
- S3 bucket to upload CloudFormation templates.
Splitting the monolith template
The new CloudFormation template structure consists of the following parts:
- Master template (new) - includes all children templates and manages dependencies between them;
- Network template - creates theVPC, Internet and NAT Gateways, subnetworks, and routing tables;
- Security Groups template (updated) - creates three groups for: EC2 instances, LoadBalancer and Bastion host;
- Load Balancer template - creates LoadBalancer, Listener and a Default Target Group;
- Bastion Host template (new) - creates a Bastion host;
- Simple API Service template (updates) - creates an AutoScaling group and roles for the service.
Ok, lets review what’s new and updated. Here’s the Master template overview:
As you can see in the Parameters section, I added KeyName parameter. Also, I created an EC2 key pair named SimpleAPIKey. In your case this value can be different - provide the name of the key pair you created.
Also, templates must be stored on an AWS S3. Thus you will need to allocate a bucket and provide the link to your templates in TemplateURL property value.
The other useful function I used in this template was !GetAtt. It allows to get attribute values of the created resources. This function is helpful when you work with nested stacks. Here you can find more information about this function.
Let’s see what’s changed in the security groups.
Here I added a new Security Group called BastionSecurityGroup. This group is supported by two additional groups:
- BastionSecurityGroupToEC2HostEgress
- EC2HostSecurityGroupFromBastionIngress
Also, I added BastionSecurityGroup to the list of ingress security groups of the EC2HostSecurityGroup. Together these groups define inbound and outbound traffic rules for the Bastion and service hosts.
There are the following rules defined:
- Load balancer accepts any traffic;
- Bastion host accepts connections only by SSH. Moreover the list of the source IP addresses can be limited by the parameter SSHFrom;
- EC2 instances accepts any traffic from load balancer;
- EC2 instances accepts only SSH connection from Bastion host.
Let’s review the Bastion host template.
Here I created an instance and allocated a public IP address for it. Also, dedicated log group and alarms on the certain events have been added.
This template is quite simple and does not have all the useful features. Please, check this template for advanced usage.
And the last update was in the EC2 launch configuration. I added KeyName to support SSH access from the Bastion host.
The full template is available at SimpleAPI repository. The previous version of the template is available here.
Some automation
It’s quite easy to create and update CloudFormation stacks from the web-console. But as soon as you need to deal with the nested templates the manual update becoming complicated and error prone. That is why I added several bash scripts to upload, create, and update CloudFormation templates.
The main thing here is AWS_PROFILE. As soon as you create an AWS User you can download it’s credentials. The credentials include AWS Access Key ID and AWS Secret Access Key and should never be committed to the public repositories. These assets allow you to remotely call AWS Service with AWS CLI.
Here is the structure of AWS credentials file:
The AWS credentials file - located at ~/.aws/credentials on Linux, macOS, or Unix, or at C:\User\USERNAME\.aws\credentials on Windows.
Thank you
That’s all for today. I thank you for reading till the end. Please ‘clap’ or/and comment if you liked it. If you did not like it, please comment why - so that I will be able to improve it next time.
References
- SimpleAPI - NodeJS service and CloudFormation source code.
- Monolith CloudFormation template (from the Part 1).
- AWS CloudFormation templates startup kit.
- Create an EC2 key pair.
