Provision Resources in AWS using your own REST API

What if I can provide you one “API” URL, where you can just simply send a “POST” request & it will start provisioning resources in AWS cloud. Then it will return you the required info to access that…

Raktim Midya

Published in

Geek Culture

14 min readJul 19, 2022

Hey There… 🙌

Hope you are having a great time with your journey of Cloud Computing. It’s been a while I haven’t wrote because I was learning many new ways of implementing various ideas & today I’m going to share one amazing used case. Have a tea/coffee & keep on reading… 😇

Problem Statement :

Create one CloudFormation Template to automate the provisioning of the resources on AWS.
Create one REST API using AWS API Gateway & connect it with a Lambda Function.
Use that Lambda Function to deploy the CloudFormation Stack.
Definitely create the associated IAM Roles & Policies.

Sounds simple right… But not so simple. 😅 Don’t worry because I know the best way to make it simple for you. 😎
Let’s discuss some prior needs for this used case…

Pre-requisites :

You obviously need one AWS account. Also having the basic knowledge of AWS VPC, S3, IAM is required. I’m going to discuss about CloudFormation, Lambda & API Gateway in this blog.

Amazon API Gateway :

The full form of API is Application Programming Interface & Gateway stands for a path using which you can send your requests to a certain location. AWS provide one managed API service using which you can very easily create one REST API. I’m not discussing about REST right now, but simply understand that API Gateway will give you one URL where our Clients can send requests.
When we will start setting up the API Gateway, you will understand it’s capabilities & working principals. One more thing to note that we will be using HTTPS API, which means our clients will be Web Clients like your Web Browser or Curl command.
Official Documentation Link : https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html

AWS Lambda :

Lambda stands for serverless function execution in AWS. Let’s say you have one python function which upon giving two numbers, return the sum. Now to run that function you need one OS with python compiler installed. Cloud in this case works very uniquely.
You just need to provide your function code to Lambda Service & obviously you need to tell which runtime you want to select. Then Lambda will run that function & will return the output. Most fascinating part of this whole thing is for that you need to pay for only those few seconds of your function execution. It’s very much cost effective in a sense that you don’t need to launch any kind of VM or you don’t need any kind of storage etc. This is the reason it’s known as serverless service which means the backend server is managed by AWS.
Official Documentation Link : https://docs.aws.amazon.com/lambda/latest/dg/welcome.html

AWS CloudFormation :

CloudFormation is a service which allows you to implement all the concepts of IAC (Infrastructure as Code). We’re going to write scripts also knows as CloudFormation Template to automate the provisioning of the resources on AWS. It’s very easy to write these codes because of a concept known as Declarative Programming Approach, where you just need to tell your requirements.
Next CloudFormation knows how to provision those things & if already provisioned then once updating the code, it knows how to modify the resources on AWS based on latest code base. You will see how simple this coding is respective to other DSA coding. 😅
Official Documentation Link : https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html

Step 1 : Creating CloudFormation Template & deploying in S3

Let’s start simple, we’re going to create one CloudFormation Template which will provision a VPC in AWS. For that I highly recommend to go to official documentation of CloudFormation.

Official Link : https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc.html

I picked the example provided in the official doc which is really simple to understand…

---
AWSTemplateFormatVersion: "2010-09-09"
Description: Template to provision one VPCResources:
  myVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      Tags:
      - Key: stack
        Value: production

You are creating one VPC in AWS with a network address of “10.0.0.0/16”. You have also enabled DnsSupport & DnsHostname for that particular VPC. Lastly we attach one tag to our VPC.

Now simply save this code in a file called — “cf-resources.yaml”. You can give any name you want to your file.

Next, go to Amazon S3 & create one bucket with below configurations -

Bucket Name: “raktim-cf-bucket”
Region: “ap-south-1”
ACL enabled & uncheck “Block all public access”.
Enable versioning & disable server-side encryption.

We can do tons of configuration in S3 to protect your code, but it’s not a blog related to security. That’s why I’m making things simple to work around.

Next, upload your “cf-resources.yaml” file in the bucket & make sure while uploading the file you choose “Permissions” to “Grant public-read access”. Otherwise Lambda will not be able to use the template through it’s public URL.

Copy it’s Object URL. In my case the Object URL looks something like — https://raktim-cf-bucket.s3.ap-south-1.amazonaws.com/cf-resources.yaml

Step 2 : Creating IAM Roles

I always suggest don’t just write IAM Roles beforehand because that’s never going to help you to learn IAM & security. First see what kind of basic requirements you need for one service to do it’s work.

If your Lambda needs to access CloudFormation to deploy the stack, then only create role with that much power. If CloudFormation needs to create VPC in your account, then create role with those limitations. Hope you got my point :)

Role for Lambda :

Let’s go to AWS IAM Service & then go to → Roles & click on “Create Role” & follow the below configurations…

Trusted Entity Type: AWS service
Use Case: Lambda
Permission Policies: AWSCloudFormationFullAccess
Role Name: lambda-cf-role

Finally create the role. I know that here we gave “AWSCloudFormationFullAccess” policy which might looks dangerous but believe me as we are going to write the Lambda Code by ourselves, so it’s not that bad. Plus I’m again emphasizing that this blog is not primarily focused on security, that’s why I’m not going very deep in this.

Role for CloudFormation :

Now we need to create a role for CloudFormation which will allow CloudFormation to provision the resources in VPC. Again for making things simple I will be giving VPC full access…

Trusted Entity Type: AWS service
Use Case: CloudFormation
Permission Policies: AmazonVPCFullAccess
Role Name: cf-vpc-role

Create the role & copy it’s ARN because in next step we’re going to use that.

Add extra policy in Lambda Role :

Understand one very crucial thing that Lambda function going to create the CloudFormation Stack. Now that means Lambda will also going to attach the “cf-vpc-role” to the Stack. That means Lambda needs extra permission to pass the role to the Stack, so let’s quickly attach that policy to our Lambda Function.

Go to your “lambda-cf-role” & then under “Permissions” tab click on “Add permissions” → “Create inline policy”, follow the below configurations…

Service : IAM
Actions : PassRole
Resources : Specific & here give “cf-vpc-role” ARN.
Click on Review Policy & give a name like — “lambda-cf-pass-policy”

Finally click on create policy. We are done with the IAM Part. You can see in the screenshots that I created two roles.

Note : I’m using an AWS account that has admin power. But if you are using an IAM account then your organization admin needs to provide you power to create roles in IAM as well as pass those roles to different services like in our case passing the role to the Lambda.

Step 3 : Creating Lambda Function

Next we are going to create one Lambda Function, which will help us to deploy our CloudFormation Template upon having an API call.

Go to AWS Lambda Service & click on create function & follow the below configurations…

Function Name : cf-deploy-function
Runtime : Python3.9
Architecture: x86_64
Execution role : Use an existing role → lambda-cf-role

Finally create the function. One thing you might have understood right now that we created one Lambda Function which has one role attached that can go to CloudFormation & can do everything there, not on other services.

Now it’s time for little bit of Python coding… 😒

I know the pain of Cloud DevOps guys for writing codes but this is very very simple coding…

import boto3def lambda_handler(event, context): thisStackName = str(event[‘StackName’])
 thisClientToken = str(event[‘ClientToken’]) CFClient = boto3.client(‘cloudformation’) response = CFClient.create_stack(
 StackName=thisStackName,
 TemplateURL=’https://raktim-cf-bucket.s3.ap-south-1.amazonaws.com/cf-resources.yaml',
 Capabilities=[‘CAPABILITY_NAMED_IAM’],
 RoleARN=’arn:aws:iam::775370324880:role/cf-vpc-role’,
 ClientRequestToken=thisClientToken,
 OnFailure=’DELETE’
 ) return response

We are using boto3 library to create the stack. Now here one thing to note that when you will connect your Lambda with API Gateway, then API Gateway will send it’s collected data to a variable called “event”. That means when you’re going to call the API, you will also send the StackName which then will be fetched by python using the “event” parameter. You can give any name to this variable but it’s always a good practice to go with industry standards.
If you see the “create_stack” method, then the parameters we are passing is really simple — like what will be your StackName & from where it’s going to fetch the template to deploy the stack. Or, let’s say which IAM role to attach to that stack, what will happen if stack creation fails etc.
Now the big question — What is ClientRequestToken?
First of all it’s just a string value you need to pass which will be unique. You can think this as a metadata that we will provide to our CloudFormation Stack, so that once client sends the request to the API with let’s say a client token value — “client1234” then it will start the creation of the Stack.
Now if again client sends the same request with the same StackName, then as python is a imperative language, it will start provisioning the Stack, but due to the constraints of CloudFormation, your each stack needs their own unique name. So you next call will lead to an error saying “the stack name already exists”. This is where Client token will help you to make you code idempotent. Which means next time if you send request with same StackName & the same client token, it will return 200 status code.

Enough of coding… 🤕 Now let’s continue our journey of Cloud DevOps.

Put the function code in the editor & deploy the function.

Step 4 : Creating API Gateway

This is the last step of this whole setup. I told you at the very beginning that we will have one REST API, where you can just simply send one POST request & it will start provisioning resources on AWS. Let’s start setting up that API, shall we ?

First, go to API Gateway & create one API with the below configurations…

Select REST API & then click on “build” (don’t select the private one)
API name : cf-lambda-api
Endpoint : Edge Optimized

Create the API & next click on “Actions” to setup the POST method.

Click on Create Method
Select POST from drop down menu & you will see it’s asking for the Integration Type → select Lambda Function.
Disable the Lambda proxy integration.
Finally select your lambda function name & save.

Once you save the setup, it will look something like below —

Now we need to deploy our API but before that I want to tell you something. Once you deploy the API, it will be accessible by anyone in the world which can be dangerous if you don’t have anything to secure your API. This is where we need to use API Keys which you can think as a Key to access the API.

Click one “Method Request” & you will have the option of “API Key Required”, change that to true.

Here one more small thing you need to do, that is when the Lambda will receive the response it comes as a JSON value. We want Lambda to receive the exactly same data. But API Gateway has lot of unique features to modify that data before sending to Lambda. That’s why click on “Integration Request” & scroll down to the option “Mapping Templates” → select that to “When there are no templates defined (recommended)”.

Now we can deploy our API. So, again click on “Actions” → “Deploy API”.

Deployment Stage : New Stage
Stage Name : prod

Click on “Deploy” & you will have your Invoke API URL like the below screenshot…

Now remember I just told that we need a key to secure our API. For that we need to setup one Usage Plan. Go to “Usage Plans” tab & click on “Create” & go with the below configurations…

Name : cf-lambda-api-usage-plan
Enable throttling : Rate →10, Burst →5
Enable Quota : 20 requests per Day

Click on next & here you can select the API stage, in our case it’s “prod” stage, remember.

Click on “Add API Stage” → Select API to “cf-lambda-api”, Select Stage to “prod” & then click on that check symbol. Select Next & now you need to select the API key… So simply click on “Create API Key and add to Usage Plan”. Give a name like “cf-lambda-api-key” & save. Finally click on “save”.

Done… 🙌

Time to test :

Are you excited ? If you do this practical, not only you will learn so many things, but also you will understand how frontend applications are creating online labs for us…

But question is how we’re going to test. I told you already that it’s a REST API & most of the time your frontend application access this URL to forward clients requests. But I’m simply using “curl” command from “AWS CloudShell”. You can use any shell you like.

So the structure of the command will look like —

#curl -i -H “x-api-key: TjaV50Z2Q2TTR86y” https://w5560uqso4.execute-api.ap-south-1.amazonaws.com/prod -d ‘{“StackName”:”MyStack”,”ClientToken”:”client1234"}’

Now you might ask me from where to get the API Key, simply go to — “API Keys” tab & select the api key called “cf-lambda-api-key”. You will see it’s showing you the api key. Running the command… 🙌

Hurray…🔥 We got 200 status code output, which means we are able to communicate with the Lambda using API Gateway.

Now here I want to show you 2 demos. First, I am fixing the name of the Stack but changing the Client Token…

Look it’s showing that the “Stack [MyStack1] already exists”. Which means our previous command has successfully created the Stack & also you can’t create 2 different stacks with same name.

Next let’s keep the client token same but change the Stack Name…

Again 200 status code. That means this Client Token helps to make your request idempotent… Looks at the below screenshot where I’m sending multiple requests with same name of the stack & same client access token, but as it’s the nature of idempotence so it’s returning 200 status output always…

And if you’re wondering what it looks like in CloudFormation Console, so here’s a screenshot…

Future Possibilities :

The future is full of possibility, whether you make one tiny change — or a whole invigorating, thrilling, inspiring bunch of them. — Oprah Winfrey

The Next thing in this project you can do it to create various APIs with various Lambda Functions attached for different operations like Delete Stack, Modify Stack etc.
One more amazing thing you can do is to make your CloudFormation template parameterized. Like let’s say you want your client to send the VPC CIDR range. Then just make that value parameterized & using Lambda Code you can easily feed those parameters. And obviously Lambda will receive the parameter value from API Gateway from clients.
Next you can create some API with can go to your stack to fetch certain data which then you can provide to your customers like VPC id, Subnet id. These are just examples, in real world normally we don’t provide there internal ids to clients, but we can provide them certain URL to access their Labs.
Another thing is very crucial is the IAM restrictions. I know that I put simply Full Access on CloudFormation & VPC, but in real world scenarios we always needs to provide least privileges. That’s why these things you need to keep on updating & narrow down the permissions as much as possible.
Lastly, I can suggest one more idea, that is we are using REST API which means it supports various other called like GET, DELETE, PATCH etc. You can use them to build a powerful backend.

Alright, I believe that’s the end of this blog… Little bit big, but interesting… Hopefully you learned something unique. Leave your feedbacks in comments so that I can have a conversation with you.

I keep on writing Blogs on Machine Learning, DevOps Automation, Cloud Computing, Big Data etc. So, if you want to read future blogs of mine, follow me on Medium. You can also ping me on LinkedIn, checkout my LinkedIn profile below…

Raktim Midya - Freelancer & Technical Consultant - Self-employed | LinkedIn

View Raktim Midya's profile on LinkedIn, the world's largest professional community. Raktim has 4 jobs listed on their…

www.linkedin.com

Thanks Everyone for reading. That’s all… Signing Off… 😊