As DevOps Engineers, we are in the world of “automation.” From writing scripts, whether in BASH or Python, to creating CI/CD pipelines to automate the whole software development process to deployment, CI/CD (Continuous Integration and continuous development) are essential parts of the software development process and GitHub Actions became the most popular tool in automating these process.

In this blog, we will explore GitHub Actions where we will see the effective caching strategy to follow in multiple applications because caching plays a very important role in making our pipeline execution faster and this improves the software delivery processes. Before, moving forward check out my previous blogs:

Building a Serverless Web Application with AWS Lambda, API Gateway, DynamoDB, S3

End-to-End DevOps for a Golang Web App: Docker, EKS, AWS CI/CD

Deploying Your Website on AWS S3 with Terraform

Learn How to Deploy Scalable 3-Tier Applications with AWS ECS

Introduction to GitHub Actions and Dependency Caching

Nowadays, most companies are migrating towards GitHub Actions from Jenkins because of security issues and Jenkins is self-hosted for most of the companies so we need to maintain the servers at our data center. GitHub Actions is another CI/CD platform where developers can automate their workflows directly within the GitHub repositories.

In GitHub Actions many features enhance the security, and fill some of the bottlenecks in the pipeline but when it comes to dependency caching many users often face performance bottlenecks, especially when installing dependencies or repeating tasks across multiple jobs. Dependency Caching is a technique that should be followed to decrease the pipeline build time which will also decrease the resource consumption. By caching the artifacts, and libraries we can use these as artifacts in the other builds also.

In GitHub Actions, it has the built-in mechanism through the actions/cache actions, where users can cache the dependencies by using this action in the pipeline.

Best Practices for Implementing Dependency Caching

1. Use Effective Cache Keys: Hash key files like package.json or requirements.txt to keep the cache relevant.
2. Avoid Cache Conflicts: Use unique keys for different environments or branches to prevent mix-ups.
3. Implement Cache Invalidation: Change cache keys when dependencies update.
4. Manage Cache Size: Big caches can slow things down, so stick to caching only the essential files.

How Caching Works in GitHub Actions

Now, we will implement the caching action in the NodeJs application, where it will cache the Docker image layers by using the gha cache, i.e. GitHub Actions cache, which is by default. Bit GitHub Actions cache provides only caching storage of 10GB, so most corporate environments use the remote shared cache to store artifacts on Nexus or AWS CodeArtifacts, and for Docker Builds, they can use Harbor or DockerHub.

If I had to go with like I want to store node_modules for NodeJs or requirements.txt (venv) for Python we can use GitHub Actions Cache or S3-compatible storage (Minio). For now, let’s implement docker caching layers on the GitHub Actions cache and check the build speed.

GitHub Code: github.com/amitmaurya07/DevSecOps-GHA

.github/workflows/pipeline.yml:

name : NodeJs Application and Caching
on:
    workflow_dispatch:
    push:
        branches:
            - master
jobs:
    docker-build:
        runs-on: ubuntu-latest
        steps:
            - name: Checkout Code
              uses: actions/checkout@v3

            - name: Cache Docker layers
              uses: actions/cache@v3
              with:
                path: /tmp/.buildx-cache
                key: ${{ runner.os }}-buildx-${{ hashFiles('**/Dockerfile') }}
                restore-keys: |
                  ${{ runner.os }}-buildx-

            - name: Login to docker-hub
              uses: docker/login-action@v1
              with:
                username: ${{ vars.DOCKER_USERNAME }}
                password: ${{ secrets.docker_pat }}

            - name: Set up Docker Buildx
              uses: docker/setup-buildx-action@v3

            - name: Build and Push Image
              uses: docker/build-push-action@v6
              with:
                context: .
                push: true
                tags: "amaurya07/devsecops_app:${{ github.ref_name }}"
                cache-from: type=local,src=/tmp/.buildx-cache
                cache-to: type=local,dest=/tmp/.buildx-cache,new=true

In the first part, we had given the name of the pipeline “NodeJs Application and Caching“ then we had set the trigger action by defining on when the pipeline will trigger here the pipeline will trigger on push from the master branch and workflow_dispatch means that we can trigger manually pipeline when we want. It means when anything is pushed on the master branch the pipeline is executed automatically.

jobs:
    docker-build:
        runs-on: ubuntu-latest
        steps:
            - name: Checkout Code
              uses: actions/checkout@v3

            - name: Cache Docker layers
              uses: actions/cache@v3
              with:
                path: /tmp/.buildx-cache
                key: ${{ runner.os }}-buildx-${{ hashFiles('**/Dockerfile') }}
                restore-keys: |
                  ${{ runner.os }}-buildx-

            - name: Login to docker-hub
              uses: docker/login-action@v1
              with:
                username: ${{ vars.DOCKER_USERNAME }}
                password: ${{ secrets.docker_pat }}

            - name: Set up Docker Buildx
              uses: docker/setup-buildx-action@v3

            - name: Build and Push Image
              uses: docker/build-push-action@v6
              with:
                context: .
                push: true
                tags: "amaurya07/devsecops_app:${{ github.ref_name }}"
                cache-from: type=local,src=/tmp/.buildx-cache
                cache-to: type=local,dest=/tmp/.buildx-cache,new=true

In the second part, we defined the job after the trigger i.e. docker-build then we defined the runs-on which means where the steps in the job that are defined are executed so we defined ubuntu-latest which is provided by GitHub Actions runners. In the Actions tab of the repository navigate to the Runners section and check the name of the runner in Standard GitHub-hosted runners.

Now come, the steps that are defined in a job there are 4 steps in the jobs (Checkout Code, Login to DockerHub, Docker Buildx, then Docker Build and Push)

1. Checkout Code: Actions is actions/checkout@v3 which will checkout the source code from the repository.
2. Cache Docker Layers: Actions is actions/cache@v3 where it will cache the docker layers in locally in the /tmp directory where the key is ${{ runner.os }}-buildx-${{ hashFiles('**/Dockerfile') }} that means the compute the hashes of content inside the Dockerfile whenever the Dockerfile changes then a new cache will be used.
3. Login to DockerHub: Actions is docker/login-action@v1 where it uses the variable and secrets that are in the repository for login to dockerhub to push the image.
4. Docker Buildx: Actions is docker/setup-buildx-action@v3 where we need to set up the docker build as we are using the build and push action in the next step and it is also used to cache the docker image layers and it can build multiple stages or architectures in parallel, reducing build time for complex Dockerfiles.
5. Docker Build and Push: In this step set the context to the present working directory where the Dockerfile is present then set the push to true with tags then the cache-from has the type=local where the src is set to the /tmp directory where the cache is stored in the earlier step and in cache-to is also set to type=local which has the new=true means forces the creation of new caches rather than updating the existing caches.

As we can see in the next build it was importing the cache from the gha and all the layers are cached in the next build.

Caching Dependency Examples for Popular Languages and Tools

Node.js

- uses: actions/cache@v3
  with:
    path: node_modules
    key: node-${{ hashFiles('package-lock.json') }}

Python

- uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: pip-${{ hashFiles('requirements.txt') }}

Java (Maven)

- uses: actions/cache@v3
  with:
    path: ~/.m2/repository
    key: maven-${{ hashFiles('pom.xml') }}

Conclusion: The Future of Fast CI/CD with GitHub Actions

Dependency caching is a great way to speed up your CI/CD pipelines with GitHub Actions. By using the right caching strategies, you can cut down build times, get faster feedback, and save on resources. Give caching a try today to create workflows that are more efficient and scalable.

GitHub Code: https://github.com/amitmaurya07/DevSecOps-GHA

Twitter : x.com/amitmau07

LinkedIn: linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

Support My Work ☕️
If you found this blog helpful and want to support my work, consider buying me a coffee! Your support keeps me motivated to create more content like this.

Thank you for your support!

Boost GitHub Actions Speed with Effective Dependency Caching was originally published in Towards AWS on Medium, where people are continuing the conversation by highlighting and responding to this story.

In today’s software development world, many programming languages are used for making software, solving real-world problems, creating games, and more. As DevOps engineers, we enter the world of automation, always creating CI/CD pipelines to deploy our code faster and more efficiently. When the word “automation“ comes up in the IT world, Python has proven to be the best way to solve complex problems efficiently. By writing Python programs, IT engineers’ and DevOps engineers’ lives become easier, as they just write the scripts and the output appears in front of them in just a few seconds.

Python is at the heart of DevOps workflows used to automate tasks such as deleting unused EBS volumes in AWS, managing infrastructure, or filtering out the CRITICAL or ERROR keywords from log files. In this blog, we’ll explore why Python is an indispensable skill for DevOps engineers, its key use cases, and how you can start your Python journey.

Role of DevOps in Modern IT

DevOps creates a bridge between the Development team and the Operations team. It is a culture or practice aimed at improving the software development lifecycle and contributing to business growth. As a DevOps engineer, automation is crucial for reducing manual work, minimizing errors, and boosting efficiency. This is where Python excels — its simplicity and power make it the preferred language for automating repetitive tasks and building strong workflows.

Why Python is Popular in DevOps

When we talk about scripting or automating tasks we generally use BASH scripting which is most commonly used in Linux environments but when it comes to complex automation most DevOps engineers follow Python as the scripting language as it has good community support, and extensive libraries.

1. Easy to Use
2. Cross-Platform Support
3. A rich ecosystem of libraries like boto3 for AWS, paramiko for SSH, and kubernetes for container orchestration.
4. Integration with other DevOps tools

Python Use-Cases in DevOps

1. Automating Repetitive Tasks

• Setting up servers automatically
• Making it easier to deploy apps
• Using Python scripts to check out logs

2. Infrastructure as Code (IaC)

• Managing infrastructure with code
• Adding more features with Python scripts
• Working with IaC tools like Terraform and Pulumi

3. Hooking into CI/CD Pipelines

• Customizing how builds and deployments work
• Handling API calls
• Creating plugins for tools like Jenkins and GitLab CI/CD

4. Cloud Automation

• Automating tasks in AWS, Azure, and GCP
• Using SDKs like boto3, azure-sdk, and google-cloud-sdk
• Doing things like provisioning resources, scaling, and monitoring

5. Easy Monitoring and Logging

• Collecting metrics with Prometheus-client
• Building dashboards using Flask
• Checking out logs with Loguru
• Integrating with Grafana and Elasticsearch

6. Containerization and Orchestration

• Managing Docker containers with Python APIs
• Automating Kubernetes operations
• Working with Kubernetes custom resources

7. Big Ecosystem and Community Support

• Libraries for API calls: requests
• Testing frameworks: pytest
• An active community for help and updates

8. Cross-Platform Compatibility

• Running scripts easily on Linux, Windows, and macOS
• Great for different infrastructure setups

9. Easy to Learn and Read

• Simple syntax for quick learning
• Good for engineers without a programming background
• Helps junior engineers contribute effectively

Python Libraries Every DevOps Engineer Should Know

• boto3: Easily manage AWS services with code.
• paramiko: Automate SSH and manage remote servers effortlessly.
• kubernetes: Work smoothly with Kubernetes clusters.
• os and subprocess: Perform system tasks and run shell commands with ease.
• fabric: Simplify tasks on remote servers.
• requests: Make HTTP requests for API interactions a breeze.

Conclusion

Python isn’t just a programming language — it’s like a best friend for DevOps engineers. Its flexibility, simplicity, and strong libraries make it a must-have tool for automating tasks, managing infrastructure, and improving workflows. Whether you’re just beginning or aiming to boost your skills, learning Python can elevate your DevOps career.

Resources to follow to learn Python

1. Automate the Boring Stuff with Python by Al Sweigart
2. Python Full Course Video By BroCode
3. Python.org Official Documentation

In today’s computing industry these two terms you have heard a lot or you have heard the term how to achieve zero downtime which is very crucial in application and business needs. In this article, we will be learning about High Availability, and Scalability and how we can achieve this in Cloud Modern architectures (AWS).

By keeping our system highly available and scalable our business would grow more and our end-users would be more happy. These two terms are most asked questions on System Design Interview by big tech companies (FAANG) so it is good to know these two terms in detail and in AWS cloud perspective.

Before heading to Introduction section, do read my latest blogs:

Building a Serverless Web Application with AWS Lambda, API Gateway, DynamoDB, S3

End-to-End DevOps for a Golang Web App: Docker, EKS, AWS CI/CD

Deploying Your Website on AWS S3 with Terraform

Learn How to Deploy Scalable 3-Tier Applications with AWS ECS

High Availability

As a DevOps Engineer, our end task is not deploy the only application to the Kubernetes cluster, or on AWS services but to ensure that the system is available every time to the end-user. So, to keep the system available we need to make the system highly available.

To measure high availability we need to keep the system’s percentage in five 9’s means if the system’s or application has 90% then the system is down for 36.5 days which is very bad for the end-user or if the system has 99.999% then system will be down for 6 minutes a year which has become standard for achieving the HA for most the companies. Remember any system would not be 100% available.

To understand this in a better way, let’s take an example of grocery store which is running 24/7. After some-day the grocery store needs an maintenance so he shuts down the grocery store which means that the grocery store would not be available for few days which results in business loss. So to keep the business running every time, what store owner did he opens another grocery store in nearby so that if any customer comes he/she can go to another grocery store during maintenance of another grocery store which keeps the business running smoothly and by this whenever any sales comes like Black Friday the crowd is distributed to 2 stores.

By this it ensures the business continues to operate without interruptions, which is the essence of High Availability.

Now, let’s understand this in terms of AWS cloud perspective.

This picture is AWS architecture, which depicts the High Availability on AWS.

Let’s understand the previous example of grocery store in AWS terms.

1. Multiple Stores (Redundancy) → AWS Availability Zones (AZs)

• Like operating the grocery stores in multiple locations keeps the business running smoothly. In AWS we had to deploy the infrastructure in multiple AZ’s (Availability Zones).
• If one AZ fails in AWS like our one store goes in maintenance the other store is operating likewise the other AZ will be working to handle customer’s workloads.

2. Spreading Demand Across Stores (Load Balancing) → Elastic Load Balancer (ELB)

Like the festival came Black Friday so the crowd is distributed on 2 stores meanwhile in AWS we use Load Balancing to distribute the traffic efficiently so that one server has not the most requests.

3. Shared Inventory (Centralized Data) → Shared Databases or S3

In grocery store it uses the centralized data to keep the stocks updated for every store. Likewise in AWS we use database to keep our data safe, secure and highly available by deploying AWS RDS with Multi-AZ feature or DynamoDB.

4. Restocking (Scaling) → Auto Scaling

Just like when any festival season came users increased to buy items which results in huge spikes in users. In AWS to manage the traffic to scale efficiently Auto Scaling is the feature that scale the workload by setting Auto Scaling Groups on AWS so that when the users increases the workloads increased when the user decreases the workload comes to an normal state which also results in cost optimization.

5. Delivering Popular Items Globally (Global Content Delivery) → CloudFront

CloudFront is the feature in AWS which enables the static content to make available to every user that is distributed globally. With the help of CDN (CloudFront) it ensures there no downtime to end user to accessing any static website.

Scalability

Now, let’s talk about scalability. It’s all about how a system or an application can handle more or fewer resources without losing performance, reliability, or availability. Basically, it’s about keeping things running smoothly no matter how much you throw at it.

There are 3 types of scalability:

1. Vertical Scalability:

Vertical Scalability means scaling UP and scaling DOWN which means increasing the capacity of the single resource.

2. Horizontal Scalability:

In this type of scalability, means scale IN and scale OUT means to increase the resources, multiply the resource adding more servers.

3. Elastic Scalability:

In AWS by defining AutoScaling we can achieve the elastic scalability to dynamically adjust the resources based on real-time demands.

Now, let’s understand the grocery store example in terms of scalability in AWS perspective.

1. Adding More Cashiers (Vertical Scaling):

Like in festive season in High Availability we had increased the stores which balances the crowd. Now, in terms of scalability (Vertical Scaling) we need to increase the capacity to handle more people in the stores. So, we increased cashiers to handle vertical scaling but in terms of AWS in EC2 service we increase the capacity from t2.micro to m5.large.

2. Opening More Stores (Horizontal Scaling):

To handle more people not to make the crowd overloaded in one store we opened the another store to handle the crowd efficiently. This what makes horizontal scaling. In AWS we can increase the EC2 instances by setting ASG (Auto Scaling Groups) to increase the workloads based on traffic demands.

3. Temporary Staff (Elastic Scaling):

During the busy season, the store hires extra workers and opens another locations. Once the festival season ends, they return to their normal staff levels. In AWS this also done by Auto Scaling which dynamically adds resources based on real-time that saves the cost.

4. Scaling Data and Storage

1. Now, to prevent running out of stock for grocery stores we had to keep the stocks available 24/7 for this in AWS that manages the scaling storage and databases:
2. Amazon RDS with Read Replicas: It add the replicas to distribute database read traffic with Multi-AZ feature.
3. Amazon S3: Automatically scales to store unlimited amounts of data.

Conclusion

In this article, we learned about High Availability and scalability how to achieve this in AWS cloud. These two are essential pillars in building the resilient system and fault-tolerance that ensures the minimal downtime or zero downtime. In upcoming posts, we will delve into the creation of various AWS services, DevOps tools, CNCF tools and their automation with Terraform and also do projects with DevSecOps approach. Stay tuned for the next blog!

GitHub Code : github.com/amitmaurya07

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

Achieving High Availability and Scalability in AWS: Best Practices for Modern Cloud Architectures was originally published in Towards AWS on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we will cover important Dockerfile Instructions that should be included in Dockerfile to make the container secure, follows the best practices and optimized for production.

Before heading to Introduction section, do read my latest blogs:

Building a Serverless Web Application with AWS Lambda, API Gateway, DynamoDB, S3

End-to-End DevOps for a Golang Web App: Docker, EKS, AWS CI/CD

Deploying Your Website on AWS S3 with Terraform

Learn How to Deploy Scalable 3-Tier Applications with AWS ECS

FROM Instruction

FROM python:3.10-slim

Always use smaller size images to make the image more optimized.

EXPOSE Instruction

EXPOSE 8080

Include EXPOSE instruction to document the port number on which container listens on.

USER Instruction

USER nonroot

Always avoid to run the container as non-root user because of security reasons.

COPY & ADD Instruction

COPY requirements.txt /app/
ADD app.tar.gz /app/

Always prefer COPY instruction over ADD instruction to copy the files into the image. Use ADD instruction only when you need to extract the files.

HEALTHCHECK Instruction

HEALTHCHECK --interval=30s CMD curl -f http://localhost/ || exit 1

Include HEALTHCHECK instruction always to test the container is working properly or not.

RUN Instruction

RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

Always use single RUN Instruction to minimize the number of layers so that our build will become faster.

LABEL Instruction

LABEL maintainer="Amit Maurya"
LABEL version="1.0"

In an organisation there are multiple developers, devops engineers are working there so to check who has made the Dockerfile or to contact you, always use LABEL Instruction for documentation, versioning.

WORKDIR Instruction

WORKDIR /app

Always use WORKDIR to not to use RUN cd command to get inside the directory. This instruction will set the working directory in the image and copies all the files into this working directory only.

ARG Instruction

ARG APP_VERSION=1.0
ARG API_KEY
ENV API_KEY=${API_KEY}

Include ARG instruction to create build-time variables so that when the image is build we can pass the value at build time. It is used for secrets and versioning.

docker build --build-arg APP_VERSION=2.0 .
docker build --build-arg API_KEY=your_api_key .

CMD Instruction

CMD ["python", "app.py"]

Use CMD Instruction to specify the default command to run when the container starts.

ENTRYPOINT Instruction

ENTRYPOINT ["sh", "-c"]

This instruction is use to configure the container to run as executable. CMD command can be override by ENTRYPOINT instruction.

Conclusion

In upcoming posts, we will delve into the creation of various AWS services, DevOps tools, CNCF tools and their automation with Terraform and also do projects with DevSecOps approach. Stay tuned for the next blog!

GitHub Code : github.com/amitmaurya07

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

Essential Dockerfile Instructions: Best Practices for Optimized Containers was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article we are going to deploy web application on AWS in serverless way means we don’t have to manage servers at all, no scaling of servers nothing. In today’s world, most of the business need serverless architecture where they should mainly focus on building the businesses not managing the time in managing the servers or any kind of downtime.

Before heading to Introduction section, do read my latest blogs:

End-to-End DevOps for a Golang Web App: Docker, EKS, AWS CI/CD

Step-by-Step Guide: How to Create and Mount AWS EFS (Elastic File System)

Deploying Your Website on AWS S3 with Terraform

Learn How to Deploy Scalable 3-Tier Applications with AWS ECS

Introduction to Serverless Architecture

Serverless means we just don’t have to manage the servers, allowing developers to write the code can deploy the code easily without managing the servers. Serverless is FaaS service means Functions As a Service by this serverless computing is highly available, efficient and makes it scalable.

Serverless architectures are based on EDA (Event Driven Architectures) that means it will triggered by some events like HTTP requests, file uploads or on schedule basis also.

As serverless is Pay As You go model that makes the cost-effective which means we are paying for the compute power that we use.

PreRequisites and AWS Services

1. AWS Account
2. IAM Role
3. AWS Lambda for writing functions or code where we will deploy
4. AWS API Gateway to create a RestFul API
5. NOSQL Database i.e DynamoDB

Deployment of Website on AWS

Before heading to AWS let me show you how the website look, and what we are going to achieve.

At the end of our blog, we had deployed this website on AWS and access this website with S3 Static Website Endpoint which means we are hosting this website on S3 (index.html) and when we fill the details it will show “Data Submitted Successfully“ and add the record into DynamoDB table.

GitHub Code : https://github.com/amitmaurya07/Serverless-AWS

IAM Role

Create the IAM role that will communicate with required services. Navigate to IAM, click on Create Role.

1. Now, select AWS service and choose Lambda under Usecase then give the required permissions for API Gateway, DynamoDB, S3 Bucket then click on Next and name the role ServerlessWeb and click on Create Role.
2. AmazonAPIGatewayInvokeFullAccess
3. AmazonDynamoDBFullAccess
4. AmazonS3FullAccess

AWS Lambda

In AWS, we had EC2 instances which are virtual servers that has limited RAM and CPU where the EC2 instances are continuously running if they are stopped also so we are paying also this, and to scale EC2 instances we create Autoscaling Groups .

But in AWS Lambda they are virtual function which means we don’t need to manage the servers, don’t need to provision the servers and they run ON-DEMAND where scaling is automated.

It supports multiple programming languages like Node.js, Python, Golang, Java, C# and Ruby. On AWS Lambda it follows Pay AS You go model. In Free-tier AWS account it provides 1 million free requests.

Navigate to AWS Management Console and search Lambda

Click on Create Function name Serverless-Web-App, choose Runtime Python 3.13 then change the default execution role. Select Use an existing role which you had created earlier ServerlessWeb then click on Create on Create Function.

Don’t worry we will see the API Gateway also when we create API Gateway. Now, paste the code and Deploy the changes.

import json
import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('serverless-web-app')  # Replace with your DynamoDB table name

def lambda_handler(event, context):
    print(json.dumps(event)) 

    if event['httpMethod'] == 'OPTIONS':
        return {
            'statusCode': 200,
            'headers': {
                'Access-Control-Allow-Origin': '*',  # or your frontend domain
                'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,Authorization,X-Api-Key',
                'Access-Control-Allow-Methods': 'POST,OPTIONS'  # Allow POST and OPTIONS methods
            },
            'body': json.dumps('CORS Preflight Request handled successfully')
        }

    # POST Request Handling
    if event['httpMethod'] == 'POST':
        # Check if the body is present in the event
        if 'body' not in event:
            return {
                'statusCode': 400,
                'headers': {
                    'Access-Control-Allow-Origin': '*'  # Allow all domains or specify one
                },
                'body': json.dumps('Invalid request: No body in the request')
            }

        try:
            body = json.loads(event['body'])
            name = body.get('name')
            email = body.get('email')

            # Ensure both 'name' and 'email' are provided
            if not name or not email:
                return {
                    'statusCode': 400,
                    'headers': {
                        'Access-Control-Allow-Origin': '*'  # Allow all domains or specify one
                    },
                    'body': json.dumps('Invalid input: Name and Email are required')
                }

            # Save to DynamoDB
            table.put_item(
                Item={
                    'email': email,
                    'name': name
                }
            )

            return {
                'statusCode': 200,
                'headers': {
                    'Access-Control-Allow-Origin': '*'  # Allow all domains or specify one
                },
                'body': json.dumps('Data stored successfully!')
            }

        except json.JSONDecodeError:
            return {
                'statusCode': 400,
                'headers': {
                    'Access-Control-Allow-Origin': '*'
                },
                'body': json.dumps('Invalid JSON format')
            }
        except Exception as e:
            return {
                'statusCode': 500,
                'headers': {
                    'Access-Control-Allow-Origin': '*'
                },
                'body': json.dumps(f"Internal Server Error: {str(e)}")
            }

In this code we have to replace our DynamoDB table name in 5th line. After deploying the changes go to API Gateway

API Gateway

API Gateway are serverless API’s that provide Restful API’s to the clients and after integrating with Lambda it proxy the requests to our Lambda functions.

It has 3 endpoint types:

1. Regional — It is for the same region.
2. Private — It can be accessed from private VPC with the help of VPC Endpoint.
3. Edge-Optimized (default) — To improve latency requests are routed through Cloudfront edge locations.

Now, navigate to AWS Management Console and search for API Gateway.

1. Click on Create API and choose the API type REST API.
2. Click on Build and name it Web-App , select Regional Endpoint Type then click on Create API.

Now, create the method POST and OPTIONS. Click on Web-App then click on Create Method.

Select Method Type POST, Integration type Lambda enable the Lambda Proxy Integration by which entire HTTP request will be forwarded to Lambda function. Choose the Lambda function ARN which we created earlier Serverless-Web-App then click on Create Method.

Now, we have to create another method type OPTIONS that will handle CORS error and preflight requests. Select MOCK Integration Type and choose the same Lambda function Serverless-Web-App.

1. Now, click on OPTIONS method and add method Method Response under 200 Status Code. Click on Create Response under Header Name add these headers.
2. Access-Control-Allow-Headers
3. Access-Control-Allow-Methods
4. Access-Control-Allow-Origin
5. Then click on Save.
6. Go to Integration Response under OPTIONS method add the value to the Response headers.
7. Access-Control-Allow-Headers 'Content-Type, Authorization'
8. Access-Control-Allow-Methods 'POST, GET, OPTIONS'
9. Access-Control-Allow-Origin '*'

1. Click on Save.

2. Click on POST method and add same Method Response headers.

Now, click on deploy the API where you need to create New Stage to save the changes.

Scroll down, you will get the Invoke URL and add this URL in index.html file.

DynamoDB

AWS offers DynamoDB which is NoSQL database which is highly scalable, and fully managed serverless database solution that can handle millions of requests per second, offers low latency which stores and retrieve structured data.

Navigate to AWS Management Console and search DynamoDB

Click on Create Table name web-app , write partition key email as we are only storing email that wants to be unique ID then click on Create Table.

After this, copy the table name web-app and paste in Lambda function where we had defined.

After doing changes, paste the same to Lambda function and deploy the changes in Lambda.

Host Static Website on S3

Now, we are going to host our static website index.html on S3 bucket which is serverless.

If you want to host static website on S3 with Terraform Modules you can refer to this article.

Deploying Your Website on AWS S3 with Terraform

Navigate to AWS Management Console and search S3. Remember we will be creating the bucket in Mumbai (ap-south-1)

Create the bucket with your preferred name as name has to be unique across Global. While creating bucket uncheck the “Block all Public Access” then click on Create bucket and upload index.html file in the bucket.

Scroll down in Properties section of S3 bucket and enable the Static Website Hosting type index document index.html

You will receive endpoint to open the website but it will give you error because you hadn’t added the Permissions. Navigate to Permissions tab in S3 bucket and add bucket policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::serverlesswebapp07/*"
        }
    ]
}

In Resource change the bucket name to your bucket name.

Now, when you refresh the page you will access the website. So, we hosted the static website successfully on S3 bucket.

Now, let’s check our API, Lambda and DynamoDB is perfectly working or not.

As we can see data is submitted successfully our API is getting 200 status code. Let’s check also our DynamoDB table recorded the email or not.

As we can see our email and Name is there in database. We can check also with PostMan, for this select the POST method response and paste the API Gateway URL and in raw select JSON body.

Conclusion

In this article, we explored the deployment of our website on AWS using a serverless approach. In upcoming posts, we will delve into the creation of various AWS services and their automation with Terraform. Stay tuned for the next blog!

GitHub Code : https://github.com/amitmaurya07/Serverless-AWS

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

Vantage — Cloud Cost Observability and Optimization

Vantage is a modern cloud cost management platform that helps companies manage costs across AWS, Azure, Datadog, Google Cloud, and 10 other providers. Create a free account in minutes to get an automated savings health check.

Building a Serverless Web Application with AWS Lambda, API Gateway, DynamoDB, S3 was originally published in Towards AWS on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we are going to build a Lambda Function in which there is Python code that will generate the AWS Cost Report of your specific region and send it to your email through SNS Topic. We had automated this Lambda function by creating CloudWatch that will execute on schedule basis on 1st day of every month.

Before heading to Introduction section, do read my latest blogs:

End-to-End DevOps for a Golang Web App: Docker, EKS, AWS CI/CD

Deploying Your Website on AWS S3 with Terraform

Learn How to Deploy Scalable 3-Tier Applications with AWS ECS

Overview of AWS Cost Management

In today’s modern computing world, most of the companies or startups are switching their traditional servers to virtual servers to reduce their server setup cost and by AWS it reduces upto 50%.

But as a DevOps Engineer or Cloud Engineer we had to monitor the costs of AWS resources also how to use efficiently AWS resources we had to keep in mind so that unwanted costs can be cut.

To see the AWS Resources Costs, navigate to Billing and Cost Management Dashboard (https://us-east-1.console.aws.amazon.com/billing/home#/bills)

Prerequisites

1. Setup IAM (Identity & Access Management) Role to communicate with other AWS services.
2. Write Python Code for specific region that will fetch the cost and sent the report to Email through SNS (Simple Notification Service).
3. Lambda Function
4. SNS (Simple Notification Service) Topic
5. CloudWatch Alarm

Implementation

1. Create an IAM role for Lambda Function that will execute the SNS service. Navigate to IAM then select Roles

1. Click on Create Role then select Lambda as UseCase then Next. Now Add the required Permissions.
2. AmazonSNSFullAccess - SNS Full Access
3. AWSCostAndUsageReportAutomationPolicy - Cost Explorer Access
4. CloudWatchEventsFullAccess - CloudWatch Events Access to create Rules
5. Then, add Inline Policy also for GetCostUsage name it cost-optimization.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ce:GetCostAndUsage",
                "ce:GetCostForecast",
                "ce:GetDimensionValues",
                "ce:GetUsageForecast"
            ],
            "Resource": "*"
        }
    ]
}

1. Now, IAM role is created setup the AWS Lambda function to execute our Python code.
2. AWS Lambda is serverless compute service that enables us to run our code without managing servers. It automatically scales up and down the servers that will be cost-efficient. It follow Event-Driven architecture where we can trigger Lambda by multiple events by integrating CloudWatch or other AWS services.

import boto3
import datetime
from botocore.exceptions import ClientError

# Initialize AWS Cost Explorer and SNS clients
cost_explorer = boto3.client('ce')
sns = boto3.client('sns', region_name="ap-south-1")

# SNS topic ARN
sns_topic_arn = 'arn:aws:sns:ap-south-1:880849992790:CostOptimization'  # Replace with your actual SNS topic ARN

# Define the Lambda function
def lambda_handler(event, context):
    # Get current date and start of the month
    today = datetime.date.today()
    start_date = today.replace(day=1)
    end_date = today

    try:
        # Query AWS Cost Explorer for cost breakdown per service in ap-south-1 region
        ap_south_1_response = cost_explorer.get_cost_and_usage(
            TimePeriod={
                'Start': start_date.strftime('%Y-%m-%d'),
                'End': end_date.strftime('%Y-%m-%d')
            },
            Granularity='MONTHLY',
            Metrics=['UnblendedCost'],
            GroupBy=[{
                'Type': 'DIMENSION',
                'Key': 'SERVICE'
            }],
            Filter={
                'Dimensions': {
                    'Key': 'REGION',
                    'Values': ['ap-south-1']
                }
            }
        )

        # Query AWS Cost Explorer for cost breakdown per service for all regions
        all_regions_response = cost_explorer.get_cost_and_usage(
            TimePeriod={
                'Start': start_date.strftime('%Y-%m-%d'),
                'End': end_date.strftime('%Y-%m-%d')
            },
            Granularity='MONTHLY',
            Metrics=['UnblendedCost'],
            GroupBy=[{
                'Type': 'DIMENSION',
                'Key': 'SERVICE'
            }]
        )

        # Prepare the breakdown of costs for ap-south-1
        ap_south_1_total_cost = 0
        ap_south_1_details = "Cost Breakdown for the ap-south-1 Region for the Current Month:\n\n"
        for result in ap_south_1_response['ResultsByTime']:
            for group in result['Groups']:
                service_name = group['Keys'][0]
                cost_amount = float(group['Metrics']['UnblendedCost']['Amount'])
                ap_south_1_total_cost += cost_amount
                ap_south_1_details += f"{service_name}: ${cost_amount:.2f}\n"
        
        ap_south_1_details += f"\nTotal Cost for ap-south-1: ${ap_south_1_total_cost:.2f}\n\n"

        # Prepare the breakdown of costs for all regions
        all_regions_total_cost = 0
        all_regions_details = "Cost Breakdown for All Regions for the Current Month:\n\n"
        for result in all_regions_response['ResultsByTime']:
            for group in result['Groups']:
                service_name = group['Keys'][0]
                cost_amount = float(group['Metrics']['UnblendedCost']['Amount'])
                all_regions_total_cost += cost_amount
                all_regions_details += f"{service_name}: ${cost_amount:.2f}\n"
        
        all_regions_details += f"\nTotal Cost for All Regions: ${all_regions_total_cost:.2f}"

        # Combine both reports (ap-south-1 and all regions) into one message
        full_report = ap_south_1_details + all_regions_details

        # Publish the cost report to SNS
        sns.publish(
            TopicArn=sns_topic_arn,
            Subject='AWS Cost Breakdown Report',
            Message=full_report
        )
        return {
            'statusCode': 200,
            'body': 'Cost report for ap-south-1 and all regions sent successfully via SNS!'
        }

    except ClientError as e:
        print(f"Error: {e}")
        return {
            'statusCode': 500,
            'body': f"Failed to send cost report: {e}"
        }

Search AWS Lambda in AWS Management Console Search Bar

Click on Create Function and select Author From Scratch, name the function CostOptimizationFunction , select Python 3.x Runtime then Open Permissions and select Use Existing Role and select the created IAM role and at last click on Create Function.

1. Scroll down and, paste the Source Code in Editor then click on Tests to test the program is working properly or not.
2. But, before executing the program navigate to AWS SNS (Simple Notification Service) to create the topic and subscription so that the report will be sent to Email.

Name the topic CostOptimization and create the Subscription where you will verify the Email.

Click on Create Subscription and select the protocol “Email“

1. Now, go to your Email to verify the subsrciption endpoint. Click on the link that is provided in the Email it will be verified.
2. Now, copy the SNS Topic ARN and paste it into Lambda function in variable sns_topic_arn and write the region from which you want the cost like I want the cost of region Mumbai I had written the region (ap-south-1).
3. Create the Tests name CostOptimizationFunc then click on Test. It will deploy the template Hello World to deploy the changes of our Python program click on Deploy it will give the output in JSON.

As we can see 200 status code that means our program is successfully executed. Now check the Email Inbox you will recieve the email of your Cost Report of your specific region and all regions.

1. As we can see in the Security Details it mailed by and signed by AWS.
2. After all this, we will automate this so we don’t run the lambda function manually we will integrate CloudWatch with Lambda on schedule basis of every 1st day of month so that lambda will be triggered automatically and cost report sent to Email.
3. Navigate to AWS Cloudwatch in AWS Management Console Search Bar.

1. Now, in the left hand side we have to create Rules under Events section. Click on Rules, then select default EventBus name it CostOptimizationRule then select the Rule Type Schedule.
2. Click on Continue to Create Rule fill the CRON expression (0 0 1 * ? *)
3. and select Local Time Zone. You will see the Next trigger dates means the Lambda will trigger on which date.

Click on Next and select AWS service type on which service you want to trigger. In our case it is Lambda function, so select Lambda then select the Lambda Function ARN.

Click on Next and create the Rule.

Conclusion

In this article, we had learnt how to create Lambda function, CloudWatch Rules to automatically trigger then generate the cost report and sent it to the Email. Meanwhile in the upcoming blogs you will see a lot of AWS Services creation and automating it with terraform. Stay tuned for the next blog !!!

GitHub Code : https://github.com/amitmaurya07

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

Vantage — Cloud Cost Observability and Optimization

Vantage is a modern cloud cost management platform that helps companies manage costs across AWS, Azure, Datadog, Google Cloud, and 10 other providers. Create a free account in minutes to get an automated savings health check.

Automating AWS Cost Reporting with Lambda and SNS was originally published in Towards AWS on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we are going to learn how to deploy Golang application end to end with on AWS EKS (Elastic Kubernetes Service) with AWS CI/CD. This will enhance your skills and you can add this project in your resume.

Before heading to Introduction section, do read my latest blogs:

Step-by-Step Guide: How to Create and Mount AWS EFS (Elastic File System)

Deploying Your Website on AWS S3 with Terraform

Learn How to Deploy Scalable 3-Tier Applications with AWS ECS

Introduction

Starting from the Project understanding we have a simple Golang application that will shorten the URL it has the main.go file and one index.html which will serve as frontend.

So, this will look like this which is running locally on http://localhost:8080 To run locally first we need to build go.mod file which is used for dependency management.

go mod init my-url-shortener
go mod tidy
go get github.com/gin-gonic/gin

This will generate the go.mod and go.sum file. In go.mod file we have the dependencies that is used in code with the specific Golang version. The go get command is installing the gin package HTTP web framework.

After this execute the command to run go file i.e. main.go

go run main.go

Tools and Technologies Used

1. Golang Programming Language
2. Docker for Containerization
3. Kubernetes for Orchestration
4. CodePipeline and CodeBuild
5. Argo CD for Continuous Deployments

Multi-Stage Docker Build

As a DevOps engineer our responsibility is to automate everything and reach to zero downtime. It all starts from building the Dockerfile where we package all code dependencies in single binary file and run it. To build Dockerfile we need some points to be considered like build the Dockerfile with a small size, always use best-practices means use smaller size or distroless images.

For our golang code, we are building Multi-Stage Dockerfile that will reduce the size from 1 GB to 33 MB !! Yes, we can achieve this to learn about best practices for creating a Dockerfile follow this article

Dockerfile Best Practices: How to Optimize Your Docker Images

Creating Dockerfile

FROM golang:1.23 AS base
WORKDIR /app
COPY go.mod ./
RUN go mod download
COPY . .
RUN go build -o main .

FROM gcr.io/distroless/base
COPY --from=base /app/main .
COPY --from=base /app/static ./static
EXPOSE 8080
CMD ["./main"]

In this Dockerfile we have used 2 stages in which 1st stage where we are building main by executing go build -o main . and storing in /app directory. The first stage is named as base and in second stage we are using distroless image gcr.io/distroless/base and copying the /app working directory from Stage 1 to Stage 2 by using COPY instruction —from=base and in the next stage we are copying static folder which contains index.html file and in last stage we added CMD instruction which will execute the main binary file.

Now, let’s build the Dockerfile and push it to DockerHub.

docker build -t amaurya07/short_url:1 .
docker push amaurya07/short_url:1

Remember to replace the tag with your own docker hub username/short_url and then give version number. After building the docker image successfully we had reduced the image size to 33 MB.

docker images

Now check that the docker image is successfully running or not, create a container and bind the port 8080.

docker run -p 8080:8080 amaurya07/short_url:1

Open the browser and open http://localhost:8080 it will open the Shorten URL page.

Creating Kubernetes Manifests

We had created the Dockerfile and pushed to the DockerHub after testing the image by creating the container and we had successfully completed the first stage. Now, we had to deploy this image on EKS Cluster i.e. Elastic Kubenetes Service so for this we need to create Kubernetes manifests file that include deployment.yaml, service.yaml and ingress.yaml.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shorturl
spec:
  selector:
    matchLabels:
      app: shorturl
  template:
    metadata:
      labels:
        app: shorturl
    spec:
      containers:
      - name: shortenurlapp
        image: amaurya07/short_url:1
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8080

In this deployment.yaml file we will create 1 pod of image amaurya07/short_url:1 on the container Port 8080.

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: shorturl
spec:
  selector:
    app: shorturl
  ports:
    - port: 8080
      targetPort: 8080 
  type: NodePort

Now, to access the application outside we need to create service of type NodePort that will assign the ports between 30000 to 32000.

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shorturl
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing 
    alb.ingress.kubernetes.io/target-type: ip        
spec:
  ingressClassName: alb 
  rules:
     - http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: shorturl 
              port:
                number: 8080

In ingress.yaml file we had defined the annotations of alb i.e. Application Load Balancer which will create the Application Load Balancer of IP and then define the ingressClassName “alb”. But before this we need to install Ingress controller that will create the Ingress resource and load balancer.

Steps to create EKS Cluster

AWS provides a managed Kubernetes cluster, i.e., EKS. With this, we can configure a Kubernetes managed cluster in minutes on AWS. In AWS, we don’t need to manage the Control Plane nodes like provisioning, scaling, and management of the Kubernetes control plane node as they are managed by AWS. We only need to manage the worker nodes if we create self-managed node groups, i.e., EC2; otherwise, in Fargate, it becomes serverless.

We need to install the kubectl, eksctl and aws CLI to create Kubernetes Cluster on EKS.

I am using Windows so these installations are for Windows.

kubectl Installation: https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/

aws CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

eksctl Install: https://eksctl.io/installation/

1. By installing these pre-requisites now configure the AWS CLI with Access Keys, Secret Keys and region where we want to deploy our cluster.
2. In this article I created the AWS Access and Secret Keys and assign the permissions for Administrator access for now.
3. https://amitmaurya.hashnode.dev/learn-how-to-deploy-scalable-3-tier-applications-with-aws-ecs

Now create EKS cluster where we are creating the Self Managed NodeGroups by eksctl CLI.

eksctl create cluster --name demo-eks-cluster --region ap-south-1 --nodes 3 --node-type t3.medium

By executing this command it will create 3 worker nodes in the region Mumbai (ap-south-1) and it is all created by CloudFormation Stack.

Now, the EKS cluster is ready execute the commands to check the nodes

kubectl get nodes

Let’s deploy the Kubernetes manifests yaml files on EKS cluster.

kubectl apply -f ./manifests

So our deployment.yaml, service.yaml and ingress.yaml are created in the default namespace.

kubectl get pods  kubectl get deployment

Now, check service it was assigned the NodePort number and we will check that our application is working or not.

kubectl get service

To access the application check the pod is scheduled on which node as we had created 3 nodes and get the External IP of the node.

kubectl get pods -owide

Now, check the External IP of this node by executing

kubectl get node -owide

The pod is scheduled on the first node so the External IP is 13.200.207.219. But before this open All traffic of the instance where the Pod is scheduled. Navigate to EKS Cluster and go to Compute section and open the Instance security Group.

Now open the browser and go to http://13.200.207.219:30911 with nodePort.

Now, check the ingress resource is created or not.

As we can see there is no Address is there because we have not deployed the Ingress Controller of ALB. Let’s deploy this, we will be using Readme.md instructions to deploy this:

https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/helm/aws-load-balancer-controller/README.md

• Install Helm so that we can deploy the ALB Ingress Controller on EKS cluster. As I am using Windows OS I am installing with chocolatey.
• https://helm.sh/docs/intro/install/

choco install kubernetes-helm

Now, we need to create OIDC (Open ID Connect) IAM provider.

  eksctl utils associate-iam-oidc-provider \
      --region ap-south-1 \
      --cluster demo-eks-cluster \
      --approve

Then we need to download the IAM policy for ALB Ingress controller.

curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

After the policy is downloaded we need to create the IAM policy.

  aws iam create-policy \
      --policy-name AWSLoadBalancerControllerIAMPolicy \
      --policy-document file://iam_policy.json

Now, we need to create IAM role and service account to interact with other AWS service from EKS.

  eksctl create iamserviceaccount \
  --cluster=<cluster-name> \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

After all this, we have to install the helm charts.

helm repo add eks https://aws.github.io/eks-charts

helm repo update eks

  helm install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=demo-eks-cluster -n kube-system --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller --set region=ap-south-1 --set vpcId=<vpc-id>

kubectl get pods -nkube-system

• As we can see the pods of ALB Ingress controller are deployed in kube-system namespace. Now, let’s check the ingress got the address i.e load balancer DNS name or not.

• We can check here we got the address of DNS name of Load Balancer, now let’s copy this and paste it on browser.
• http://k8s-default-shorturl-34b3d7650a-1068404865.ap-south-1.elb.amazonaws.com

• Hurray !! we have achieved the third phase where we deployed the application on EKS and accessed with the Load Balancer DNS name, but now I want my URL to access this, for this we need to modify the ingress.yaml file and add the hosts entry.

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: shorturl
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing 
      alb.ingress.kubernetes.io/target-type: ip        
  spec:
    ingressClassName: alb 
    rules:
      - host: shortenurl.local 
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: shorturl 
                port:
                  number: 8080

AWS CI/CD Pipeline

Now, we had dockerize the application, deployed on the EKS Cluster access the application with the ingress by deploying the ALB Ingress Controller but as a DevOps Engineer we had to implement CI/CD pipeline so that developers can run the pipeline if any changes happens in the code it will automatically deploy to EKS Cluster and we can see the new feature or new version of the image is implemented.

In AWS CI/CD pipeline, AWS offers 4 services by which we can automate the whole CI/CD process.

1. CodeCommit — Version Control System
2. CodePipeline — CI Process
3. CodeBuild — Write buildspec.yaml file that includes stages for the pipeline.
4. CodeDeploy — CD Process

Before heading to CodePipeline, we first create the IAM Role which will be used in CodeBuild to build the phases with necessary permissions.

For this, copy this script and run the command to execute the script.

iamrole.sh:

#!/usr/bin/env bash
TRUST="{   \"Version\": \"2012-10-17\",   \"Statement\": [     {       \"Effect\": \"Allow\",       \"Principal\": {         \"Service\": \"codebuild.amazonaws.com\"       },       \"Action\": \"sts:AssumeRole\"     }   ] }"
echo '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "eks:Describe*", "Resource": "*" } ] }' > /tmp/iam-role-policy

aws iam create-role --role-name CodeBuildKubectlRole --assume-role-policy-document "$TRUST" --output text --query 'Role.Arn'

aws iam put-role-policy --role-name CodeBuildKubectlRole --policy-name eks-describe --policy-document file:///tmp/iam-role-policy

aws iam attach-role-policy --role-name CodeBuildKubectlRole --policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess

aws iam attach-role-policy --role-name CodeBuildKubectlRole --policy-arn arn:aws:iam::aws:policy/AWSCodeBuildAdminAccess

aws iam attach-role-policy --role-name CodeBuildKubectlRole --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

aws iam attach-role-policy --role-name CodeBuildKubectlRole --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess

In this script we are attaching the permissions for CloudWatch Logs, AWS Code Build Admin, AWS S3, and if you are using ECR the permission would be AmazonEC2ContainerRegistryFullAccess.

When you run this shell script go to IAM > Roles > CodeBuildKubectlRole, we will see the attached permissions.

Now, Let’s navigate to AWS CodePipeline and create the Pipeline of Build Stage. As you can see I had created the pipeline named short-url-deployment

1. Click on Create Pipeline.
2. Select Build Custom Pipeline, click on Next.
3. Name the pipeline short-url-deployment , choose execution mode Queued.
4. In Service Role, create New Service Role, leave it as it is named.
5. Scroll Down click on Advanced Settings choose S3 Bucket for Default Location and Default AWS Managed Keys.
6. Now, choose the Source Provider where your code is hosted and connect it to respected source provider (VCS). In my case, my code is hosted on GitHub so I choose GitHub via GitHub App. After connecting to GitHub scroll down to trigger select No filter that will execute the pipeline on Code Push.
7. Now, you move to the Build Stage where you had to add BuildSource provider, so click on Other Build Provider and choose AWS CodeBuild.
8. As you can see click on Project it will open the new window of CodeBuild.

• Name the CodeBuild Provider as short-url
• Scroll Down, leave it as it is Provisioning Model On-demand, Environment Image Managed Instance, Compute EC2.
• Choose Operating System Amazon Linux, Runtime Standard and select latest image.
• In the Service Role choose the Existing Role CodeBuildKubectlRole which we had created earlier.
• Open Additional Configuration select Privileged.
• Scroll down, in buildspec choose Use a buildspec file if you have custom name of buildspec you can specify it otherwise it will automatically look buildspec.yml file in GitHub.
• Select CloudWatch Logs to show logs of phases that we defined in buildspec.yml when it runs.
• Then continue to CodePipeline you will move to codepipeline section

Click on Next, skip the Continuous Delivery Stage for now.

As we had talked about the buildspec.yml file in CodeBuild creation what it is ?

If we had used any other CI tool like Jenkins, GitLab CI it has Jenkinsfile and gitlab-ci.yml file where we write multiple stages like Docker build, push that will be built and deploy to Kubernetes cluster.

As same as in AWS CI/CD CodeBuild uses buildspec.yml file. So let’s understand this file.

version: 0.2
run-as: root

env: 
  parameter-store:
    DOCKER_REGISTRY_USERNAME: /golang-app/docker-credentials/username
    DOCKER_REGISTRY_PASSWORD: /golang-app/docker-credentials/password
    DOCKER_REGISTRY_URL: /golang-app/docker-credentials/url

phases:
  install: 
    commands:
      - echo Installing dependencies...
      - curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
      - chmod +x kubectl
      - mv kubectl /usr/local/bin/
  pre_build:
    commands:
      - echo "Logging to DockerHub"
      - echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USERNAME" --password-stdin $DOCKER_REGISTRY_URL
      - aws sts get-caller-identity
      - aws eks update-kubeconfig --region ap-south-1 --name demo-eks-cluster
      - kubectl config current-context

  build:
    commands: 
      - echo "Building Docker Image"
      - docker build -t "$DOCKER_REGISTRY_USERNAME/short_url:$CODEBUILD_RESOLVED_SOURCE_VERSION" .
      - docker push "$DOCKER_REGISTRY_USERNAME/short_url:$CODEBUILD_RESOLVED_SOURCE_VERSION"

  post_build:
    commands:
      - echo "Docker Image is Pushed"
      - echo "Deploying the application to EKS"
      - kubectl set image deployment/shorturl shortenurlapp=$DOCKER_USERNAME/short_url:$CODEBUILD_RESOLVED_SOURCE_VERSION
      - kubectl get svc

1. We are defining the version, like in docker-compose we define the version when we create docker-compose.yml file it was same as that.
2. Now, this will be build as root.
3. In env section, we had defined 3 parameters for Docker Build, Push commands.

• Navigate to AWS Parameter, it is a service of SSM i.e. AWS System Manager.
• Click on Create Parameter and follow this name path so that it can be better understable /golang-app/docker-credentials/url which means our application-name then which kind of credentials then what we need to store.
• Scroll down, select the Type SecureString and fill the value for URL (docker.io).
• Now, create the 2 parameters for Docker Username and Docker Password. Remember in Docker the password is PAT Token.

Add these parameters in buildspec file in env section.

 env: 
   parameter-store:
     DOCKER_REGISTRY_USERNAME: /golang-app/docker-credentials/username
     DOCKER_REGISTRY_PASSWORD: /golang-app/docker-credentials/password
     DOCKER_REGISTRY_URL: /golang-app/docker-credentials/url

Now we have phases section where we 3 phase install, pre-build, build and post-build.

In install phase we are installing the required CLI like of kubectl.

    install: 
      commands:
        - echo Installing dependencies...
        - curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
        - chmod +x kubectl
        - mv kubectl /usr/local/bin/

In pre-build phase, first we are logging to DockerHub account so that we can push the image, then we are updating the kubeconfig of our EKS Cluster and at last we check the current-context of kubeconfig.

    pre_build:
      commands:
        - echo "Logging to DockerHub"
        - echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USERNAME" --password-stdin $DOCKER_REGISTRY_URL
        - aws sts get-caller-identity
        - aws eks update-kubeconfig --region ap-south-1 --name demo-eks-cluster
        - kubectl config current-context

Now, come to build section in which we are building Docker Image and Pushing to DockerHub.

    build:
      commands: 
        - echo "Building Docker Image"
        - docker build -t "$DOCKER_REGISTRY_USERNAME/short_url:$CODEBUILD_RESOLVED_SOURCE_VERSION" .
        - docker push "$DOCKER_REGISTRY_USERNAME/short_url:$CODEBUILD_RESOLVED_SOURCE_VERSION"

In post-build, we are setting the new image of the deployed app on our EKS Cluster with the command kubectl set image

    post_build:
      commands:
        - echo "Docker Image is Pushed"
        - echo "Deploying the application to EKS"
        - kubectl set image deployment/shorturl shortenurlapp=$DOCKER_USERNAME/short_url:$CODEBUILD_RESOLVED_SOURCE_VERSION
        - kubectl get svc

As we can see our 2 stages Source and Build both are successful and shows the output in CodePipeline.

Deploy ArgoCD on EKS Cluster

Now, we will deploy the CD part (Continuous Deployment) on EKS cluster so that when any changes happens in kubernetes manifests it will automatically deploy to the EKS cluster.

ArgoCD is a GitOps tool that will manage the deployments of applications in Kubernetes clusters. It pulls the desired state and sync with the current state and if not matched it update the changes on Kubernetes cluster.

Create the namespace of ArgoCD

kubectl create namespace argocd

Now deploy the CRD’s (Custom Resource Definitions), deployments, services in namespace of argocd.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl get po -n argocd

Edit the service of argocd server to access the ArgoCD outside.

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "NodePort"}}' 
kubectl get svc -n argocd

Now, we can access the ArgoCD UI.

• Username: admin
• Password: It is stored in secrets of argocd-initial-admin-secret in namespace of argocd.

kubectl get secrets argocd-initial-admin-secret -n argocd -oyaml   echo "Your password" | base64 -d

As secrets are stored in base64 encrypted, so we had decoded it.

We need to create an application in ArgoCD, so I had created the file name go-app-deployment.yaml.

 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
   name: go-app-deployment
   namespace: default
 spec:
   project: default
   source:
     repoURL: 'https://github.com/amitmaurya07/Golang_AWS_Deployment.git'
     targetRevision: main
     path: kuberenetes/manifests
   destination:
     server: 'https://kubernetes.default.svc'
     namespace: default
   syncPolicy:
     automated:
       prune: true
       selfHeal: true

In this yaml file see the apiVersion this is the CRD of ArgoCD to create an application of kind. Then we had given the repoURL of our github repository where argocd will check the changes and there is path of kuberenetes manifests where our manifests files are stored. In syncPolicy there is automated selfHeal is true which means it monitors the kubernetes resources if any changes happens it matches the desired state to the current state automatically.

As we can see the application on ArgoCD and our pods with service and ingress is shown in ArgoCD UI.

Conclusion

In this article, we had learnt how to deploy Golang Application on EKS Cluster through CodePipeline and CodeBuild. Meanwhile in the upcoming blogs you will see a lot of AWS Services creation and automating it with terraform. Stay tuned for the next blog !!!

GitHub Code : Golang AWS Deployment

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

End-to-End DevOps for a Golang Web App: Docker, EKS, AWS CI/CD was originally published in Towards AWS on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we will learn about AWS EC2 Instance Storage i.e. EFS (Elastic File System), how to create it, how to mount it to EC2 instances. In today’s modern world we need a storage solution that is scalable, efficient and flexible to scale our business to handle growing data demands.

Overview of AWS EFS (Elastic File System)

As we know AWS offers 200+ services which makes the best cloud solution which meets our business needs whether it’s in AI field, DevOps field. If we talk about EC2 i.e. Elastic Compute Cloud storage solutions it offers EBS (Elastic Block Storage) and EFS (Elastic File System).

Elastic File System is Network Shared Filesystem (NFS) that means multiple 100s of EC2’s in different AZ’s (Availability Zone’s) can talk to single central storage that is mounted where multiple EC2’s can store files on network. In simple, we have 100’s of EC2 in AZ1 and 100’s of EC2 in AZ2 they have single filesystem where they can store files and can see these files in other AZ’s also. That’s make it very highly scalable, flexible and also expensive as it is 3 times of gp2 price.

Prerequisites for Setting Up AWS EFS

1. AWS Account
2. EFS Client Installed
3. IAM Permissions

Step-by-Step Guide to Creating an AWS EFS

Open AWS Management Console and on top in Search Bar, write EFS.

Open EFS and click on Create Filesystem. Select the default VPC and the Name is optional so keep it blank or you can provide name.

Go to Customize you will see the lot of configuration options, we will go step by step understand it what it is.

• Filesystem Type: In filesystem type there are 2 options. This is for Availability and Durability.
• Regional: In regional, the EFS will be available in Multi-AZ which will be great for Production.
• One-Zone: One-Zone means EFS will be available in single AZ which is great for testing purpose or dev environments that will be cost-saving.

• Automatic Backups: It will take backup of your data automatically. For now, we will check this.

• Lifecycle Management: EFS provides lifecycle that means it will automatically move the file to the Storage Tier after N number of days. EFS provides 3 types of storage tiers that are:
• Standard (Default): When we are frequently accessing files it lies in Standard storage tier.
• EFS-IA (Infrequent Access): Infrequent Access means a certain number of days passed but we do not accessing the file frequently. Like if I set 60 Days if the file is not accessed in 60 Days it will automatically move to the EFS-IA that will be cost effective and cheaper.
• Archive: In Archive, it is more 50% cheaper, the data that we are accessing very few times in a year so it will automatically move to Archive storage tier.
• Archive Storage Tier is not available in One-Zone.

This all done by setting lifecycle management on EFS, we will leave it default.

In this, we had set the Transition to Standard to On First Access means whenever the file is accessed first time it will automatically to Standard Storage tier.

• Performance Settings: EFS provides 2 throughput modes i.e. Enhanced and Bursting.
• In Enhanced Throughput mode there are 2 methods:

1. Elastic: This is the recommended one because it automatically scales the throughput UP and DOWN on workload.
2. Provisioned: In provisioned one, we had to decide the throughput size.

• In Bursting Mode it has the General Purpose and Max I/O performance settings. General Purpose is recommended for Web Servers and Max I/O is recommended where we need low latency like for data analytics.

• For now, we are going with default settings Elastic Throughput Mode and General Purpose Performance Setting. Click on “Next”

Now, we will move to the Network Access where we choose the VPC and set the Mount Targets. As it is One-Zone so, we will set the Mount Target in single Availability Zone (AZ)

But, before attaching this security group we will create new Security Group for EFS where we will allow ALL-TRAFFIC.

After this, select the security group of EFS-Demo in Mount Target.

Then, click on Next where the page opens Filesystem Policy leave it as it is then you will reach to Review and Create and click on Create.

Now, the EFS filesystem is successfully created. Let’s move to the EC2 section where we will create 2 EC2 instances in different Availability Zone’s.

1. In EFS Filesystem we only Pay what we use and one thing more only Linux distributions can be attached to EFS filesystem not Windows.
2. Create EC2 Instance name EFS-Demo-1 in us-west-1a (N.California), then select the AMI (Amazon Machine Image)OS of Ubuntu 22.04 you can select anyone of Linux distribution, then select t2.micro Instance Type and select Proceed without KeyPair.
3. Now, here comes the Network Settings Part, click on Edit and choose Subnet of us-west-1a other things leave it as default.

1. Scroll Down to Configure Storage section where EBS volume have 8 Gib of gp2 and scroll down you will see you will see Filesystems click on Edit then you can see our created EFS Filesystem.
2. If we don’t want to do this we had to run some commands but for now I am configuring this with EC2 configuration settings.

Now click on Add Shared Filesystem it will automatically set the mount point and create Security Groups and attach it to EFS.

1. Now, click on Create the Instance and with same settings click 2nd instance in another Availability zone us-west-1c and select the existing security group that is attached to Instance 1 and use the same EFS Filesystem.
2. Here, now you see security group (efs-sg-1) is automatically attached to EFS Mount Targets.

In newly created Security Group the Inbound Rules is NFS of port 2049 where the destination is Instance Security Group ID.

Now our 2 instances are ready which are in different Availability Zone’s and we can see in mount targets Instance 2nd Security Groups is also attached.

1. Let’s connect to Instance 1 and create file on mount target (/mnt/efs/fs1). Go to Instance 1 and click on Connect to Instance, in Ubuntu OS we need to run the commands on both the instances
2. Then check the mount point exists or not if not exists create this.
3. Check that the filesystem is mounted or not by executing the command.

df -h | grep efs

Create Sample hello.txt in this folder in Instance 1 and checks that the file is created in Instance 1 is we can see or not.

echo "hello-world from Instance 1" > /mnt/efs/fs1/hello.txt

Now connect to Instance-2 and check that the file is accessible or not.

Conclusion

In this article, we had learnt how to create EFS (Elastic File System) and mount it to EC2 instances. EFS can scale up to Petabytes automatically. Meanwhile in the upcoming blogs you will see a lot of AWS Services creation and automating it with terraform. Stay tuned for the next blog !!!

GitHub Code : github.com/amitmaurya07/AWS-Terraform/tree..

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter | Podcast
• Create a free AI-powered blog on Differ.
• More content at PlainEnglish.io

Step-by-Step Guide: How to Create and Mount AWS EFS (Elastic File System) was originally published in AWS in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the previous article, we deployed the 3-tier architecture on GKE (Google Kubernetes Engine) using Jenkins (CI/CD). In this article, we will deploy the same application, Wanderlust, on AWS ECS (Elastic Container Service) with ALB (Application Load Balancer).

GitHub Terraform Code : github.com/amitmaurya07/AWS-Terraform/tree/..

GitHub Application Code : https://github.com/amitmaurya07/wanderlust-devsecops/tree/aws-ecs-deployment

Overview of 3-Tier Architecture

As we had already discussed the 3 tier architecture in previous article so let’s again discuss this.

A three-tier application is a software architecture pattern which includes the Application Layer, Data Layer, and Presentation Layer. It enhances the maintainability, scalability, and flexibility.

1) Presentation Layer: When we open the web browser and then opens any website it means we are interacting with the website that will take inputs from client side. In simple the frontend part, or GUI (Graphical User Interface) is the presentation layer.

2) Application Layer: We have heard the term business logic in the tech industry a lot. The application layer contains the logic part, i.e., the backend. When we give any input on the frontend, this layer processes some logic against that input and provides the result.

3) Data Layer: As we had heard the data word that means it consists the information which is stored in database that is connected to Presentation Layer and Application Layer.

AWS ECS (Elastic Container Service)

Before diving into benefits of AWS ECS, let’s first understand or take the overview of ECS (Elastic Container Service).

ECS (Elastic Container Service) is one of the container services that is used to deploy your application code, binaries, dependencies in a container like docker does. In ECS there are three components -

1. Capacity : Decide the Infrastructure on which you want to deploy the container (EC2, or AWS Fargate)
2. Controller : Here, ECS Scheduler comes in place that deploys the application.
3. Provisioning : After deploying containers on ECS, from this we can manage the containers.

ECS provides scalability that will automatically scale containers based on the demand.

Prerequisites

1. AWS Account: AWS account with permissions to use ECS, ECR, DocumentDB (MongoDb), VPC. For now we will assign Administrative permissions to user.
2. Docker: Ensure Docker is installed locally to build and push container images.
3. Terraform: Install Terraform CLI in your terminal. (Terraform For Windows)

Setting Up AWS Environment

Creating IAM User with Required Permissions for Terraform

• Navigate to Search Bar of AWS Console and search IAM (Identity Access and Management)

• Click on IAM, in left side go to Users section and click on Create User.

• After naming the user “terraform“, assign permissions for now Administrator Access.

• Now, click on terraform user go to Security Credentials and scroll down for Access Key.

• As we can see there is no Access Keys so let’s create access key, choose CLI for usecase and click on Next provide Description and then Next you will get Access and Secret Key.

• Download the CSV file and now copy the Secret and Access Key for usage in terraform.

Networking Creation with Terraform

Let’s create the infrastructure on AWS by creating VPC, Subnets, Route Tables, Internet Gateway by Terraform Modules.

1. Create the directory structure for module VPC creation vpc-module in which it contains the VPC Creation, Subnets, Interet Gateway and Route Table.

Inside the VPC module we have main.tf, variables.tf and output.tf. In the main.tf file we are going to create VPC, then Public and Private Subnet, then Internet Gateway and Route Table for Private and Public and then attach it with Subnets and Internet gateway.

vpc-module/main.tf:

 resource "aws_vpc" "three-tier" {
   cidr_block       = "10.0.0.0/16"
   instance_tenancy = "default"

   tags = {
     Name = "three-tier"
   }
 }

 data "aws_availability_zones" "available" {
 }

 resource "aws_subnet" "public" {
   vpc_id = aws_vpc.three-tier.id
   cidr_block = cidrsubnet(aws_vpc.three-tier.cidr_block, 8, count.index + 1)
   count = 2
   availability_zone = element(data.aws_availability_zones.available.names, count.index)
   map_public_ip_on_launch = true

   tags = {
     Name = "three-tier-public"
   }
 }

 resource "aws_subnet" "private" {
   vpc_id = aws_vpc.three-tier.id
   cidr_block = cidrsubnet(aws_vpc.three-tier.cidr_block, 8, count.index + 4)
   count = 2

   tags = {
     Name = "three-tier-private"
   }
 }

 resource "aws_internet_gateway" "three-tier-igw" {
   vpc_id = aws_vpc.three-tier.id
   tags = {
     "Name" = "three-tier-igw"
   }
 }

 resource "aws_route_table" "public-rt" {
   vpc_id = aws_vpc.three-tier.id
   route {
     cidr_block = "0.0.0.0/0"
     gateway_id = aws_internet_gateway.three-tier-igw.id
   }
   tags = {
     "Name" = "three-tier-public-rt"
   }
 }

 resource "aws_route_table" "private-rt" {
   vpc_id = aws_vpc.three-tier.id
   tags = {
     "Name" = "three-tier-private-rt"
   }
 }

 resource "aws_route_table_association" "public-rt-association" {
   count = length(aws_subnet.public)
   subnet_id = aws_subnet.public[count.index].id
   route_table_id = aws_route_table.public-rt.id
 }

 resource "aws_route_table_association" "private-rt-association" {
   count = length(aws_subnet.private)
   subnet_id = aws_subnet.private[count.index].id
   route_table_id = aws_route_table.private-rt.id
 }

Let’s break each resource block to proper understand what each resource block is going to create on AWS.

 resource "aws_vpc" "three-tier" {
   cidr_block       = "10.0.0.0/16"
   instance_tenancy = "default"

   tags = {
     Name = "three-tier"
   }
 }

 data "aws_availability_zones" "available" {
 }

There are 2 blocks in main.tf file of VPC Module:

• Resource Block name three-tier is going to create VPC (Virtual Private Cloud) of CIDR (Classless-InterDomain Routing) “10.0.0.0/16“.
• Data block is going to fetch the availaiblity zones.

    resource "aws_subnet" "public" {
      vpc_id = aws_vpc.three-tier.id
      cidr_block = cidrsubnet(aws_vpc.three-tier.cidr_block, 8, count.index + 1)
      count = 2
      availability_zone = element(data.aws_availability_zones.available.names, count.index)
      map_public_ip_on_launch = true

      tags = {
        Name = "three-tier-public"
      }
    }

    resource "aws_subnet" "private" {
      vpc_id = aws_vpc.three-tier.id
      cidr_block = cidrsubnet(aws_vpc.three-tier.cidr_block, 8, count.index + 4)
      count = 2

      tags = {
        Name = "three-tier-private"
      }
    }

In this snippet of terraform code there are 2 resource blocks that will create Public and Private Subnet.

• Resource Block of Public Subnet has the VPC ID that will attach the VPC which will be created next is cidr_block which contains cidrsubnet function that will calculate the subnets in the given network address and subnet mask in CIDR format.
• aws_vpc.three-tier.cidr_block will picks the CIDR block of created VPC.
• 8 is the to add subnets in the CIDR block like I have “10.0.0.0/16“ now it will add 8 in subnet mask /16+8 it will become /24 bits.
• count.index + 1 is used to create multiple subnets in AWS with Terraform. In the CIDR block 10.0.0.0/24 is created on index[0] and on index[1] the new CIDR block is there 10.0.1.0/24
• Now next is count which will create 2 resources of Public Subnet in AWS.
• In availability zone there is element function which will take the value from the data block of availability zone ["us-east-1a", "us-east-1b", "us-east-1c"] after this it will select the availability zone on basis of count.index
• map_public_ip_on_launch is true means in the Public Subnet the resources that are in this subnet will have Public IP that is automatically assigned on launch.

  resource "aws_internet_gateway" "three-tier-igw" {
    vpc_id = aws_vpc.three-tier.id
    tags = {
      "Name" = "three-tier-igw"
    }
  }

In this resource we are going to create Internet Gateway with the name three-tier-igw that is attached with VPC ID (vpc_id).

  resource "aws_route_table" "public-rt" {
    vpc_id = aws_vpc.three-tier.id
    route {
      cidr_block = "0.0.0.0/0"
      gateway_id = aws_internet_gateway.three-tier-igw.id
    }
    tags = {
      "Name" = "three-tier-public-rt"
    }
  }

  resource "aws_route_table" "private-rt" {
    vpc_id = aws_vpc.three-tier.id
    tags = {
      "Name" = "three-tier-private-rt"
    }
  }

• Now, there are 2 resources in which Public and Private Route Table is created that is used to route the traffic for the resources lying in Subnets.
• In the Public Route Table (public-rt) as we can see there is a route block that contains the CIDR block of 0.0.0.0/0 which means allowing all traffic and next is gateway ID where Internet Gateway is attached. In the Public Subnet we would attach the IGw (Internet Gateway) to access the resources from Internet.

  resource "aws_route_table_association" "public-rt-association" {
    count = length(aws_subnet.public)
    subnet_id = aws_subnet.public[count.index].id
    route_table_id = aws_route_table.public-rt.id
  }

  resource "aws_route_table_association" "private-rt-association" {
    count = length(aws_subnet.private)
    subnet_id = aws_subnet.private[count.index].id
    route_table_id = aws_route_table.private-rt.id
  }

In these 2 resource blocks we are attaching the Subnets (Public Subnet in Public Route Table and Private Subnet in Private Route Table).

/vpc-module/variables.tf:

  variable "public_subnet_cidrs" {
    description = "List of CIDR blocks for public subnets"
    type        = list(string)
  }

  variable "private_subnet_cidrs" {
    description = "List of CIDR blocks for private subnets"
    type        = list(string)
  }

We had declared two variables for CIDR’s where the type is list.

/vpc-module/output.tf:

  output "vpc_id" {
    value = aws_vpc.three-tier.id
  }

  output "public_subnet_id" {
    value = aws_subnet.public[*].id
  }

  output "subnet_id" {
    value = aws_subnet.private[*].id
  }

• In the output.tf file we are printing the output of vpc_id, public and private Subnet ID.

Now here comes the parent main.tf file in which we call this vpc-module and the required parameters.

module "vpc" {
  source = "./modules/vpc-module"
  public_subnet_cidrs = var.public_subnet_cidrs
  private_subnet_cidrs = var.private_subnet_cidrs
}

Okay, we have automated the VPC creation on AWS with Terraform now, let’s break down the parent main.tf file.

If we will going to break all the modules main.tf the blog will become lengthier so we will cover the modules with its proper understanding in the next blog.

main.tf:

provider "aws" {
  region = var.region
}
module "ec2_db" {
  source = "./modules/aws_ec2_module"
  security_group_ids = [module.aws_security_group.security_group_id]
  name = "${var.name}-mogodb"
  ami = "ami-09b0a86a2c84101e1"
  instance_type = "t2.micro"
  subnet_id = module.vpc.public_subnet_id[0]
}
module "vpc" {
  source = "./modules/vpc-module"
  public_subnet_cidrs = var.public_subnet_cidrs
  private_subnet_cidrs = var.private_subnet_cidrs
}
module "aws_security_group" {
  source = "./modules/security-group"
  sg_name = "${var.name}-sg"
  vpc_id = module.vpc.vpc_id
}
module "aws_lb" {
  source = "./modules/aws_alb_module"
  name = "${var.name}-alb"
  security_group_id = module.aws_security_group.security_group_id
  subnet_id = module.vpc.public_subnet_id
  vpc_id = module.vpc.vpc_id
}
module "frontend-aws_ecr" {
  source = "./modules/aws_ecr_module"
  name = "${var.name}-frontend"
}
module "backend-aws_ecr" {
  source = "./modules/aws_ecr_module"
  name = "${var.name}-backend"
}
module "aws_ecs_cluster" {
  source = "./modules/aws_ecs_cluster_module"
  name = var.name
}
module "frontend-ecs" {
  source = "./modules/aws_ecs_module"
  name = "${var.name}-frontend-ecs"
  ecr_repo_url = module.frontend-aws_ecr.frontend-repository_url
  image_tag = "3"
  ecs_cluster_id = module.aws_ecs_cluster.ecs_cluster_id
  port = 5173
  vpc_id = module.vpc.vpc_id
  subnet_id = module.vpc.public_subnet_id
  security_group_id = module.aws_security_group.security_group_id
  target_group_arn = module.aws_lb.tg-arn
}
module "backend-ecs" {
  source = "./modules/aws_ecs_module"
  name = "${var.name}-backend-ecs"
  ecr_repo_url = module.backend-aws_ecr.frontend-repository_url
  port = 5000
  ecs_cluster_id = module.aws_ecs_cluster.ecs_cluster_id
  image_tag = "2"
  vpc_id = module.vpc.vpc_id
  subnet_id = module.vpc.public_subnet_id
  security_group_id = module.aws_security_group.security_group_id
  target_group_arn = module.aws_lb.tg-arn-backend
}

This looks so big but not to worry we will understand each module block and how we are taking values from output.tf file and variables.tf file.

This is the provider block which contains on which cloud platform we are going to deploy whether it’s AWS, Azure or GCP. So , we are with AWS and defined the region in which var.region is the value where region variable is declared in variables.tf file.

module "ec2_db" {
  source = "./modules/aws_ec2_module"
  security_group_ids = [module.aws_security_group.security_group_id]
  name = "${var.name}-mogodb"
  ami = "ami-09b0a86a2c84101e1"
  instance_type = "t2.micro"
  subnet_id = module.vpc.public_subnet_id[0]
}

This module creates the EC2 instance for the database server where MongoDB is deployed on Docker. In this block, source indicates where our EC2 module is stored; in our case, it's locally under the modules folder. Next is security_group_ids, which contains a list []. Within this list, another module is called, specifically the security group module, which outputs the security group id, an attribute of the security group.

In name the value is "${var.name}-mogodb" which means when we do terraform apply or plan we will give the input for the name variable means if the input is “three-tier“ it will give the name “three-tier-mongodb“ for the EC2 instance.

In the ami we had given the AMI ID of Ubuntu to create the Ubuntu Operating System.

Instance Type will be t2.micro that is free-tier in AWS. Now, we are attaching the subnet in which subnet this resource is going to lie this subnet_id will decide for this we are creating in Public Subnet.

module "vpc" {
  source = "./modules/vpc-module"
  public_subnet_cidrs = var.public_subnet_cidrs
  private_subnet_cidrs = var.private_subnet_cidrs
}

This module will create VPC where public and private subnet CIDR’s variables are used.

module "aws_security_group" {
  source = "./modules/security-group"
  sg_name = "${var.name}-sg"
  vpc_id = module.vpc.vpc_id
}

This module will create the security group for the EC2 instance and ECS Cluster in which we will use further.

module "aws_lb" {
  source = "./modules/aws_alb_module"
  name = "${var.name}-alb"
  security_group_id = module.aws_security_group.security_group_id
  subnet_id = module.vpc.public_subnet_id
  vpc_id = module.vpc.vpc_id
}

This module will create loadbalancer which will be used to access the application through the DNS of Network Load Balancer where Subnet ID, security group ID and vpc_id is attached. In Subnet_ID we are calling the module of vpc which prints the attribute of public_subnet_id.

module "frontend-aws_ecr" {
  source = "./modules/aws_ecr_module"
  name = "${var.name}-frontend"
}

Now, this is ECR (Elastic Container Registry) creation for frontend and backend images, we had just changed the name here with -frontend and -backend.

module "aws_ecs_cluster" {
  source = "./modules/aws_ecs_cluster_module"
  name = var.name
}

Now the AWS ECS (Elastic Container Service) comes in the picture where we deploy the ECS cluster in which 2 services are created for frontend and backend.

module "frontend-ecs" {
  source = "./modules/aws_ecs_module"
  name = "${var.name}-frontend-ecs"
  ecr_repo_url = module.frontend-aws_ecr.frontend-repository_url
  image_tag = "3"
  ecs_cluster_id = module.aws_ecs_cluster.ecs_cluster_id
  port = 5173
  vpc_id = module.vpc.vpc_id
  subnet_id = module.vpc.public_subnet_id
  security_group_id = module.aws_security_group.security_group_id
  target_group_arn = module.aws_lb.tg-arn
}

In these 2 modules we are creating 2 ECS services for frontend and backend which contains ecr_repo_url where the value is taking from backend AWS ECR module which will take the output of frontend repository URL which is defined in parent output.tf file.

output "backend-repository_url" {
  value = module.backend-aws_ecr.frontend-repository_url
}

Next is image tag which we are declaring, then the ECS cluster ID in which we want to deploy our services the cluster ID is defined in output.tf file and used to call with module of AWS ECS cluster with cluster ID attribute.

output "ecs_cluster_id" {
  value = module.aws_ecs_cluster.ecs_cluster_id
}

Next is port, vpc_id, subnet_id and security_group_id and in last target group arn that will be taken from load balancer target group which is defined parent output.tf file.

output "tg-arn" {
  value = module.aws_lb.tg-arn
}

Build Docker Images for Each Tier

Dockerfile Build

GitHub Code : https://github.com/amitmaurya07/wanderlust-devsecops.git

Let’s build the Dockerfile for frontend and push it to ECR (Elastic Container Registry) then deploy it to ECS (Elastic Container Service).

When the ECR Repository is created go to ECR repository for frontend and click on View Push Commands which will give the commands to build and push the Dockerfile in ECR frontend repository.

 aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin 880849992790.dkr.ecr.ap-south-1.amazonaws.com
 docker build -f .\backend\Dockerfile -t backend:1 .
 docker tag backend:1 880849992790.dkr.ecr.ap-south-1.amazonaws.com/three-tier-backend:1 
 docker push 880849992790.dkr.ecr.ap-south-1.amazonaws.com/three-tier-backend:1

1. The first line means aws ecr get-login will login with the AWS Access Key and Secret Key then it is building the backend Dockerfile and then tagging the Dockerfile then pushed to the ECR Registry.
2. Now paste the commands in the in VS Code (IDE).

 aws sts get-caller-identity

If you recieve this error that the user is not authenticated to perform GetAuthorizationToken then grant the permission to the user. For now, I am adding the permission of “AmazonElasticContainerRegistryPublicFullAccess” to the user.

Set Up Database MongoDB on Amazon EC2

For cost optimization and we are building this project for article purpose so we will install the MongoDB on EC2 instance otherwise for production environment use DocumentDB that is managed AWS service for mongoDB.

We had discussed earlier in the main.tf file we are creating EC2 instance with the Terraform module

 resource "aws_instance" "ec2_instance" {
   ami = var.ami 
   instance_type = var.instance_type
   subnet_id = var.subnet_id
   vpc_security_group_ids = var.security_group_ids

   user_data = <<-EOF
     #!/bin/bash
     sudo apt-get update -y
     curl -fsSL https://get.docker.com -o get-docker.sh
     sudo sh get-docker.sh
     sudo systemctl start docker
     sudo systemctl enable docker
     sudo docker run -d --name mongodb -p 27017:27017 mongo:4.4
   EOF
   tags = {
     Name = var.name
   }
 }

This will install the MongoDB in the EC2 instance from the User Data Script.

Apply the Terraform Modules:

As we had discussed about Terraform Modules earlier of parent main.tf file now let’s do the terraform CLI execution.

Download the AWS dependencies that are in provider section. As we had defined in provider block. It initializes the modules folder and create .terraform directory and module.json file which contains about modules.

terraform init

Now do the terraform plan to check that the infrastructure that is going to create on AWS has right values or not.

terraform plan

1. When we execute this command it will ask for the input from the user i.e. it is asking for the name.

1. As we can see it prints the output which we had defined in the output.tf file.
2. It’s the time to create the infrastructure on AWS when we execute Apply command.

terraform apply --auto-approve

1. Here it is saying we have 28 resources to add in AWS (Automation) and it starts to creating the resources on AWS.
2. When we had seen the Terraform Plan output some of the values are unknown where it has written “known after apply” but when you do terraform apply command it created that infrastructure and prints the outputs.

1. We got DNS name of ALB, repository URL, cluster ID everything. Now let’s hit the DNS name of ALB and check whether we can access the frontend or not.
2. http://three-tier-alb-alb-b35382f79712e765.elb.ap-south-1.amazonaws.com:5173

Hurray !!! We had successfully deployed the frontend service, now let’s check the backend service but right now it will not be accessible because we need to change in .env file of backend github repository. After this make changes also in server.js file for CORS ISSUE.

.env file

 PORT=5000
 MONGODB_URI="mongodb://<Public IP of instance>:27017/wanderlust"
 CORS_ORIGIN="http://<ALB DNS Name>:5173"

1. After changes has been build Dockerfile again and push and change the image tag in parent main.tf of backend ECS module. As we can see our database MongoDB is connected now let’s check that backend is accessible or not.
2. http://three-tier-alb-alb-b35382f79712e765.elb.ap-south-1.amazonaws.com:5000

As we can see in ECS Cluster the backend service is updating (1 Pending and 1 Running) after applying terraform apply command.

When the backend is perfectly deployed with the changes, we have to do little changes in .env file of frontend to add VITE_API_PATH which contains the value of Load Balancer:5000 (Backend URL) then build the new frontend image and change the image tag in terraform parent main.tf frontend ECS module.

To get the Load Balancer URL from AWS GUI navigate to AWS Cluster three-tier then click on any service and go to configuration and networking section scroll down you will get Load Balancer DNS and see the status of Target Group is healthy or not.

After everything is done, you have done the deployment of 3 tier application now clean the resources on AWS by executing the command.

terraform destroy --auto-approve

Conclusion

In this article, we had learnt the deployment of 3 tier application on AWS ECS with Terraform Modules. Another Project will come soon so stay tuned for the next blog !!!

GitHub Terraform Code : github.com/amitmaurya07/AWS-Terraform/tree/..

GitHub Application Code : https://github.com/amitmaurya07/wanderlust-devsecops/tree/aws-ecs-deployment

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter | Podcast
• Create a free AI-powered blog on Differ.
• More content at PlainEnglish.io

Learn How to Deploy Scalable 3-Tier Applications with AWS ECS was originally published in AWS in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we are going to learn about the most used service i.e. AWS EC2 (Elastic Compute Cloud) and Security Group with the Terraform Modules, how to provision with terraform. We all know AWS offers 200 services, and to manage to make it scalable, efficient is what all businesses want but to manage these services manually is a big challenging task and also time-consuming and error-prone.

By introducing Terraform as IaC (Infrastructure As Code) tool it enables to automate the entire infrastructure within minutes by ensuring consistency and repeatability across all environments.

Prerequisites

1. Terraform
2. AWS Account
3. Little bit of knowledge of AWS and Terraform :)

Setting Up Environment

1. Installing Terraform on Windows

Terraform Install on Windows

1. Creating IAM User with Required Permissions for Terraform

• Navigate to Search Bar of AWS Console and search IAM (Identity Access and Management)

• Click on IAM, in left side go to Users section and click on Create User.

• After naming the user “terraform“, assign permissions for now Administrator Access.

• Now, click on terraform user go to Security Credentials and scroll down for Access Key.

• As we can see there is no Access Keys so let’s create access key, choose CLI for usecase and click on Next provide Description and then Next you will get Access and Secret Key.

• Download the CSV file and now copy the Secret and Access Key for usage in terraform.

Understanding Terraform Basics

In corporate world, if we talk about Terraform it was the most used IaC tool that was also most asked in interviews. We are going to learn about Terraform Modules, variables.tf, terraform.tfvars.

Terraform Modules:

When we start writing Terraform code, we usually have main.tf, variables.tf, and output.tf. But what if we need to use these files every time with different settings? Writing them over and over is a hassle and can lead to mistakes.

That’s where Terraform Modules come in handy. They let us separate the service code from the main main.tf file and the other two files. For example, if we want to create an EC2 instance, we make a module folder with main.tf, variables.tf, and outputs.tf. Then, we just call this module in the main.tf file in the parent folder.

Variables:

In terraform to create variables we create variable.tf in which we create variable block and call the variables in main.tf file. We will see how we are calling the variables.

terraform.tfvars:

It is used to store variables, while variables.tf is used to declare the variables and in tfvars we assign the values for them.

Creating AWS EC2 Instance

Overview of EC2 Instance:

Let’s say we want to launch some servers to deploy some applications in organization, for this we need servers in our datacenters that will be costly and we had to handle all the servers manually.

To reduce this setup of datacenters cost AWS has launched EC2 service that creates virtual servers in our nearby datacenters by this we just need to only manage virtual machines on AWS with minimal cost.

If you need to check the prices of Amazon EC2, open the AWS Calculator pricing that will give the real time price of AWS EC2.

AWS Calculator

Configuring EC2 Instance with Terraform:

Let’s create directory structure for ec2-module and parent folder. As we can see that modules folder contain two ec2-instance and security-group.

1. First, let’s create the ec2-instance module in which it contains main.tf, variable.tf and outputs.tf.

main.tf:

 resource "aws_instance" "ec2_instance" {
   ami = var.ami 
   instance_type = var.instance_type
   vpc_security_group_ids = var.security_group_ids

   user_data = <<-EOF
               #!/bin/bash
               apt-get update
               apt install nginx -y
               systemctl start nginx
               systemctl enable nginx
               EOF
   tags = {
     Name = "Terraform Instance"
   }
 }

In main.tf folder of ec2-instance module it has the resource block which contains parameter ami, instance_type, and vpc_security_group. It also has the User Data script that contains the commands to install nginx and at last it has tags.

variables.tf:

 variable "ami" {
   description = "AMI ID for the EC2 instance"
   type        = string
 }

 variable "instance_type" {
   description = "Type of EC2 instance"
   type        = string
 }

 variable "security_group_ids" {
   description = "List of security group IDs to attach to the instance"
   type        = list(string)
 }

In variables.tf we declare the variables ami, instance_type and security_group_ids. To call the variables in main.tf we write var.ami or var.instance_type

output.tf:

 output "instance_id" {
   value = aws_instance.ec2_instance.id
 }

 output "public_ip" {
   value = aws_instance.ec2_instance.public_ip
 }

1. In output.tf file we are printing outputs of attributes of EC2 instance like public_ip and instance_id.
2. Now, let’s do the same for the security group module in which same directory structure is there main.tf, variables.tf and output.tf

main.tf:

 resource "aws_security_group" "ec2_sg" {
   name        = var.sg_name
   description = "Security group for EC2 instance"

   ingress {
     from_port   = 22
     to_port     = 22
     protocol    = "tcp"
     cidr_blocks = [var.allowed_ssh_cidr] 
   }

   ingress {
     from_port   = 80
     to_port     = 80
     protocol    = "tcp"
     cidr_blocks = [var.allowed_ssh_cidr]
   }

   egress {
     from_port   = 0
     to_port     = 0
     protocol    = "-1"
     cidr_blocks = [var.allowed_ssh_cidr]
   }

   tags = {
     Name = var.sg_name
   }
 }

In the main.tf of security group we have resource block in which we are creating inbound rules (ingress) that allows the port 80 and SSH port 22.

variables.tf:

 variable "sg_name" {
   description = "Name of the security group"
   type        = string
   default     = "ec2_security_group"
 }

 variable "allowed_ssh_cidr" {
   description = "CIDR block allowed to SSH"
   type        = string
   default     = "0.0.0.0/0" 
 }

In the variables.tf we had declared two variables that is sg_name and allowed_ssh_cidr which contains default “0.0.0.0/0“ that will allow all incoming traffic.

output.tf:

 output "security_group_id" {
   description = "The ID of the security group"
   value       = aws_security_group.ec2_sg.id
 }

In the output.tf file we are getting the output of sg_id. aws_security_group is the parameter name and ec2_sg is the local name which had taken from the resource block of security group and id is the attribute.

Now we had done the creation of modules, but how we call them in main.tf which is in parent folder. Let’s see this

main.tf:

 provider "aws" {
   region = var.region
   access_key = var.aws_access_key_id
   secret_key = var.aws_secret_access_key
 }

 module "ec2_instance" {
   source = "./modules/ec2-instance"
   ami = "ami-09b0a86a2c84101e1"
   instance_type = var.instance_type
   security_group_ids = [module.ec2_sg.security_group_id]
 }

 module "ec2_sg" {
   source = "./modules/security-group"
   sg_name = "EC2-securitygroup"
 }

 output "instance_id" {
   value = module.ec2_instance.instance_id
 }

 output "public_ip" {
   value = module.ec2_instance.public_ip
 }

 output "security_group_id" {
   value = module.ec2_sg.security_group_id
 }

1. In this main.tf file we had first block of provider which will install aws dependencies then there are 2 blocks of module which contain source which is calling the ec2-instance module and same for security group and at last there are 3 blocks of output.
2. GitHub Code: https://github.com/amitmaurya07/AWS-Terraform/tree/master
3. Okay, we discussed a lot now let’s do the creation of AWS EC2 instance.
4. First initialize the terraform by executing the command. It will download the plugins of AWS and store in .terraform folder and also creates modules folder which contains the module.json

terraform init

Now execute to plan the infrastructure by executing which will ask the value of Type of instance and AWS Region as we had declared in the variables.tf file in parent folder.

terraform plan

1. While executing terraform plan it prints the result that you are going to create on AWS.
2. At last we will create the EC2 instance and security group on AWS by executing

terraform apply --var-file="terraform.tfvars"

In terraform.tfvars file I declared the access key and secret key which is not pushed in the GitHub for security purposes.

When we type “yes” it will start creating the EC2 instance and Security Group on AWS that will print the output which we had given in the output file.

Let’s copy and paste the Public IP of instance and see the “Welcome to Nginx“ page. Navigate to http://<Your Public IP>

After this do the most important thing clean all the resources on AWS Just execute the command.

terraform destroy --var-file="terraform.tfvars"

As we can see the line “Destroy complete! Resources: 2 destroyed.“ that means the resources are deleted. See so simple just execute one command and destroys all in seconds.

Conclusion

In this article, we had learnt the creation of EC2 instance and security group with terraform. Meanwhile in the upcoming blogs you will see a lot of AWS Services creation and automating it with terraform. Stay tuned for the next blog !!!

GitHub Code : https://github.com/amitmaurya07/AWS-Terraform/tree/master

Twitter : x.com/amitmau07

LinkedIn : linkedin.com/in/amit-maurya07

If you have any queries you can drop the message on LinkedIn and Twitter.

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter | Podcast
• Create a free AI-powered blog on Differ.
• More content at PlainEnglish.io

How to Provision AWS EC2 Instances with Terraform: A Step-by-Step Guide was originally published in AWS in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.