In real-world AWS environments, organizations rarely operate from a single account. A common pattern is to have a dedicated Identity Account or Management Account where all users live, and separate accounts for different environments sandbox, staging, production with limited direct access.

This guide walks through setting up cross-account role switching so a user in the Identity Account can switch into a Sandbox account using the AWS console or CLI, without needing separate credentials for each account.

How it works: AWS Security Token Service (STS) lets a principal in Account A call sts:AssumeRole for a role defined in Account B. The role’s trust policy explicitly allows this. No passwords are shared only short-lived temporary credentials are issued, valid for the session duration.

Prerequisites

Account Details AWS Console accessIdentityAdmin or IAM permissions to create usersAWS Console accessSandboxAdmin permissions to create IAM rolesAccount IDsBoth12-digit account numbers for both accounts Browser

For console-based role switching

Step 1

Create an IAM User in the Identity Account

This user will authenticate here, then switch into other accounts

Log into the Identity Account (account 183295435445) and navigate to IAM → Users → Create user.

• Go to IAM → Users in the AWS Console and click Create user.
• Enter a username such as dev-user. Enable AWS Management Console access and set a password. Optionally require a password reset on first login.
• On the Set permissions page, you can attach a minimal policy now. We’ll scope it further using sts:AssumeRole below.
• Review and click Create user. Note the console sign-in URL and credentials.

Attach the following inline policy to this IAM user (or to a group the user belongs to) so they can assume the role in the Sandbox account:

JSON — Allow sts:AssumeRoleCopy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSandboxRoleSwitch",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::987654321987:role/cross-account-role"
    }
  ]
}

Replace 185863138492 with your Sandbox account’s 12-digit ID, and cross-account-role with the exact role name you’ll create in Step 2. These must match exactly.

Best practice: Create an IAM group (e.g. sandbox-switchers) with this policy and add users to it. This scales easily as your team grows.

Step 2

Create the Cross-Account Role in the Sandbox Account

This role trusts the Identity Account and grants permissions when assumed

Go to IAM → Roles → Create role.

• Under Trusted entity type, select AWS account (not AWS service or Web identity).
• Select Another AWS account and enter the Identity Account’s 12-digit ID: 183295435445.
• Leave Require external ID unchecked for internal accounts. Only enable it when a third-party vendor will assume this role.
• Click Next.

• On the Add permissions page, search for and attach the policies this role should carry. For a sandbox, AdministratorAccess is common. For production, scope it tightly.
• On the final page, set the Role name to cross-account-role and optionally add a description like “Assumed by Identity Account users for sandbox access.”
• Click Create role.

Use descriptive names like identity-sandbox-admin-role in production environments. This makes it easy to identify roles across multiple accounts in AWS Organizations, especially in CloudTrail logs.

Step 3

Verify the Trust Policy & Copy the Switch Role Link

AWS generates a pre-filled URL that makes switching frictionless

After creating the role, open it at IAM → Roles → cross-account-role. Click the Trust relationships tab and confirm the trust policy was set up correctly.

The auto-generated trust policy should look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789123:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

Using :root as the Principal trusts the entire Identity Account. Any IAM entity in that account with the sts:AssumeRole permission pointing to this role ARN can assume it. For tighter control, replace :root with a specific user ARN: arn:aws:iam::123456789123:user/dev-user.

Now, from the role Summary section (top of the page), locate Link to switch roles in console and copy it:

URL — Switch Role Console LinkCopy

https://signin.aws.amazon.com/switchrole?roleName=cross-account-role&account=123456789123

Share this link with the IAM user you created in Step 1, or bookmark it for yourself. When clicked while logged into the Identity Account, it pre-fills the switch role dialog.

Step 4

Log In as the IAM User and Switch to the Sandbox Account

Complete the loop — test end-to-end in the console and CLI

Log out of any current AWS session and log in as the IAM user from the Identity Account.

• Navigate to the Identity Account sign-in URL: https://987654321987.signin.aws.amazon.com/console
• Log in with the IAM user credentials you created in Step 1 (username + password).
• Paste the switch role URL from Step 3 into your browser, or click your account name in the top-right corner of the console and select Switch role.
• In the dialog, enter: Account = 987654321987, Role = cross-account-role. Optionally set a display name (e.g. Sandbox) and a color for the badge.
• Click Switch Role. You’re now operating in the Sandbox account under the assumed role with temporary credentials.

The top-right account badge changes to show the role name and account alias. Recent role switches are remembered, so future switches take just two clicks. Click Back to [username] to return to the Identity Account.

If you see “Not authorized to assume role”: verify (1) the IAM user has an sts:AssumeRole policy pointing to the correct role ARN, and (2) the trust policy in the Sandbox role has the correct Identity Account ID.

You can also switch roles via the AWS CLI by adding a profile to your config:

~/.aws/configCopy

# Base profile for the Identity Account IAM user
[profile identity]
aws_access_key_id     = AKIA...YOUR...KEY
aws_secret_access_key = your-secret-key
region                = us-east-1

# Profile that assumes the Sandbox role
[profile sandbox]
role_arn              = arn:aws:iam::123456789123:role/cross-account-role
source_profile        = identity
region                = us-east-1

Shell — Verify the role switch via CLICopy

# Use the sandbox profile for any AWS CLI command
aws sts get-caller-identity --profile sandbox

# Expected output confirms you are in the Sandbox account
{
    "UserId":  "AROAEXAMPLEID:session-name",
    "Account": "123456789123",
    "Arn":     "arn:aws:sts::123456789123:assumed-role/cross-account-role/session-name"
}

Cross-Account Access Configured

You’ve set up a secure, credential-free way for IAM users to switch between AWS accounts using temporary STS credentials. This pattern scales across many accounts in AWS Organizations, repeat Steps 2–3 for each target account and share switch role links with your team.

Instead of signing up individually for different AI companies, OpenRouter gives you a single API that can route your request to many popular models.

That means you can test models, compare responses, switch providers, and build apps faster.

In this guide, I’ll walk through:

• How to sign up and log in
• How to create an API key
• Where to find free models
• How to use OpenRouter in Python code
• Tips for beginners

What is OpenRouter?

OpenRouter is an API gateway that gives developers access to many AI models through one endpoint.

Instead of using different APIs for:

• OpenAI
• Google
• Anthropic
• Meta

You can use one OpenRouter API key.

This is especially useful for developers who want to experiment with multiple models.

Step 1: Create an OpenRouter Account

Go to the official website:

openrouter.ai

Then click Sign In.

You’ll usually see login options like:

• Google login
• GitHub login
• Email login (if available)

Use whichever is easiest for you.

After login, you’ll enter the dashboard.

Step 2: Create an API Key

Once logged in:

1. Open Dashboard
2. Go to Keys or API Keys section
3. Click Create Key
4. Give it a name
5. Copy and save it safely

Example:

sk-or-v1-xxxxxxxxxxxxxxxx

Important:

Never expose your API key in public GitHub repositories.

Step 3: Find Free Models

One of the best things in OpenRouter is free community models.

Examples often available:

• Google Gemma free models
• Meta Llama free variants
• Mistral AI free models
• Small open-source instruct models

Look for models with:

:free

Examples:

google/gemma-3-4b-it:free
meta-llama/llama-3.2-3b-instruct:free
mistralai/mistral-small:free

These are great for learning and testing.

Step 4: Install Python Library

Use OpenAI-compatible SDK.

pip install openai

Step 5: Python Example Using OpenRouter

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_OPENROUTER_KEY",
    base_url="https://openrouter.ai/api/v1"
)

response = client.chat.completions.create(
    model="google/gemma-3-4b-it:free",
    messages=[
        {"role": "user", "content": "Explain Docker in simple words"}
    ]
)

print(response.choices[0].message.content)

Why Developers Like OpenRouter

1. One API for Many Models

No need to manage multiple providers.

2. Easy Model Testing

Switch models by changing one line:

model="google/gemma-3-4b-it:free"

to

model="meta-llama/llama-3.2-3b-instruct:free"

3. Budget Friendly

Free models available for experiments.

4. Great for Side Projects

Useful for:

• Chatbots
• Resume analyzers
• Code assistants
• Summarizers
• AI tools

Tips

Use Environment Variables

Instead of hardcoding keys:

import os
api_key = os.getenv("OPENROUTER_API_KEY")

Handle Rate Limits

Some free models are busy during peak time.

If one model fails, switch to another free model.

Compare Output

Different models give different quality.

When building secure cloud architectures, encryption at rest is not optional, it’s foundational.

While Azure automatically encrypts managed disks using platform-managed keys, many organizations require Customer-Managed Keys (CMK) for compliance, regulatory, or security control reasons.

In this article, we’ll walk through an end-to-end process to encrypt an existing Azure VM disk using:

• Azure Key Vault
• Customer-managed encryption key
• Disk Encryption Set
• Existing Virtual Machine

Understanding the Architecture:

Before jumping into configuration, let’s understand how Azure disk encryption with CMK works.

Azure does not encrypt the disk directly using your Key Vault key.

Instead, it uses a layered model:

1. Azure generates a Disk Encryption Key (DEK).
2. Your Key Vault key encrypts (wraps) that DEK.
3. The wrapped DEK is stored securely.
4. When the VM starts, Azure retrieves and unwraps the key.

Architecture flow:

Step 1: Create a Key Vault

Go to the Azure Portal and create a Key Vault.

Important configuration settings:

• Enable Soft Delete
• Enable Purge Protection
• Ensure region matches your VM region
• Assign a role based access control

• Configure networking (Public or Private based on your architecture)

Note: If using Private Endpoint, ensure network connectivity is properly configured. Even you cannot access through azure portal since your browser sending request with public IP

• If your creating private key vault below steps, if not please ignore below steps, make it public and continue with review and create
• select vnet, location and subnet where your vm is deployed

After creation, confirm the vault is successfully deployed.

Important: Assign yourself key vault administrator role

Step 2: Create an Encryption Key

Inside your Key Vault:

1. Go to Keys
2. Click Generate/Import
3. Select: RSA, 2048 or 3072 bits
4. Provide a name (e.g., vm-disk-key)
5. Create the key

Ensure the key status shows Enabled.

Step 3: Create a Disk Encryption Set (DES)

Next, create a Disk Encryption Set.

Search for Disk Encryption Sets in the Azure Portal and click Create.

Configuration:

• Same region as VM and Key Vault
• Encryption type: Customer-managed key
• Select your Key Vault
• Select your key
• Select a specific key version (recommended)

Why Disk Encryption Set?

It acts as a bridge between your disk and your Key Vault key, using a managed identity to access the key securely.

🔐 Step 4: Grant Key Permissions to the Disk Encryption Set

This is the most critical step.

The Disk Encryption Set has a system-assigned managed identity. That identity must have permission to use the key.

Go to:

Key Vault → Access Control (IAM) → Add Role Assignment

Assign:

Key Vault Crypto Service Encryption User

Assign this role to:

• The managed identity of the Disk Encryption Set

Simple way: Go to disk encryptionset and just click on top up. Role will be assinged automatically

Without this permission, encryption will fail.

Step 5: Stop the Virtual Machine

Before changing disk encryption settings:

1. Go to your Virtual Machine
2. Click Stop
3. Wait until it shows Stopped (deallocated)

This ensures safe modification of the disk configuration.

Step 6: Attach Disk Encryption Set to Existing Disk

Now we modify the disk.

For OS Disk:

1. Go to VM → Disks
2. Click the OS disk
3. Open Configuration
4. Under Encryption:

• Select Customer-managed key
• Choose your Disk Encryption Set
• Save changes

For Data Disk:

Repeat the same process for each data disk.

Step 7: Start the VM

Go back to your VM and click Start.

If everything is configured correctly:

• VM starts successfully
• No permission errors
• No key access errors

Behind the scenes, Azure:

• Retrieves your key
• Unwraps the disk encryption key
• Mounts the disk securely

When you deploy a Pod in Kubernetes, have you ever wondered how it decides which node to run on? At first point, it might seem like magic. You apply a YAML file, and a few moments later, your Pod is running somewhere in your cluster.

But under the hood, Kubernetes follows a well-defined decision-making process to find the most suitable node. This decision isn’t random. It involves intelligent filtering, scoring, and binding. Especially when you use constraints like a nodeSelector.

In this blog, we’ll walk through what really happens step-by-step when you apply a Deployment with a nodeSelector. We'll uncover how the Kubernetes scheduler evaluates nodes, shortlists the ones that match, scores them based on multiple factors, and finally assigns your Pod to the best possible location.

Whether you’re debugging a pending Pod or trying to control Pod placement more precisely, understanding this flow will help you use Kubernetes more effectively.

Node Labeling:

Lets assume you labeled a node with key=disktype value=ssd

Label a node using simple command kubectl label node <nodeName> disktype=ssd

once you labeled, describe your node (kubectl describe no <nodeName>). You should label like this:

Pod Scheduling:

Lets assume, you defined nodeSelector as shown in below deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: node-selector
  name: node-selector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: node-selector
  strategy: {}
  template:
    metadata:
      labels:
        app: node-selector
    spec:
      nodeSelector:
        disktype: ssd
      containers:
      - image: nginx
        name: nginx
        resources: {}

Apply the manifest file using kubectl apply -f <fileName>

Now… the actual kubernetes intelligence comes in..

1. API Server Interaction

• kubectl sends the YAML manifest to the Kubernetes API Server (the entry point of the control plane).

1.1 The API Server:

• Authenticates and authorizes your request.
• Validates the schema of the Deployment object.
• Stores the Deployment definition in etcd, the Kubernetes backing store (key-value database).
• Triggers relevant informers and controllers (specifically the Deployment controller).

Deployment Controller (part of kube-controller-manager)

2. Deployment → ReplicaSet

• The Deployment controller sees a new Deployment and:
• Creates a ReplicaSet that matches the Deployment’s spec.
• The ReplicaSet defines the desired number of Pod replicas with the given Pod template (including the nodeSelector).

3. ReplicaSet → Pods

• The ReplicaSet controller creates the required number of Pods.
• These Pods include the nodeSelector in their .spec.

Kubernetes Scheduler

What is the Scheduling Cycle?

When a Pod is created, it doesn’t immediately run on a node. It needs to be scheduled. That is, assigned to a node in the cluster.

The Kubernetes Scheduler decides which node the Pod should run on, and it does that in two steps:

Step 4.1: Filtering Phase — Who Can Host the Pod?

In this step, Kubernetes checks which nodes are eligible to run the Pod.

What is Filtering?

Filtering is like shortlisting nodes that:

• Have enough CPU and memory.
• Have matching labels (nodeSelector, affinity, etc.).
• Are Ready (not down or under maintenance).
• Are not tainted in a way that would block this Pod.

Example:

You create a Pod with:

nodeSelector:
  disktype: ssd

Imagine your cluster has 3 nodes:

+-----------+----------------+-------------+-------+
| Node Name | disktype Label | Free Memory | Ready |
+-----------+----------------+-------------+-------+
| node-a    | ssd            | 4Gi         | ✅    |
| node-b    | hdd            | 8Gi         | ✅    |
| node-c    | ssd            | 0.5Gi       | ✅    |
+-----------+----------------+-------------+-------+

Filtering checks:

• node-a: OK (label matches, enough memory, Ready)
• node-b: NO (label does not match)
• node-c: NO (not enough memory)
• Only node-a passes the filter.

Step 4.2: Scoring Phase — Who Is the Best Host?

Now Kubernetes scores the filtered nodes to find the best one.

Even if multiple nodes pass filtering, Kubernetes wants to pick the most optimal node based on:

• How much free resources they have (e.g., avoid overloading a node).
• Pod affinity/anti-affinity rules.
• Topology (spread Pods across zones/hosts).
• Custom policies.

Why Score?

To balance the load and use resources smartly.

Example Scoring :

Say 2 nodes passed filtering: node-a and node-d.

+--------+----------+-------------+-------+
| Node   | CPU Free | Free Memory | Score |
+--------+----------+-------------+-------+
| node-a | 4 core   | 4Gi         | 70    |
| node-d | 8 cores  | 10Gi        | 90    |
+--------+----------+-------------+-------+

Scheduler picks node-d (highest score) and assigns the Pod to it.

Final Step 4.3: Binding

Once the best node is selected:

• The scheduler tells the API Server: “Bind this Pod to node-d”.

Kubelet on the Target Node

5. Kubelet Pulls and Starts Pod

• The kubelet on node-d:
• Watches the API server for Pods assigned to it.
• Sees the new Pod with matching nodeName.
• Pulls the container images via CRI (Container Runtime Interface) — e.g., containerd, dockershim, or CRI-O.
• Mounts volumes, sets up networking (via CNI), and starts containers.

Runtime Status

6. Pod Transitions Through States

• API server reflects changes to Pod status:
• Pending → image pulled → ContainerCreating → container started → Running.

Overall flow of scheduling a pod on specific node

kubectl apply
     ↓
API Server → store in etcd → notify Deployment controller
     ↓
Deployment → ReplicaSet → Pod
     ↓
Pod has `nodeSelector`
     ↓
Scheduler filters nodes → scores nodes → binds Pod to Node
     ↓
Pod assigned → Kubelet on node → container runtime pulls and starts containers
     ↓
Pod status updates → Running

In this blog we will just going to discuss about how to set up kubernetes cluster using kubeadm. All you have to do is, just follow the steps

In my case, I used AWS ubuntu instances. you can also use this blog for on-prim servers or any other cloud vms. You just have to make sure that all your nodes communicate each other on ports 6443 and 10250 port.

Before you begin:

• 2 GB or more of RAM per machine (any less will leave little room for your apps)
• 2 CPUs or more.
• Full network connectivity between all machines in the cluster (public or private network is fine).
• Unique hostname, MAC address, and product_uuid for every node using command cat /sys/class/dmi/id/product_uuid
• You MUST disable swap if the kubelet is not properly configured to use swap. use sudo swapoff -a command to disable the swap memory

Installing a container runtime

• This part should perform on every node(master and worker nodes)
• By default, the Linux kernel does not allow IPv4 packets to be routed between interfaces. so lets enable IPv4 packet forwarding

echo net.ipv4.ip_forward = 1 | sudo tee /etc/sysctl.d/k8s.conf

# Apply sysctl params without reboot
sudo sysctl --system

• Verify that net.ipv4.ip_forward is set to 1 with:

sysctl net.ipv4.ip_forward

Step 1: Installing containerd

• Download and setup the containerd on each node using below command

# refere this link for different version of containerd https://github.com/containerd/containerd/releases
wget https://github.com/containerd/containerd/releases/download/v2.0.0/containerd-2.0.0-linux-amd64.tar.gz

# extract the files in /usr/local folder
tar Cxzvf /usr/local containerd-2.0.0-linux-amd64.tar.gz

#create a systemd directory for the containerd
mkdir -p /usr/local/lib/systemd/system/

#create a file and past the below content in this file
vi /usr/local/lib/systemd/system/containerd.service

# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target dbus.service

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

#run the below commands 
systemctl daemon-reload

systemctl enable --now containerd

Step 2: Installing runc

• Download the runc.<ARCH> binary from https://github.com/opencontainers/runc/releases according your system configuration it may be amd64 or arm64. In my case I am using amd64

#run the below commands to install runc
wget https://github.com/opencontainers/runc/releases/download/v1.2.1/runc.amd64

install -m 755 runc.amd64 /usr/local/sbin/runc

Step 3: Installing CNI plugins

• Download the cni-plugins-<OS>-<ARCH>-<VERSION>.tgz archive from https://github.com/containernetworking/plugins/releases , verify its sha256sum, and extract it under /opt/cni/bin

wget https://github.com/containernetworking/plugins/releases/download/v1.6.0/cni-plugins-linux-amd64-v1.6.0.tgz

mkdir -p /opt/cni/bin

tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.6.0.tgz

Configuring the systemd cgroup driver

• Run the following command to create a default configuration file:

sudo containerd config default > /etc/containerd/config.toml

• To use the systemd cgroup driver in /etc/containerd/config.toml with runc, set as shown below images

• as shown in the above image add SystemdCgroup = true in config.toml file
• Once add enabled the SystemdCgroup run sudo systemctl restart containerd

Overriding the sandbox (pause) image

• In your containerd config you can overwrite the sandbox image by setting the following config:
• There is aleardy everything set. You just need to add sandbox_image = “registry.k8s.io/pause:3.2” line in it

• You might need to restart containerd as well once you've updated the config file: systemctl restart containerdInstalling kubeadm, kubelet and kubectl

Installing kubeadm, kubelet and kubectl

• You will install these packages on all of your machines:
• kubeadm: the command to bootstrap the cluster.
• kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.
• kubectl: the command line util to talk to your cluster.
• These instructions are for Kubernetes v1.30.

#Update the apt package index and install packages needed to use the Kubernetes apt repository:
sudo apt-get update

sudo apt-get install -y apt-transport-https ca-certificates curl gpg

#Download the public signing key for the Kubernetes package repositories. The same signing key is used for all repositories so you can disregard the version in the URL:
# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

#Update the apt package index, install kubelet, kubeadm and kubectl, and pin their version:
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

#Enable the kubelet service before running kubeadm:
sudo systemctl enable --now kubelet

Initializing your control-plane node

• These steps needs to be perform only on master node
• To initialize the control-plane node run kubeadm init
• Once you run the kubeadm init the below information will be shown. Follow the instruction which have been showed after executing the kubeadm init

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  /docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

• note down the kubeadm join whole command and paste it somewhere. use that command to add a worker node to your cluster.
• whenever you want to add a worker node, simply have network connectivity between master and node install the required packages as shown in previous steps and run this command(which you have noted)in worker node
• make sure you stopped kubelete in worker node before running the kubeadm join command. once the command executed, start again the kubelete
• Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

• Installing a Pod network add-on

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

• restart the kubelete in all servers using systemctl restart kubelete
• list the nodes and try to launch a pod.

Here we goooo cluster set up is done

In this story we will be going to install Hashicorp-Vault in kubernetes with PostgreSQL as a backend and also we will be looking at how to mount or pass secrets which are in vault, to the kubernetes pod.

Lets have a basic idea on Hashicorp-Vault, vault backend and External secret Operator.

Hashicorp-Vault: A vault server which stores and manages the secrets like db credentials, tokens, certificates and more. Vautl server is also does data encryption and data management by using secret engines like AWS Secret Manager, MYSQL, KV Secret Engine and many more. We will be using KV Secret Engine in this blog.

Vault backend: A backend is nothing but, a storage which stores the state of vault server. For example you already installed and configured your vault server, suddenly your vault server goes down due to some reason, your data may be lost which is there in vault. To overcome this situation, there is a consept called backend. When your vautl server goes down, you want to launch again with the data that was present in previous vault, simply you just have to configure your previous backend, so that your vault server will launched with previous data that is present in backend. In this blog, we will be using PostgreSQL as a backend.

External secret Operator(ESO): ESO is a kubernetes operator which is used to get the secrets from vault server and pass that secret to kubernetes pod. As of now, simply we can assume that, it is a mediator for Vault server and kubernetes pod.

Pre-requisites:

• Kubernetes Cluster
• PostgreSQL Database
• Kubectl and Helm configured with cluster
• jq package: You can use the given command based on your OS to install the JQ tool

For Debian/Ubuntu-based systems:-
sudo apt-get update
sudo apt-get install jq

For Mac Systems:-
brew install jq

For Fedora/RHEL-based systems:-
sudo dnf install jq

PostgreSQL Database setup

• Once your database is ready, connect to the database and run below schema in your database.

CREATE TABLE vault_kv_store (
  parent_path TEXT COLLATE "C" NOT NULL,
  path        TEXT COLLATE "C",
  key         TEXT COLLATE "C",
  value       BYTEA,
  CONSTRAINT pkey PRIMARY KEY (path, key)
);

CREATE INDEX parent_path_idx ON vault_kv_store (parent_path);

CREATE TABLE vault_ha_locks (
  ha_key                                      TEXT COLLATE "C" NOT NULL,
  ha_identity                                 TEXT COLLATE "C" NOT NULL,
  ha_value                                    TEXT COLLATE "C",
  valid_until                                 TIMESTAMP WITH TIME ZONE NOT NULL,
  CONSTRAINT ha_key PRIMARY KEY (ha_key)
);

• vault_kv_store: Table name, you can use name of your choice
• vault_ha_locks: Table name, you can use name of your choice.
• Once you run above schema, then your database ready to use with the vault server. Before that, vault server required your database connectionURL to access your database.
• ConnectionURL is look something like this below

postgresql://<USERNAME>:<PASSWORD>@<HOST>:5432/<DB_NAME>?sslmode=disable

• Once you get your connection URL, note it somewhere we will going to use that connection url in vault installation.

Hashicorp-Vault Installation in Kubernetes

NOTE: Vault pods has to access database, configure your cluster to have access to database. if your using AWS Cloud, Attach neccessary permissions to node group or vault pod by service account

• Create file called values.yaml and paste the below content in the file

injector:
  # True if you want to enable vault agent injection.
  enabled: false

server:

  # Affinity Settings
  # Commenting out or setting as empty the affinity variable, will allow
  # deployment to single node services such as Minikube
  affinity: null
  
  # Run Vault in "standalone" mode. This is the default mode that will deploy if
  # no arguments are given to helm. This requires a PVC for data storage to use
  # the "file" backend.  This mode is not highly available and should not be scaled
  # past a single replica.
  standalone:
    enabled: false

  # Run Vault in "HA" mode. There are no storage requirements unless audit log
  # persistence is required.  In HA mode Vault will configure itself to use Consul
  # for its storage backend.  The default configuration provided will work the Consul
  # Helm project by default.  It is possible to manually configure Vault to use a
  # different HA backend.
  ha:
    enabled: true
    # config is a raw string of default configuration when using a Stateful
    # deployment. Default is to use a Consul for its HA storage backend.
    # This should be HCL.

    # Note: Configuration files are stored in ConfigMaps so sensitive data
    # such as passwords should be either mounted through extraSecretEnvironmentVars
    # or through a Kube secret.  For more information see:
    # https://www.vaultproject.io/docs/platform/k8s/helm/run#protecting-sensitive-vault-configurations
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "postgresql" {
        connection_url="<ConnectionURL that we already got it in STEP 1>"
        table="vault_kv_store",
        ha_enabled=true,
        ha_table="vault_ha_locks" 
      }
      service_registration "kubernetes" {}

• Execute the below commands to install the vault

helm repo add hashicorp https://helm.releases.hashicorp.com

helm repo update

helm search repo vault --versions

helm install vault hashicorp/vault -f values.yaml -n vault

• Verify the vault pods using command kubectl get pods -n vault, it should look something like below image.

• To Initialise and unseal your vault server pods follow the below steps.

a) execute the following command kubectl exec vault-0 -n vault -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json

b) you’ll notice that cluster-keys.json file has been created in your system (NOTE: keep it safe)

c) Display the unseal key found in cluster-keys.json using following command cat cluster-keys.json | jq -r ".unseal_keys_b64[]" copy the output of command and keep it safe somewhere.

d) Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key using following command VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")

e) Unseal Vault running on the vault-0 pod using following command kubectl exec vault-0 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY

f) Retrieve the status of Vault on the vault-0 pod using following command kubectl exec vault-0 -n vault -- vault status

• To Join the other Vault pods to the Vault cluster execute the below commands one by one.

a) CLUSTER_ROOT_TOKEN=$(cat cluster-keys.json | jq -r ".root_token")

b) kubectl exec vault-0 -n vault -- vault login $CLUSTER_ROOT_TOKEN

c) Unseal the other two vault pods using below commands so that they will automatically join with the vault master pod

kubectl exec vault-1 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY

kubectl exec vault-2 -n vault -- vault operator unseal $VAULT_UNSEAL

d) Once you done, verify all the vault pods using following command kubectl get pods -n vault output should look like below image

• Access Vault server

a) To access our vautl server first we need have a root key to login, extract the root key from cluster-keys.json file using following command cat cluster-keys.json | jq -r ".root_token" once you run the command, you’ll get root key. keep it safe and do not share with anyone.

b) Expose the vault server using following command kubectl port-forward —namespace vault svc/vault 8200:8200 output shoud look like this Or you can expose vault via proxy/LB so that you can access it directly from your browser

c) keep this window exposing vault, if you want to execute any command launch an other command window and use. Or you can also expose vault by attaching service type NodePort, LoadBalancer or any other way.

d) open your browser and access vault by search as follow localhost:8200 output will be like below image

e) Enter the root key and login. To get root key, excute this command cat cluster-keys.json | jq -r ".root_token"

• Once you login as root with token enable the kv secret engine as shown in below images

• Leave all fields as default and click on Enable engine.

Install External Secret Operator

• Install the External Secret Operator using helm. To install ESO use the following helm commands

  helm repo add external-secrets https://charts.external-secrets.io

  helm install external-secrets \
     external-secrets/external-secrets \
     -n external-secrets \
     --create-namespace

• Verify the external secrets installation by running the following command kubectl get pods,crd -n external-secrets
• Create the secret using following command kubectl create secret -n vault generic vault-token --from-literal=token='your-copied-root-token' (NOTE: you can also create a specific user, attach necessary policies and you can use that user token to access these secrets)
• Once you create secret, verify using command kubectl get secrets -n vautl you should be able to see your secret.
• Create a SecretStore. To do this, we create a file called secret-store.yml with the following content:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore # You could also choose ClusterSecretStore 
metadata:
  name: secret-store-vault
spec:
  provider:
    vault:
      server: "http://vault.vault:8200" #https://your-vault-server:8200
      # Version is the Vault KV secret engine version.
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          namespace: "vault" # secrets namespace that we create with root token
          name: "vault-token" # secrets that we create with root token
          key: "token

• Once you create the above file apply the file using command kubectl apply -f secret-store.yaml
• Once you apply the file, verify secret store is created or not kubectl get secretstore -n eso you should be see your secretstore.
• Describe the secretstore to verify if it successfully working or not using command kubectl describe secretstore secret-store-vault -n eso and should see as shown in below

• Till now, our set up is done. Now, lets create secret in vault, get the secret and mount to the pod

Create Secret In Vault, Get the Secret and Mount to the Pod

• Login using the root token and create a secret. go to kv/ path click on Create secret→ enter path key value pairs

• Then create the externalsecret with kubectl apply -f external-secret.yml.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-github
  namespace: vault
spec:
  refreshInterval: "15s" # The secrets will be refreshed every 15 seconds
  secretStoreRef:
    name: secret-store-vault # Reference to the secret store we have created above
    kind: SecretStore 
  target:
    name: my-secret # K8S secret name
  data:
    - secretKey: MY_TOKEN # K8S secret key in the secret
      remoteRef:
        key: kv/db-pass # Path to the vault secret
        property: password # Property name of the secret 

• Let’s check again if everything is working properly: kubectl get externalsecret -n eso you should see as shown in the below image.

• Once externalsecret working fine, there should be one new secret created in the namespace with the name of my-secret. verify using command kubectl get secrets -n vault
• Here you go…… verify if your vault secret is present in this secret or not using the command kubectl get secret my-secret -n vault -o jsonpath='{.data}' you should get response as shown in the below

• Now copy the MY_TOKEN value and decode it via online base 64 decoder or base64 CLI. You should be able to see the original value that is there in vault secret.
• Try to change the value in vault server, you can see the change here in the secret. (This is the beauty of ESO)

Create a pod and pass your secrets as environment variables

• create a my-pod.yaml and paste the below content

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: vault
spec:
  containers:
    - image: jweissig/app:0.0.1
      name: test-app
      env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: my-secret
            key: MY_TOKEN

• create the pod using command kubectl apply -f my-pod.yaml
• exec into the pod kubectl exec -it mypod -n vault -- bash
• check your secret value echo $DB_PASSWORD

END

We had a requirment is that, Hashicorp Vault server should be there in kubernetes cluster, should be managed various components like Authentication,authorization, Secret Engine and many more. Vautl secrets should be consumed by kubernetes pods. So, to fulfill this requirment, we choose to have the KubeVault Operator with Hashicorp Vault, CSI Driver with Vault provider.

PART: A (Basic Theory about KubeVault Operator)

If you don’t want to read theory part, you can skip and start with the PART: B which is handson practicle

Lets understand some basics and will go with the practicle thing.

1. What is KubeVault Operator ?

KubeVault Operator is an kubernetes controller for the Hashicorp Vault. Kubevault operator manages the vault server. Managing in the sence, like authentication, authorization, vault client and server communication, Vault’s Secret Engine and many more. Ideally, We can also deploy Hashicorp Vault in kubernetes without using of KubeVault Operator but, it is very defficult to manage the vault server.

2. Why to use KubeVault ?

KubeVault operator makes it easy to deploy, maintain and manage Vault servers in Kubernetes. It covers automatic initialization and unsealing, and securely stores unseal keys and root tokens in a cloud KMS (Key Management Service) service. It provides the following features:

• Deploy TLS Secured Vault Server
• Manage Vault Server TLS using Cert-manager
• Automate Initialization & Unseal process of Vault Servers
• Add Durability to Vault’s Data using Storage Backend
• Enable & Configure Secret Engines
• Create & Configure Vault Roles
• Manage Vault Policy & Vault Policy Binding
• Manage user privileges using SecretAccessRequest
• Manage user privileges using SecretRoleBinding
• Inject Vault secrets into K8s resources
• Automate tedious operations using KubeVault CLI
• Monitor Vault using Prometheus & Grafana Dashboard

PART: B (Practicle with KubeVult Operator)

Step: 1 Generating the license

• Before installation, Make sure that you have kubectl and helm packages and should be configured to your kubernetes cluster
• To install KubeVault, first we need to have the license. So first we need to generate the license.
• To get the license go to KubeVault official website and generate the license https://kubevault.com/docs/v2024.3.12/setup/install/kubevault/

• As shown in the above image, enter the details. For cluster ID run the give command kubectl get ns kube-system -o=jsonpath=’{.metadata.uid}’ once your license recieved to your mail keep it safe.
• Note: The license is valid only for month, if you want to implement it in production, you have to buy the license from AppsCode.com

Step: 2 KubeVault Operator Installation in kubernetes

• Install the KubeVault operator using helm

helm install kubevault oci://ghcr.io/appscode-charts/kubevault \ 
 --version v2024.3.12 \ 
 --namespace kubevault --create-namespace \ 
 --set-file global.license=/path/to/the/license.txt

• To check if KubeVault operator pods have started, run the following command kubectl get pods --all-namespaces -l "app.kubernetes.io/instance=kubevault

• Now, to confirm CRD groups have been registered by the operator, run the following command kubectl get crd -l app.kubernetes.io/name=kubevault

Step: 3 CSI Driver and Vault Provider Setup

• Install the CSI Driver using the helm commands

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --set "syncSecret.enabled=true" \
  --set "enableSecretRotation=true" \
  --set "rotation-poll-interval=15s" \
  --namespace kube-system

• Once you run the helm commands, verify with the following command kubectl get pods -n kube-system -l "app=secrets-store-csi-driver

• Install the Vault Provider using helm

helm repo add hashicorp https://helm.releases.hashicorp.com

helm install vault hashicorp/vault \
  --set "server.enabled=false" \
  --set "injector.enabled=false" \
  --set "csi.enabled=true"

• Once you executed successfully, verify the provider pods using command kubectl get pods -l "app.kubernetes.io/name=vault-csi-provider"

Step: 4 Vault Server Installation

• Create a namespace for isolation using command kubectl create ns demo
• Verify vault server version crds using command kubectl get vaultserverversions you should see like the below image.

• Configure or install vaultServerVersion, Create a vault-server-version.yaml and paste the below yaml configuration in it

apiVersion: catalog.kubevault.com/v1alpha1
kind: VaultServerVersion
metadata:
  name: 1.10.3
spec:
  exporter:
    image: kubevault/vault-exporter:v0.1.1
  unsealer:
    image: kubevault/vault-unsealer:v0.8.0
  vault:
    image: vault:1.10.3
  version: 1.10.3

• Then apply the yaml file using kubectl apply -f vault-server-version.yaml
• Install Vault Server, Create a vault-server.yaml file and paste the below configuration in it.

apiVersion: kubevault.com/v1alpha2
kind: VaultServer
metadata:
  name: vault
  namespace: demo
spec:
  replicas: 1
  version: 1.10.3
  serviceTemplates:
  - alias: vault
    metadata:
      annotations:
        name: vault
    spec:
      type: NodePort
  backend:
    inmem: {}
  unsealer:
    secretShares: 4
    secretThreshold: 2
    mode:
      kubernetesSecret:
        secretName: vault-keys

• In the above configuration file, spec.backend, I used inmemory as per testing. You can use other backend to persist the vault server data as following the link https://kubevault.com/docs/v2024.3.12/concepts/vault-server-crds/vaultserver/ Once you open the link scroll down to the spec.backend and click on any backend that you want to configure.

• Change the spec.backend configuration as per your requirement and apply the vault-server.yaml using command kubectl apply -f vault-server.yaml
• Verify vault server’s state using command kubectl get vaultserver -n demo It should in ready state as shown in below image (NOTE: it will take 2 or 3 minutes)

• Once, the vaultserver is ready, you should a pod launched in the same namespace. Verify using command kubectl get pods -n demo you should see like below image.

• Once server pod is running, there should be “vault-keys“ secret created in the same namespace. verify using command kubectl get secrets -n demo
• Get the secret as a yaml configuration using command kubectl get secrets vault-keys -n demo -o yaml once you get yaml configuration, in data field you will find vault-root-token: copy the root token and decode using online base64 decoder or in command line and store it somewhere.
• Expose the vault server using the command kubectl port-forward -n demo pod/vault-0 8200
• Go to browser search for localhost:8200 then you will be able to see vault UI

• Sign in method should be Token and Enter the token that you have stored somewhere and click on sign in
• once you sign in Enable the KV secret Engine, Create the secret by following the below images (Please follow the arrow directions)

• NOTE: You can also use version 2, if you want to use version 2, then the path should be kv
• Go to secret/ folder and click on Create secret, Enter the path, key and value and then click on save

Step: 5 Access secrets from the vault

• Create a serviceAccount.yaml and paste the below configuration in file

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-sa
  namespace: demo

• Create a VaultPolicy-VaultPolicyBinding.yaml and paste the below yaml configuration in it

apiVersion: policy.kubevault.com/v1alpha1
kind: VaultPolicy
metadata:
  name: kv-se-policy
  namespace: demo
spec:
  vaultRef:
    name: vault
  policyDocument: |
    path "secret/db-password" {
      capabilities = ["read"]
    }    
---
apiVersion: policy.kubevault.com/v1alpha1
kind: VaultPolicyBinding
metadata:
  name: kv-se-role
  namespace: demo
spec:
  vaultRef:
    name: vault
  policies:
  - ref: kv-se-policy
  subjectRef:
    kubernetes:
      serviceAccountNames:
      - "pod-sa"
      serviceAccountNamespaces:
      - "demo"

• Create a secretProviderClass.yaml and paste the below yaml configuration in it.

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: demo
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.demo:8200"
    roleName: "k8s.<Your_ClusterID>.demo.kv-se-role"   
    objects: |
      - objectName: "db-password"
        secretPath: "/secret/db-pass"
        secretKey: "password"
  secretObjects:            
    - secretName: db-secret  # k8s secret name that you want SecretProviderClass to provision
      type: Opaque
      data:
        - objectName: "db-password"
          key: db-password

• In the above configuration file, roleName field:

ClusterID = Your cluster id, which you provided to get the licence
demo = your vault namespace
kv-se-role = The VaultPolicyBinding name that you have created above

• Verify secretproviderclass using kubectl get secretproviderclass -n demo you should be able see your secretproviderclass named with vault-database.
• Once your created secretProviderClass, one new secret should be created with the name db-secret as you mention in the secretProviderClass yaml configuration.
• Create a demo-pod.yaml and paste the below configuration init.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: demo
spec:
  containers:
    - image: jweissig/app:0.0.1
      name: test-app
      env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: db-secret
            key: db-password

• Verify your pod is up or not using command kubectl get pods -n demo you should see pod Running.
• Exec into the pod using command kubectl exec -it mypod -n demo -- sh
• Once your inside the pod, go to /secrets-store/test using command cd /secrets-store/test you should a file name called db-password open the file using command cat db-password then you should be able to see the value that is present in the vault.
• Keep changing the values in vault server and check if values are updated in the secret that is created by secretProviderClass.
• NOTE: Pod gets updated values once it restarts, you can use any custom script to do this task.

Use case: When you have multiple account under the organization, and you want to access AWS resources across all accounts privately and this solution should be cost effective.

What is VPC Endpoints ?

VPC endpoints are virtual devices. They are horizontally scaled, redundant, and highly available Amazon VPC components that allow communication between instances in an Amazon VPC and services without imposing availability risks or bandwidth constraints on network traffic. There are two types of VPC endpoints

When you make use of VPC Endpoints, When an EC2 instance or other resources wants to communicate with the other AWS Service, the traffic will flow within the AWS Network. It will not flow via internet

Resources that we gonna use in this task:

1. AWS Route 53
2. Transit Gateway
3. VPC Endpoint
4. EC2 instance

Step 1: Transit Gateway setup with VPC Attachments
create the Transit gateway with default settings as shown in the below picture. You just have to enter the name of Transit Gateway and click on create.

Wait for the TGW to be available

Go to Transit Gateway Attachment tab and create the attachment as show in the below instructions

Enter the name of Attachment, Select the transit gateway ID which your created in the last step and select attachment type VPC and select the centralized VPC and subnets as well.

Create another attachment which points to other VPC(which consumes centralized VPC endpoint). and follow the same steps if you want to attach number of VPCs.

once both VPC attachments are available as shown in below image, We can proceed with next steps

Step 2: Create VPC Endpoint in central Account

Name → Give any name that you want, Service category → AWS services

In services search, I entered Lambda for testing purpose. You can enter any service that you want to connect privately and select service name.

Select the Central VPC and subnets. Select default security group or create a security group which allows the inbound traffic from your child account.

Click on create to create the VPC endpoint. Wait till it get activated

Select VPC Endpoint go to actions → Modify Private DNS Name → uncheck Enable for this endpoint → click on save changes.

Step 3: Create Private Hosted Zone in Route 53

create the hosted zone

Domain name → Give the name of your choice, select Private Hosted Zone.

Select the Central VPC and child VPC to which VPCs you want to share the Endpoint access. Create PHZ

Select the PHZ that you create in the last step and click on create record

Enable Alias select Alias to VPC endpoint, Select the region where your VPC endpoint has created. select the VPC endpoint and click in create records.

Launch an instance in child VPC and log in to it. Try to resolve your DNS in my case it is rakshith.com

Hence we can see that our instance is connecting to lambda using private IP. So we can conclude that traffic is flowing privately.

The story behind the this blog is…………….

In a recent project one of our client faced a challenge in their app development environment. When developer updates the code, he needs to push the updated code to GitHub. Once commit happens, pipeline will trigger deploy it in AWS Lambda function. Some times the code changes worked great but some other times code changes were crashed developers heads. So we decided to take this up and make this development process smoother. while this remains in my mind, I came up with a solution where they can develop their code and test in their local system. If changes were worked fine in developing environment, then they pushed code to GitHub and deployed in AWS Lambda through pipeline. And the solution is Serverless Framework with serverless-offline plugin.

• The main agenda of this blog is to know the greatness of serverless framework and its plugins like serverless-offline
• You can simply follow this blog when your lambda function have to fetch data from database and give response back to the user. Every application functionality. Your code might simply process the request and give the response back to the user (you can follow this blog in that case as well but, you don’t need launch a docker container.)
• In my case when a user or external sources triggers the Lambda, the lambda runs, fetch the data from database and return the response.

My Project Introduction

• When user triggers the lambda function by manual or by some external sources, Lambda function will fetch database credentials from AWS Secretes Manager and create a Database connection pool. So that my lambda can communicate with the database. Then the lambda fetch some data from database and give back to the user

SERVERLESS FRAMEWORK

• Before going to do practical, lets explore serverless framework and its plugins and serverless-offline plugin at a high level.
• Serverless Framework helps you to develop and deploy lambda functions, along with the infrastructure they require.
• The main advantage of serverless framework, It manages your application code and as well as the Infrastructure the you defined.
• When you define infrastructure in serverless framework, In the backend the Cloud Formation stack will be generated and applied in AWS cloud.
• It supports multiple languages like Node.js, Python, Java and more…

Note:- I am not defining any resources in this block as I worked around the API Gateway, Lambda and Database.

Serverless-offline Plugin

This Serverless plugin emulates AWS Lambda and API Gateway on your local machine to speed up your development cycles. To do so, it starts an HTTP server that handles the request’s lifecycle like API Gateway does and invokes your handlers.

Lets get started……

Step 1:

• Log into AWS account. Open Secretes Manager console. Create secret with your database credentials.

• In my project code, The secrets are fetching from the AWS secrets manager so that I created secretes In AWS cloud.

Step 2:

• Create a folder and place all your lambda function code in that particular folder
• Install the serverless Framework using npm command npm install -g serverless. If you don’t already have Node.js on your machine, install it first If you don’t want to install Node or NPM, you can install serverless as a standalone binary. for more information about installation, please follow official serverless documentation.
• create a serverless.yaml file

service: lambda-invoke-aws-sdk-v3

plugins:
  - serverless-offline

provider:
  iamRoleStatements:
    - Action:
        - lambda:InvokeAsync
        - lambda:InvokeFunction
      Effect: Allow
      Resource: "*"
  memorySize: 128
  name: aws
  region: ap-south-1
  runtime: nodejs18.x
  versionFunctions: false

functions:
  hello:
    handler: index.handler
    events:
      - httpApi:
          method: get
          path: /filterData

• service: Specifies the name of the service or application.
• plugins: Lists the Serverless Framework plugins you want to use. In this case, the serverless-offline plugin is included, which allows you to simulate AWS Lambda and API Gateway locally for testing purposes.
• provider: Describes the configuration for the cloud provider (in this case, AWS). It includes IAM role statements, memory size, AWS region, runtime, and other provider-specific settings.
• functions: Defines the AWS Lambda functions in your service. In this case, there's one function named "hello" with a handler function index.handler and an HTTP API event that responds to HTTP GET requests at the path "/filterData."

Step 3:

• Create a Dockerfile for the MySQL database (In my case, my database launched in docker container. You can also install MySQL server in local machine).

# Use the official MySQL image from Docker Hub
FROM mysql:latest

# Set environment variables for MySQL
ENV MYSQL_ROOT_PASSWORD=<Enter the password that you want to set>
ENV MYSQL_DATABASE=<Enter the name of database>

# Copy SQL files into the container
COPY --chown=mysql:mysql ./Dump20231213.sql/ /docker-entrypoint-initdb.d/ 
COPY --chown=mysql:mysql ./Dump20240108.sql/ /docker-entrypoint-initdb.d/


# Expose the MySQL port
EXPOSE 3306

• Build the docker image using command docker build -t <ImageName> .

Note: In the above command the dot(.) should also there, it represents current folder.

• once the docker image built verify

• Run the docker image in order to launch a MySQL container with the command docker run -p 3306:3306 — name=<ContainerName> -d <ImageName>
• Once the container is created, verify if it is running up and running.

• exec into the MySQL container

• Once you enter into the docker container log into mysql server

• Enter the password that you have gave in your dockerfile.
• Execute your database schema file.

• Verify your databases using command SHOW DATABASES;
• Verify your Tables using command SHOW TABLES;
• Make your database default with command USE <DatabaseName>;
• By default Node.js uses mysql_native_password protocol to communicate with the MySQL database. Latest MySQL server uses caching_sha2_passwod.
• Use the following command to change the authentication protocol ALTER USER ‘root’ IDENTIFIED WITH mysql_native_password BY ‘<Enter your db Password>’;
• Now your MySQL database is ready. You can go ahead and launch a serverless framework application now.

Step 4:

• Run the following command to install the packages that you have used in you node.js code npm install
• Now run your serverless framework using serverless-offline plugin with the command serverless-offline

• You can test this url in postman http://localhost:3000/<YourAppPath>

The problem which we faced

The pods were keep on evicted in our cluster. when we were trying to figurout the reason why this pods are evicting, then we got know that, one of the pod is consuming very high resources (lets say the pod a consuming 2gb memory and 1.5vcpu , other pods b and c were consuming only 0.5gb memory and 0.5vcpu, the node has only 20gb disk space and 2vCPU ). just because of pod a, the other pods(a and b) also evicted. but, at any cost pod b should be at running state.

Then we found a kubernetes feature called priority class

General flow of pod scheduling in kubernetes

The control plane consists of multiple components, out of which the scheduler (usually the built-in kube-scheduler) is one of the components which is responsible for assigning a node to a pod.

Whenever a pod is created, it enters a “pending” state, after which the scheduler determines which node is best suited for the placement of the new pod.

In the background, the scheduler runs as an infinite loop looking for pods without a nodename set that are ready for scheduling. For each Pod that needs scheduling, the scheduler tries to decide which node should run that Pod. If the scheduler cannot find any node, the pod remains in the pending state, which is not ideal.

Let’s start with PriorityClass

we will be discussing on three basic terms WHAT, WHY and HOW

====> what is priorityclass ?

PriorityClass is a cluster-wide API object in Kubernetes and part of the “scheduling.k8s.io/v1” API group. It contains an integer value (-1000000000<lower priority> to 1000000000<higher priority>) represents the value that the scheduler uses to determine Pod’s relative priority.

====> why priorityclass ?

let’s say we have three pods (A) dev-pod (B) qa-pod and (C) prod-pod running on the same cluser. Our least priority would be dev-pod intermediate priority is qa-pod and high priority is prod-pod. when i want to run 3 pods according to their priorityclass the, when node has enough resources for scheduling three pods then the scheduler will schedule the three pods. In a case where we don’t have enough resources to schedule the pods (if you also have cluster autoscaler to scale up the nodes, there will be a certain time required to node to be at ready state).

Pod preemption is a Kubernetes feature that allows the cluster to preempt pods (removing an existing Pod to schedule a new Pod) on the basis of priority. Pod priority indicates the importance of a pod relative to other pods while scheduling. If there aren’t enough resources to run all the current pods, the scheduler tries to evict lower-priority pods over high-priority ones.

Also, when a healthy cluster experiences a node failure, typically, lower-priority pods get preempted to create room for higher-priority pods on the available node. This happens even if the cluster can bring up a new node automatically since pod creation is usually much faster than bringing up a new node.

====> How to use priorityclass

Let’s have manifest configurations for three pods dev-pod, qa-pod and prod-pod.

dev-pod manifest

create a manifest file with below code

apiVersion: v1
kind: Pod
metadata:
  name: dev-pod
  labels:
    env: dev
spec:
  containers:
  - name: dev-nginx
    image: nginx
    resources:
      requests:
        memory: "2Gi"
        cpu: "0.5"
      limits:
        memory: "2Gi"
        cpu: "0.5"

use command kubectl apply -f <yourFileName>

qa-pod manifest

apiVersion: v1
kind: Pod
metadata:
  name: qa-pod
  labels:
    env: preprod
spec:
  containers:
  - name: preprod-nginx
    image: nginx
    resources:
      requests:
        memory: "2Gi"
        cpu: "0.5"
      limits:
        memory: "2Gi"
        cpu: "1"

use command kubectl apply -f <yourFileName>

prod-pod manifest

apiVersion: v1
kind: Pod
metadata:
  name: prod-pod
  labels:
    env: prod
spec:
  containers:
  - name: prod-nginx
    image: nginx
    resources:
      requests:
        memory: "1Gi"
        cpu: "1.5"
      limits:
        memory: "2Gi"
        cpu: "1.8"

use command kubectl apply -f <yourFileName>

NOTE:- In my case, for testing I have only one node with 20gb disk and 2vcpu

when I deploy thos three pods one of the pod went to pending state as you can absorve in the below image

when i describe the prod-pod to see why it was gone to pending state, it was stating that due to insuffiecent cpu, the particular prod-pod was not scheduled as you can observer in the below image.

So, now prod-pod has to be in running state, for that we will be using priorityclass

priorityclass manifest

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: sample
value: 10000000 # any integer value between -1000000000 to 1000000000 
globalDefault: false 

as you can observe the integer value, “the higher the value, the higher the priority” it doesn’t mean that this pod will not evict. But it will be the last pod to evict whenever resources are degreded in the node.

Refer priorityclass in pod manifest:-

Before you make changes to the pod manifest you will have to delete the existing pod (prod-pod) which is in pending state use command kubectl delete pod <yourPodName> to delete the pod . while using priorityclass we cannot configure changes which is already in running or pending state.

The allowed fields that can be updated are:

• spec.containers[*].image
• spec.initContainers[*].image
• spec.activeDeadlineSeconds
• spec.tolerations (only additions to existing tolerations)
• spec.terminationGracePeriodSeconds (allow it to be set to 1 if it was previously negative)

All other fields are immutable after the Pod is created, including spec.hostAliases and spec.priorityClassName.

The below code is the code which is updated with the priorityclass

apiVersion: v1
kind: Pod
metadata:
  name: prod-pod
  labels:
    env: prod
spec:
  containers:
  - name: prod-nginx
    image: nginx
    resources:
      requests:
        memory: "1Gi"
        cpu: "1.5"
      limits:
        memory: "2Gi"
        cpu: "1.8"
  priorityClassName: sample

use command kubectl apply -f <yourFileName>

you can obsorve in the image that qa-pod was terminated and prod-pod is up and running and dev-pod is in pendig state due to insufficient cpu

you can also create indivisual priorityclasses for each pod like for dev-pod, priorityclass value 1000000 and for qa-pod, priorityclass value 2000000 as per your requirement

NOTE:- If you set globalDefault: true then the particular priorityclass resource priority value( example:-1000000 ) will be applied to everypod in the cluster

To know more about priorityclasse and pod preemption please visit official kubernetes documentation https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/