Introduction

In the previous episode, we explored Pods, the smallest deployable unit in Kubernetes.

We learned that:

• A Pod can contain one or more containers
• Containers inside a Pod share networking and storage
• Kubernetes manages Pods as a single unit

But in real-world production environments, not all containers inside a Pod serve the same purpose.

Some containers:

• Prepare the environment before the app starts
• Continuously support the application while it runs

These advanced patterns are called:

👉 Init Containers
👉 Sidecar Containers

And they are heavily used in production-grade Kubernetes deployments.

What are Init Containers?

Init Containers are special containers that run BEFORE the main application containers Run inside a Pod.

They are designed for:

• Initialization tasks
• Environment preparation
• Dependency validation

Important characteristics:

• Run sequentially
• Run only once
• Must complete successfully before app containers start

Why Do We Need Init Containers?

In production systems, applications often depend on external services or setup tasks.

Instead of embedding initialization logic into the application itself, Kubernetes allows us to separate it cleanly using Init Containers.

Common Real-World Use Cases

⏳ Waiting for Dependencies

Example:

• Wait for database readiness
• Wait for Redis/cache services

🗄 Database Migrations

Run migration scripts before application startup.

📦 Configuration Generation

Generate dynamic configuration files before app launch.

🔐 Permission Setup

Set proper file permissions or create directories.

✅ Environment Validation

Checks:

• DNS resolution
• External APIs
• Secrets availability

Internal Working of Init Containers (VERY IMPORTANT)

Let’s understand the execution flow.

1️⃣ Pod Creation

User creates a Pod YAML

Kubernetes creates the Pod object.

2️⃣ Init Container Starts First

Before application containers start:

👉 Kubernetes launches Init Containers sequentially.

3️⃣ Must Complete Successfully

If Init Container succeeds:
✅ Next step in Pod startup continues

If Init Container fails:
❌ Main App does NOT start
❌ Pod does NOT proceed

4️⃣ Main Application Starts

Only after all Init Containers finish successfully:

• Main application containers start

5️⃣ Failure & Restart Behavior

If an Init Container fails:

• Kubernetes restarts it
• Pod remains in:

Init:CrashLoopBackOff

until success.

🔑 Init Container Characteristics

⏱ Execution: Before app starts
🔁 Lifecycle: Runs once
📋 Order: Sequential
♻️ Restart Behavior: Retries until success
🕒 Runtime: Temporary
📦 Resources: Independent

🌍 Real-World Init Container Example

Imagine:

• Your app depends on PostgreSQL
• DB takes 20 seconds to start

Instead of failing the app: 👉 Init Container waits until DB becomes reachable.

Example:

initContainers:
- name: wait-for-db
  image: busybox
  command:
    - sh
    - -c
    - until nc -z postgres 5432; do echo waiting; sleep 2; done

What are Sidecar Containers?

Sidecar Containers are containers that run alongside the main application container inside the same Pod.
They continuously provide supporting functionality.

Why Do We Need Sidecars?

Modern applications require:

• Logging
• Monitoring
• Security
• Traffic management

Instead of embedding all of this inside the application: 👉 Kubernetes uses Sidecars.

Common Real-World Use Cases

📜 Logging Agents

Example:

• Fluentd
• Filebeat

Collect application logs.

📈 Monitoring Agents

Example:

• Prometheus exporters

Expose metrics.

🔐 Service Mesh Proxies

Example:

• Istio Envoy Proxy

Handle:

• mTLS
• Traffic routing
• Security

🔄 Configuration Reloaders

Reload configs dynamically without restarting apps.

🛡 Security Agents

Inject secrets/certificates securely.

⚙️ Internal Working of Sidecars (VERY IMPORTANT)

1️⃣ Pod Starts

Pod is scheduled to a node.

2️⃣ Main Container + Sidecar Start Together

Unlike Init Containers: 👉 Sidecars run continuously.

3️⃣ Shared Networking & Storage

Containers inside the Pod share:

• Same IP address
• Same port namespace
• Shared volumes

4️⃣ Sidecar Continuously Supports Main App

Examples:

• Collect logs
• Sync files
• Monitor traffic
• Encrypt communication

🔑 Sidecar Characteristics

• Execution: Runs alongside the main application
• Lifecycle: Continuous throughout the pod lifecycle
• Networking: Shares the same network namespace
• Storage: Shares storage volumes with the app container
• Runtime: Long-running
• Purpose: Provides supporting functionality

⚔️ Init Containers vs Sidecars

Init Containers

• Execute before the app starts
• Short-lived and temporary
• Used for setup tasks
• Retry until successful
• Commonly used for DB wait checks and initialization

Sidecars

• Run together with the app
• Long-running and continuous
• Provide supporting capabilities
• Restart with the Pod
• Commonly used for logging and monitoring

🔑 Key Concepts

Multi-Container Pods

One Pod can run multiple containers.

Shared Volumes

Containers share files/data.

Sequential Startup

Init Containers start before app containers.

Shared Network Namespace

Same IP + localhost communication.

🧠 Real-World Analogies

🏗 Init Container

Like a setup crew preparing a stage before an event begins.

🧑‍🔧 Sidecar

Like an assistant working beside you continuously during the event.

✅ Best Practices

✔ Keep Init Containers lightweight
✔ Avoid long-running init tasks
✔ Use Sidecars only when necessary
✔ Separate responsibilities clearly
✔ Monitor resource usage carefully

❌ Common Mistakes

🚫 Using Init Containers as regular app containers
🚫 Overloading Pods with too many Sidecars
🚫 Confusing Sidecars with helper processes
🚫 Running business logic inside Sidecars

🎯 Interview Questions

1. What is an Init Container?

A special container that runs before application containers.

2. Can Init Containers run in parallel?

No, they run sequentially.

3. What happens if an Init Container fails?

Kubernetes retries until success.

4. What is a Sidecar Container?
A supporting container running alongside the main app.

5. Do Sidecars share the same network?
Yes.

6. What are common Sidecar use cases?
Logging, monitoring, service mesh.

7. Difference between Init and Sidecar?
Init = setup before app
Sidecar = support during runtime

📝 Summary (TL;DR)

• Init Containers prepare the environment before apps start
• Sidecars continuously support the application
• Both patterns are critical in production Kubernetes
• Multi-container Pods enable modular architecture

🔜 What’s Next?

👉 Next: Kubernetes Service Deep Dive

We’ll explore how Pods are exposed and how networking works across applications.

Init Containers & Sidecars Explained: Advanced Pod Patterns in Kubernetes was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

So far in this series, we’ve explored how Kubernetes works behind the scenes:

• Episode 1: API Server → Entry point
• Episode 2: etcd → Stores cluster state
• Episode 3: Scheduler → Decides placement
• Episode 4: Controller Manager → Maintains state
• Episode 5: Kubelet → Executes workloads
• Episode 6: Kube Proxy → Handles networking

Now comes the most important piece:

👉 What actually runs inside Kubernetes?

The answer is: Pods

What is a Pod?

A Pod is the smallest deployable unit in Kubernetes.

It represents a group of one or more containers that run together on the same node.

Think of a Pod as: 👉 A wrapper around containers

Why Do We Need Pods?

You might ask:

👉 Why not run containers directly?

Because Kubernetes needs:

• A higher-level abstraction
• Shared resources between containers
• A consistent deployment unit

Pods provide:

✅ Shared networking
✅ Shared storage
✅ Lifecycle management
✅ Tight coupling between containers

🏗️ Pod Architecture

A Pod consists of:

🔹One or More Containers

• Usually one main container
• Optional sidecars

🌐 Shared Network

All containers share:

• Same IP address
• Same port space

👉 Containers communicate via localhost

💾 Shared Storage

Volumes are shared between containers

Used for:

• Data sharing
• Persistence

⚙️ Internal Working (Step-by-Step)

1️⃣ Pod Definition (YAML)

User defines Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: nginx

2️⃣ API Server Stores It

• Request goes to API Server
• Stored in etcd

3️⃣ Scheduler Assigns Node

• Pod is in Pending state
• Scheduler selects best node

4️⃣ Kubelet Creates Pod

• Kubelet reads PodSpec
• Pulls image
• Creates containers

5️⃣ Containers Start Running

• Pod moves to Running state
• Kubelet monitors continuously

🔄 Pod Lifecycle

Pods go through different phases:

🟡 Pending

• Pod created, waiting for scheduling

🟢 Running

• Containers are running

🔵 Succeeded

• Completed successfully (batch jobs)

🔴 Failed

• One or more containers failed

⚪ Unknown

• Node communication lost

🧩 Multi-Container Pods

🔹 Sidecar Patter

• Helper container alongside main app

Example

• Logging
• Monitoring

🔄 Workflow (End-to-End)

User → YAML
        ↓
API Server
        ↓
etcd
        ↓
Scheduler assigns node
        ↓
Kubelet pulls image
        ↓
Containers start
        ↓
Pod Running

🔑 Key Concepts

Pod

• Smallest deployable unit

Container

• Runs application

Volume

• Shared storage

Networking

• Same IP for all containers

Lifecycle

• Different Pod states

🧠 Real-World Analogy

Think of a Pod as a:

🏠 Shared Apartment

• Containers = roommates
• Share network & storage
• Live together

✅ Best Practices
✔ Use Deployments instead of standalone Pods
✔ Keep Pods lightweight
✔ Use sidecars only when needed
✔ Define resource limits

❌ Common Mistakes
🚫 Treating Pods as permanent
🚫 Ignoring lifecycle states
🚫 Overusing multi-container Pods
🚫 Running unmanaged Pods

🎯 Interview Questions

1. What is a Pod?
Smallest deployable unit in Kubernetes.

2. Can a Pod have multiple containers?
Yes.

3. Do containers in a Pod share IP?
Yes.

4. What is Pod lifecycle?
Different phases like Pending, Running, Failed.

5. Should we use standalone Pods in production?
No, use Deployments.

📝 Summary (TL;DR)

• Pod = smallest unit in Kubernetes
• Runs one or more containers
• Shares network & storage
• Managed by Kubelet
• Has defined lifecycle

🔜 What’s Next?

👉 Next: Service Deep Dive

We’ll explore how Pods are exposed and accessed.

Kubernetes Pods Explained: The Smallest Deployable Unit in K8s was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In this series so far:

• Episode 1: API Server → Entry point
• Episode 2: etcd → Stores cluster state
• Episode 3: Scheduler → Decides where Pods run
• Episode 4: Controller Manager → Maintains desired state
• Episode 5: Kubelet → Runs containers on nodes

Now everything is running…

👉 But here’s the real question:

How do Pods communicate with each other reliably?

That’s where Kube Proxy comes in.

What is Kube Proxy?

Kube Proxy is a network component that runs on every node in a Kubernetes cluster.

It enables communication between Services and Pods by routing traffic correctly.

Think of it as: 👉 The network traffic manager inside Kubernetes

Why Do We Need It?

In Kubernetes:

• Pods are dynamic (they come and go)
• Pod IPs change frequently
• Direct communication is unreliable

Without Kube Proxy:

❌ No stable networking
❌ No service abstraction
❌ No load balancing

👉 Kube Proxy solves this by:

• Providing stable access via Services
• Routing traffic to the correct Pods
• Load balancing requests

Where It Fits in Architecture

• Runs on every worker node
• Works with Services & Endpoints
• Watches API Server for updates
• Configures networking rules on the node

⚙️ Internal Working (Step-by-Step)

1️⃣ Watches Services & Endpoints

Kube Proxy continuously monitors:

• Services (ClusterIP)
• Endpoints (Pod IPs)

👉 Via API Server

2️⃣ Maintains Network Rules

It programs rules using:

• iptables OR
• IPVS

These rules define: 👉 How traffic should be routed

3️⃣ Routes Traffic to Pods

When a request comes to a Service:

👉 Kube Proxy redirects it to one of the backend Pods

4️⃣ Load Balances Traffic

If multiple Pods exist:

👉 Traffic is distributed across them

🔀 Modes of Kube Proxy

🔹 iptables Mode (Default)

• Uses Linux iptables rules
• Simple and widely used
• Slightly less efficient at scale

🔹 IPVS Mode

• Uses kernel-level load balancing
• More scalable and performant
• Supports advanced algorithms

🌐 Service Networking Flow

Client → Service (ClusterIP) → Kube Proxy → Pod

Explanation:

1. Client sends request to Service IP
2. Kube Proxy intercepts request
3. Selects a backend Pod
4. Forwards traffic

🔄 Workflow (End-to-End)

User → Service (ClusterIP)
        ↓
Kube Proxy (Node)
        ↓
Select Pod (via Endpoints)
        ↓
Forward request
        ↓
Pod processes request
        ↓
Response sent back

🔑 Key Concepts

Service

• Stable endpoint for Pods

ClusterIP

• Internal virtual IP for service

Endpoints

• List of backend Pod IPs

Load Balancing

• Distributes traffic across Pods

iptables / IPVS

• Mechanisms used for routing

🧠 Real-World Analogy

Think of Kube Proxy as a:

Traffic Manager

• Receives incoming requests
• Decides which route to take
• Ensures smooth flow without congestion

✅ Best Practices

✔ Use correct Service types (ClusterIP, NodePort, LoadBalancer)
✔ Prefer IPVS for large-scale clusters
✔ Monitor network latency
✔ Understand traffic flow

❌ Common Mistakes

🚫 Confusing Service with Pod
🚫 Ignoring networking issues
🚫 Misunderstanding load balancing
🚫 Not checking endpoints

🎯 Interview Questions

1. What is Kube Proxy?
A network component that routes traffic to Pods.

2. What is ClusterIP?
A virtual IP used to access a Service.

3. Difference between iptables and IPVS?
iptables = rule-based, IPVS = scalable load balancing.

4. Does Kube Proxy run on master?
No, runs on worker nodes.

5. How does load balancing happen?
Via iptables/IPVS rules across Pods.

📝 Summary (TL;DR)

• Kube Proxy enables Pod communication
• Uses Services as abstraction
• Routes & load balances traffic
• Runs on every node

🔜 What’s Next?

👉 Next: Pod Deep Dive
We’ll explore what actually runs inside Kubernetes.

Kube Proxy Explained: How Kubernetes Handles Networking Between Pods was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

So far in this series:

• Episode 1: API Server → Entry point
• Episode 2: etcd → Stores cluster state
• Episode 3: Scheduler → Decides where Pods run
• Episode 4: Controller Manager → Ensures desired state

Now comes the most important question:

👉 Kubernetes plans everything… but who actually runs your containers?

The answer is Kubelet.

What is Kubelet?

Kubelet is a node-level agent that runs on every worker node in a Kubernetes cluster.

It ensures that containers described in PodSpecs are running correctly.

Think of it as: 👉 The executor of Kubernetes decisions

Why Do We Need It?

Control plane components make decisions, but:

• They don’t run containers
• They don’t interact with the OS
• They don’t manage runtime execution

Without Kubelet:

❌ Pods would never start
❌ No container lifecycle management
❌ No node-level monitoring

👉 Kubelet bridges: Control Plane → Worker Node Execution

Where It Fits in Architecture

• Runs on every worker node
• Communicates with API Server
• Works with container runtime (containerd / Docker)
• Reports back node & pod status

Internal Working (Step-by-Step)

1️⃣ Watches PodSpecs from API Server

Kubelet continuously polls or watches:

👉 API Server for Pods assigned to its node

2️⃣ Ensures Containers Are Running

Kubelet compares:

• Desired state (PodSpec)
• Actual state (running containers)

👉 If mismatch → it fixes it

3️⃣ Interacts with Container Runtime

Kubelet does NOT run containers directly.
Instead it uses:

• containerd
• CRI (Container Runtime Interface)

👉 Example:

Kubelet → CRI → containerd → runc → container

4️⃣ Performs Health Checks

Kubelet runs:

• Liveness probes → Is container alive?
• Readiness probes → Is container ready to serve traffic?

👉 If failed:

• Restart container
• Remove from service

5️⃣ Reports Status to API Server

Kubelet continuously sends:

• Pod status
• Node health
• Resource usage

🔄 Pod Lifecycle Handling

Here’s how Kubelet manages a Pod:

1. Scheduler assigns Pod to node
2. API Server updates PodSpec
3. Kubelet detects assignment
4. Pulls container image
5. Starts containers
6. Monitors continuously
7. Restarts if needed

🔁 Workflow (End-to-End)

User → kubectl apply
        ↓
API Server
        ↓
etcd (stores state)
        ↓
Scheduler assigns node
        ↓
API Server updates Pod
        ↓
Kubelet detects Pod
        ↓
Pull image from registry
        ↓
Container runtime starts container
        ↓
Kubelet monitors & reports

🔑 Key Concepts

PodSpec

• Blueprint of Pod
• Defines containers, resources

Container Runtime

• Actually runs containers
• Example: containerd

Node Status

• Health of node
• Sent to API Server

Health Checks

• Liveness & Readiness probes

Sync Loop

• Continuous reconciliation at node level

Real-World Analogy

Think of Kubelet as a:

👷 Machine Operator in a Factory

• Control plane gives instructions
• Kubelet executes them
• Ensures machines (containers) run smoothly

✅ Best Practices

✔ Define proper resource requests
✔ Always use liveness/readiness probes
✔ Monitor node health
✔ Use reliable container images

❌ Common Mistakes

🚫 Ignoring health checks
🚫 Misconfigured containers
🚫 Assuming Scheduler runs containers
🚫 Not monitoring node status

🎯 Interview Questions

1. What is Kubelet?
Agent that runs on nodes and manages Pods.

2. Does Kubelet run containers directly?
No, it uses container runtime.

3. What are probes?
Health checks for containers.

4. What happens if a container fails?
Kubelet restarts it.

5. How does Kubelet communicate?
Via API Server.

📝 Summary (TL;DR)

• Kubelet executes workloads on nodes
• Works with container runtime
• Manages Pod lifecycle
• Performs health checks
• Reports status

🔜 What’s Next?
👉 Next: Kube Proxy Deep Dive
We’ll explore how networking works inside Kubernetes.

Kubelet Explained: The Agent That Brings Your Pods to Life was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In this series so far:

• Episode 1: API Server → The entry point
• Episode 2: etcd → Stores cluster state
• Episode 3: Scheduler → Decides where Pods run

Now comes a deeper question:

👉 Once Pods are scheduled… who ensures they keep running correctly?

That responsibility belongs to the Kubernetes Controller Manager.

🧠 What is Kubernetes Controller Manager?

The Controller Manager is a control plane component that runs multiple controllers.

It continuously ensures that the actual state of the cluster matches the desired state.

Instead of doing things once, it works in a loop:

• Observe
• Compare
• Fix

⚙️ Why Do We Need It?

In distributed systems:

• Pods can crash
• Nodes can fail
• Network issues can occur

Without a Controller Manager:

❌ Applications would break and stay broken

❌ No automatic recovery

❌ Manual intervention required

👉 Controller Manager enables:

• Self-healing
• Automation
• Reliability

🏗️ Where It Fits in Architecture

• Part of the Control Plane
• Runs continuously in the background
• Communicates with API Server
• Reads state from etcd (via API Server)

⚙️ Internal Working (Step-by-Step)

Let’s break it down:

1️⃣ Desired State Defined

Users define the desired state using YAML:

• Deployment (e.g., 3 replicas)
• Job
• ReplicaSet

Example:

replicas: 3

2️⃣ Controller Watches Cluster State

Each controller continuously watches:

👉 API Server for changes

3️⃣ Compare Desired vs Actual State

Controller checks:

• Desired: 3 Pods
• Actual: 2 Pods

4️⃣ Take Corrective Action

Controller acts:

👉 Creates 1 more Pod

This loop never stops.

🔁 Reconciliation Loop (Core Concept)

This is the heart of Controller Manager.

Observe → Compare → Act → Repeat

• Detects drift
• Fixes automatically
• Runs continuously

👉 This is why Kubernetes is self-healing

🧩 Types of Controllers (IMPORTANT)

Controller Manager is actually a collection of controllers:

🔹 Node Controller

• Detects node failures
• Marks nodes unhealthy
• Reschedules Pods

🔹 ReplicaSet Controller

• Ensures correct number of Pods
• Maintains replicas

🔹 Deployment Controller

• Manages ReplicaSets
• Handles rolling updates

🔹 Job Controller

• Runs batch jobs
• Ensures completion

🔹 Endpoint Controller

• Updates service endpoints
• Tracks Pod IPs

🔄 Workflow (End-to-End)

User creates Deployment
↓
API Server
↓
etcd stores state
↓
Controller Manager detects change
↓
Creates ReplicaSet
↓
Ensures Pods are running
↓
If Pod fails → recreate

🔑 Key Concepts

Desired vs Actual State

• Desired = defined by user
• Actual = current cluster state

Reconciliation Loop

• Continuous correction mechanism

Controllers

• Specialized logic units

Self-Healing

• Automatic recovery from failure

🧠 Real-World Analogy

Think of Controller Manager as a:

🌡️ Thermostat

• You set temperature (desired state)
• Room changes (actual state)
• Thermostat adjusts automatically

✅ Best Practices

✔ Always define desired state clearly
✔ Use Deployments instead of raw Pods
✔ Monitor controller logs
✔ Use health checks

❌ Common Mistakes

🚫 Confusing Scheduler with Controller
🚫 Ignoring reconciliation loop
🚫 Creating unmanaged Pods
🚫 Not defining replicas

🎯 Interview Questions

1. What is Controller Manager?
A component that ensures desired state matches actual state.

2. What is reconciliation loop?
Continuous process of observing, comparing, and correcting.

3. Difference between Scheduler and Controller?
Scheduler assigns nodes; Controller maintains state.

4. What happens if a Pod crashes?
Controller recreates it.

5. Does Controller Manager talk to etcd directly?
No, via API Server.

📝 Summary (TL;DR)

• Controller Manager maintains cluster stability
• Uses reconciliation loop
• Ensures desired = actual
• Enables self-healing

🔜 What’s Next?

👉 Next: Kubelet Deep Dive

We’ll explore how Pods actually run inside nodes.

Introduction

In the previous episodes of this series:

• We explored how the Kubernetes control plane communicates through the API Server
• We understood how cluster state is stored reliably in etcd

Now comes the critical question:

👉 Once a Pod is created and stored… who decides where it actually runs?

That’s where the Kubernetes Scheduler comes in.

🧠 What is Kubernetes Scheduler?

The Kubernetes Scheduler is a control plane component responsible for:

Assigning Pods to the most suitable Node in the cluster

When a Pod is created, it does NOT automatically run anywhere.

Instead:

• It stays in a Pending state
• The Scheduler evaluates available nodes
• Then selects the best one based on multiple factors

⚙️ Why Do We Need It?

In a real-world cluster:

• Multiple nodes with different resources
• Multiple Pods competing for CPU, memory
• Constraints like labels, affinity rules, taints

Without a scheduler:

❌ Pods could land on overloaded nodes

❌ Resources would be wasted

❌ Applications could fail unpredictably

👉 The Scheduler ensures:

• Efficient resource utilization
• Balanced workload distribution
• Intelligent placement decisions

🏗️ Where It Fits in Architecture

The Scheduler is part of the Kubernetes Control Plane.

It works closely with:

• API Server → Receives Pod definitions
• etcd → Stores cluster state
• Kubelet → Runs Pods on assigned nodes

⚙️ Internal Working (Step-by-Step)

let’s break down how the scheduler actually works:

1️⃣ Watching for Unscheduled Pods

The Scheduler continuously monitors the API Server for:

👉 Pods with no assigned node
These are Pods in:

status: Pending
nodeName: <empty>

2️⃣ Filtering Phase (Feasibility Check)

The Scheduler filters out nodes that cannot run the Pod.

Checks include:

• 🧠 Available CPU & Memory
• 🏷️ Node selectors
• 🚫 Taints & tolerations
• 📍 Node affinity / anti-affinity
• 🔒 Port conflicts

👉 Result: A list of valid candidate nodes

3️⃣ Scoring Phase (Ranking Nodes)

Now the Scheduler ranks the remaining nodes.

Each node gets a score based on:

• Resource availability
• Load balancing
• Affinity preferences
• Custom scheduling policies

👉 Higher score = better fit

4️⃣ Selection

The Scheduler selects:

🏆 The highest-ranked node

5️⃣ Binding

Finally:

• Scheduler sends decision to API Server
• Pod gets assigned:

nodeName: selected-node

Then: 👉 Kubelet takes over and runs the Pod

🔄 Scheduling Workflow (End-to-End)

Here’s the full workflow:

User → kubectl apply
        ↓
API Server
        ↓
etcd (stores Pod)
        ↓
Scheduler detects Pod
        ↓
Filtering + Scoring
        ↓
Best Node selected
        ↓
Binding via API Server
        ↓
Kubelet runs Pod

🔑 Key Concepts

Scheduling vs Binding

• Scheduling = Selecting node
• Binding = Assigning Pod to node

Filtering & Scoring

• Filtering removes invalid nodes
• Scoring ranks valid ones

Node Selection

• Based on constraints + optimization

Resource Requests & Limits

• Scheduler uses requests, not limits

Taints & Tolerations

• Taints = repel Pods
• Tolerations = allow exceptions

🧠 Real-World Analogy

Think of the Scheduler as an:

✈️ Air Traffic Controller

• Many planes (Pods)
• Many runways (Nodes)
• Must assign safely and efficiently

👉 No collisions, no overloads, smooth operations

✅ Best Practices

✔ Always define resource requests
✔ Use node selectors carefully
✔ Use affinity rules for control
✔ Monitor scheduling behavior

❌ Common Mistakes

🚫 Not defining resource requests
🚫 Ignoring Pending Pods
🚫 Misconfigured taints/tolerations
🚫 Overloading specific nodes

🎯 Interview Questions

1. What does Kubernetes Scheduler do?
Assigns Pods to appropriate nodes.

2. What is filtering vs scoring?
Filtering removes invalid nodes, scoring ranks valid ones.

3. Does Scheduler run Pods?
No, Kubelet runs Pods.

4. What happens if no node matches?
Pod stays in Pending state.

5. What data does Scheduler use?
Cluster state from API Server (backed by etcd).

📝 Summary (TL;DR)

• Scheduler is decides where is pods run
• Works in filter → score → select → bind flow
• Ensures efficient and balanced cluster usage
• Critical for performance and reliability

🔜 What’s Next?

In the next episode:

👉 Controller Manager Deep Dive

We’ll explore how Kubernetes maintains the desired state automatically.

Kubernetes Scheduler Explained: How Pods Find Their Perfect Node was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In Episode 1, we explored the Kubernetes API Server—the entry point for all cluster interactions.

But that raises a critical question:

👉 Where does Kubernetes store all this data?

The answer is etcd.

etcd is the backbone of Kubernetes state management. Without it, your cluster cannot function—it has no memory.

What is etcd?

etcd is a distributed key-value store used by Kubernetes to store all cluster data.

In simple terms:

etcd = Kubernetes database

It stores:

• Pods
• Nodes
• Deployments
• ConfigMaps
• Secrets

All stored as key-value pairs.

Why Do We Need It?

Kubernetes is a distributed system where:

• Multiple components act independently
• Desired state must be persisted
• Failures must be recoverable

Problem it solves:

👉 Reliable cluster state storage

Without etcd:

• ❌ No persistence
• ❌ No recovery
• ❌ No consistent behavior

Where It Fits in Architectur

etcd is part of the Control Plane.

Key Rule:

👉 Only the API Server interacts with etcd

• Scheduler → API Serve
• Controllers → API Server
• Users → API Server

This ensures controlled and secure access.

Internal Working (Step-by-Step)

1. Data Stored as Key-Value Pairs

Example:

/registry/pods/nginx → Pod definition
/registry/services/frontend → Service config

2. Structured Data Paths

Kubernetes organizes data like:

• /registry/pods/
• /registry/nodes/
• /registry/services/

👉 Makes lookup fast and predictable

3. Read & Write Operations

• API Server writes → etcd stores
• API Server reads → etcd returns

👉 etcd acts as the backend database

4. Watch Mechanism

Instead of polling:

• Components watch for changes
• API Server gets notified
• System reacts immediately

Example:

• New Pod created → Scheduler reacts

5. Strong Consistency (Raft Consensus)

This is the most important concept 👇

etcd uses the Raft consensus algorithm to maintain consistency across nodes.

How Raft Works (High-Level):

• Cluster has multiple nodes (usually 3 or 5)
• One node becomes Leader 👑
• Others are Followers

Write Flow:

1. API Server sends write request
2. Leader receives it
3. Leader replicates to followers
4. Majority confirms
5. Data is committed

👉 Guarantees:

• No stale reads
• No split-brain issues
• Strong consistency across cluster

End-to-End Workflow

User → kubectl → API Server → etcd → Data Stored → Scheduler reacts → Node → Kubelet → Container Runtime → Pod

Step-by-step:

• User sends request via kubectl
• API Server validates request
• Data written to etcd
• etcd stores state (key-value)
• Scheduler detects change
• Node selected
• Kubelet + container runtime run the Pod

Interview Questions

1. What is etcd?
Distributed key-value store for Kubernetes.

2. What algorithm does etcd use?
Raft consensus.

3. Who interacts with etcd?
Only API Server.

4. What is watch mechanism?
Event-driven updates instead of polling.

What’s Next
👉 Episode 3: Kubernetes Scheduler Deep Dive

5. Why is etcd critical?
It stores the entire cluster state.

Etcd in Kubernetes Explained: The Database Behind Your Cluster was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Kubernetes is a powerful orchestration system—but behind its simplicity lies a well-structured control mechanism.

At the center of this system is the Kubernetes API Server, the most critical component of the control plane.

In this article, you’ll learn:

• What the Kubernetes API Server is
• Why it is essential for cluster operations
• How it processes every request step-by-step
• How it connects all Kubernetes components.

If Kubernetes is a system, the API Server is its brain and gatekeeper.

What is Kubernetes API Server?

The Kubernetes API Server (kube-apiserver) is the entry point to your Kubernetes cluster.

It exposes a REST API that allows users and components to:

• Create resources (Pods, Deployments, Services)
• Update configurations
• Query cluster state

Every interaction—whether from:

• kubectl
• CI/CD pipelines
• Controllers
• Internal Kubernetes components.

👉 goes through the API Server.

Why Do We Need It?

In a distributed system like Kubernetes:

• Multiple users interact simultaneously
• Components operate independently
• State must remain consistent

Without a central control layer:

• ❌ No security enforcement
• ❌ No consistent state
• ❌ No coordination

The API Server solves:

• ✅ Centralized communication
• ✅ Security (Authentication & Authorization)
• ✅ Consistency of cluster state
• ✅ Standard interface via REST

Where It Fits in Architecture

The API Server is part of the Control Plane.

Control Plane Components:

• API Server
• Scheduler
• Controller Manager
• etcd

👉 All components communicate through the API Server, not directly.

Internal Working (Step-by-Step)

This is the most important part—understanding how the API Server processes requests.

1. Request Enters API Server

Example:

kubectl apply -f deployment.yaml

This sends an HTTP request to the API Server.

2. Authentication

Question: Who are you?

The API Server verifies identity using:

• TLS certificates
• Bearer tokens
• Service accounts

👉 If authentication fails → request rejected

3. Authorization

Question: Are you allowed to perform this action?

Handled via:

• RBAC (Role-Based Access Control)

Example:

• A user may create Pods but cannot delete them

👉 If unauthorized → request denied

4. Admission Controllers

Question: Should this request be modified or validated further?

Admission controllers:

• Mutate requests (e.g., inject defaults)
• Validate policies (e.g., security constraints)

Examples:

• PodSecurity
• ResourceQuota

5. Validation

Now Kubernetes validates:

• Schema correctness
• Required fields
• API compatibility

👉 Invalid requests are rejected

6. Persistence in etcd
Finally:

• Data is stored in etcd, a distributed key-value store
• This becomes the source of truth for the cluster

End-to-End Workflow

User → kubectl → API Server → etcd → Scheduler → Node → Kubelet → Container Runtime → Pod

Step-by-step:

1. User sends request via kubectl
2. API Server authenticates and authorizes
3. Request is validated and stored in etcd
4. Scheduler → assigns node
5. Kubelet (on node) → takes over
6. Kubelet → talks to container runtime (containerd / CRI-O)
7. Runtime → pulls image + creates container
8. Runtime creates and starts containers
9. Pod becomes running

👉 All coordination happens via the API Server.

Real-World Analogy

Think of the API Server as a corporate receptionist:

• You arrive (request)
• ID is checked (Authentication)
• Permission is verified (Authorization)
• Rules are enforced (Admission Controllers)
• Entry is logged (etcd
• You’re directed to the right department (Scheduler/Controller)

👉 No one bypasses the receptionist.

Interview Questions

1. What is the Kubernetes API Server?

Answer: It is the central control plane component that exposes Kubernetes APIs and processes all requests.

2. Does API Server store cluster data?

Answer: No, it stores data in etcd.

3. What is the request flow in API Server?

Answer: Authentication → Authorization → Admission → Validation → Persistence

4. What are Admission Controllers?

Answer: Plugins that mutate or validate requests before they are persisted.

5. Can Kubernetes components bypass API Server?

Answer: No, it is the single entry point.

Summary (TL;DR)

• API Server is the brain of Kubernetes
• All requests pass through it
• It ensures:

Security

Validation

Consistency

• It does not store data — etcd does

What’s Next

👉 Episode 2: etcd Deep Dive — Kubernetes Memory Explained

Episode 1: Kubernetes API Server Explained — The Brain Behind Your Cluster was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Backing up critical data to the cloud is essential for modern server management. OneDrive is a popular choice, but mounting it on a Linux server — especially an Alma-Linux WHM server — requires some configuration. In this post, I’ll walk you through how to mount OneDrive using rclone, generate the necessary authentication token from Windows, verify it, and configure WHM to store backups directly in OneDrive.

Step 1: Install Rclone on AlmaLinux

Rclone is a powerful command-line tool that supports many cloud storage services, including OneDrive.

sudo dnf install epel-release -y
sudo dnf install wget unzip fuse -y
curl https://rclone.org/install.sh | sudo bash
rclone version

This installs rclone and ensures it’s ready to use.

Step 2: Generate OneDrive Token on Windows

Since AlmaLinux servers often do not have a GUI browser, you need a separate machine (Windows) to generate the token.

1. Download Rclone for Windows from https://rclone.org/downloads/ and extract it, e.g., C:\rclone.
2. Open Command Prompt or PowerShell and navigate to the folder:

cd C:\rclone

3. Run the authorize command:

rclone authorize onedrive

4. A browser window opens. Sign in to your personal Outlook/OneDrive account.

5. Allow rclone access.

6. Copy the entire JSON token printed in the Command Prompt. It will look like:

{"access_token":"...","token_type":"Bearer","expiry":"...","refresh_token":"..."}

This token will be used to configure rclone on your AlmaLinux server.

Step 3: Configure OneDrive Remote on AlmaLinux

1. Start rclone configuration on the server:

rclone config

2. Create a new remote:

• n → new remote
• Name: onedrive
• Storage: 38 → Microsoft OneDrive
• Leave client_id, client_secret, and tenant empty
• Advanced config? → n
• Browser auth? → n

3. At the config_token> prompt, paste the JSON token generated from Windows.

4. When prompted for config_type>, type:

1

(for OneDrive Personal or Business)

5. When prompted for config_driveid>, press Enter to select the default personal drive.

Step 4: Verify the Remote

Check access to your OneDrive folders:

rclone ls onedrive:Documents

Example output:

4451 Screenshot 2025-09-24 153807.png

Step 5: Mount OneDrive on AlmaLinux

1. Create a mount point:

sudo mkdir -p /mnt/onedrive
sudo chown root:root /mnt/onedrive

2. Test mount:

rclone --vfs-cache-mode writes mount onedrive: /mnt/onedrive &

• --vfs-cache-mode writes ensures WHM can safely write temporary files.
• Verify:

ls /mnt/onedrive/Documents

Step 6: Make the Mount Persistent with systemd

Create /etc/systemd/system/rclone-onedrive.service:

[Unit]
Description=Mount OneDrive
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone --vfs-cache-mode writes mount onedrive: /mnt/onedrive
Restart=always
User=root
Group=root

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable rclone-onedrive
sudo systemctl start rclone-onedrive
sudo systemctl status rclone-onedrive

Step 7: Configure WHM Backups

1. Log in to WHM → Backup Configuration.
2. Go to Additional Destinations → File System.
3. Set the path to:

/mnt/onedrive

4. Configure backup options (compression, retention, incremental).

5. Run a test backup to ensure it writes correctly.

✅ Tip: If you are using a custom backup script with a custom backup path, make sure to update the script to point to the mounted path (/mnt/onedrive). Otherwise, backups may still go to the old location.

✅ Tip: The mount must be available at all times; otherwise, backups may fail.

Conclusion:

Mounting OneDrive on Alma-Linux using rclone is straightforward once you understand the authentication flow. By generating the token from Windows and configuring a persistent mount, you can safely store backups directly in OneDrive. This approach is ideal for MSSQL backups, large files, and automated server backup strategies.

In this blog, I’ll show you how to generate AWS architecture diagrams using Amazon Q CLI and MCP servers. This is perfect for architects, developers, DevOps engineers, and anyone who wants to bring clarity and speed into cloud architecture work.

🌐 What is Model Context Protocol (MCP)?

MCP is an open protocol designed to standardize interactions between AI models and real-world tools like infrastructure-as-code frameworks, diagram renderers, or CLI tools.

• LLMs are like smart assistants — but they need a way to act.
• MCP defines a consistent format for tools (servers) to expose functions and for AI agents (clients) to call them.

✅ Why MCP is a Big Deal

• Interoperability: You can swap tools or hosts without reconfiguration.
• Scalability: Build powerful multi-tool agents using a single protocol.
• Standardization: All big players — AWS, Google, OpenAI, Microsoft — now support MCP.

🤖 What is Amazon Q CLI?

Amazon Q CLI is a command-line AI assistant built for developers. With recent MCP support, it can now do things like:

• Generate full-stack apps
• Create infrastructure code
• Build AWS architecture diagrams automatically
• Act as a lightweight agent for DevOps automation

⚙️ It’s like combining ChatGPT + CLI + DevOps + AWS tools into one.

🧰 Setup Guide

🔧 Step 1: Install Amazon Q CLI

macOS

brew install amazon-q

Ubuntu / WSL

sudo apt-get update
sudo apt install libfuse2
curl -sSf https://desktop-release.q.us-east-1.amazonaws.com/latest/amazon-q.deb -o amazon-q.deb
sudo apt install -y ./amazon-q.deb

🔐 Step 2: Create AWS Builder ID

You’ll need an AWS Builder ID to log in to Q CLI:

1. Visit: https://profile.aws.amazon.com
2. Click on “Create Builder ID”
3. Sign up with your email
4. Confirm email through AWS verification link
5. Done!

🔑 Step 3: Login to Amazon Q

q login

This opens a browser to log in using your Builder ID email. Once done, your terminal will say: Login successful.

🔓 You’re now ready to use Q CLI for free — no AWS billing required for Q features.

🖼️ Step 4: Install Graphviz (Diagram Dependency)

Graphviz helps render diagrams locally.

macOS:

brew install graphviz

Ubuntu:

sudo apt install graphviz

🛰️ Step 5: Install UV & Configure MCP Servers

Install the MCP runtime tool that runs your AI-powered servers:

sudo snap install astral-uv --classic

Then configure Amazon Q CLI to connect with MCP servers. These servers act as the “skills” or “tools” that your AI can use.

📄 Create MCP Configuration File

Create the file:

mkdir -p ~/.aws/amazonq
vim ~/.aws/amazonq/mcp.json

Paste the following code:

🧩 What this configuration does:
The mcp.json file tells Amazon Q CLI how to start and connect to two specific MCP servers:

• cdk-mcp-server: Generates AWS CDK (infrastructure-as-code) projects.
• aws-diagram-mcp-server: Generates AWS architecture diagrams using Python + Graphviz.

These servers run using the uvx runtime and are launched automatically when you use relevant prompts in the CLI.

{
  "mcpServers": {
    "awslabs.cdk-mcp-server": {
      "command": "uvx",
      "args": ["awslabs.cdk-mcp-server@latest"],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      }
    },
    "awslabs.aws-diagram-mcp-server": {
      "command": "uvx",
      "args": ["awslabs.aws-diagram-mcp-server"],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "autoApprove": [],
      "disabled": false
    }
  }
}

Now relaunch the CLI:

q

✅ Your Amazon Q CLI is now MCP-enabled and ready to generate diagrams or infrastructure code on-demand from natural language prompts.

✨ Let’s Generate a Diagram

Run:

q chat

Prompt:

Create an AWS architecture diagram for a serverless data pipeline using S3, Lambda, and DynamoDB.

Amazon Q will:

1. Understand your request
2. Trigger the AWS Diagram MCP Server
3. Convert the idea to Python Diagrams code
4. Render the output using Graphviz
5. Save image in your local machine (default: ~/.amazonq)

✅ Your diagram is ready!

🔁 Real-World Prompts to Try

Prompt

Build a three-tier web architecture with ALB, ECS, and RDS

Create architecture for a secure VPC with public/private subnets and NAT Gateway

Generate diagram for a CI/CD pipeline using CodePipeline, CodeBuild, and Lambda

💻 Developer Workflows (Quick Use Cases)

• Pre-Design Brainstorming — Quickly visualize architectures while discussing ideas.
• Documentation Automation — Add AI-generated diagrams to README, Confluence, or Notion docs.
• CI/CD Integration — Use in pipelines to auto-generate architecture visuals for changes.
• Infra Reviews — Share clear diagrams during pull requests or retros.

🛠️ Troubleshooting Tips (Quick Fixes)

• Browser doesn’t open on login — Run q login --no-browser and open the URL manually.
• No diagrams are generated — Make sure Graphviz is installed and dot is available in your system PATH.
• MCP servers not detected — Check that mcp.json exists in ~/.aws/amazonq and is correctly formatted.
• It’s too slow on first use — Be patient; dependencies might be downloading in the background.

🎯 Best Practices

• Use clear and descriptive prompts — the more specific, the better the results.
• Keep your mcp.json file in version control for consistency across your team.
• Combine this with IaC tools like CDK or Terraform to complete your automation workflow.
• For longer prompts or structured inputs, use q chat /editor.

🤝 Final Thoughts

Using Amazon Q CLI + MCP transforms how you build, explain, and document cloud systems. Whether you’re:

• Pitching a new architecture
• Creating client deliverables
• Documenting infra for your team
• Or building projects on the fly

This toolchain helps you move fast with clarity and confidence.

📬 Let’s Connect

🔗 LinkedIn