Kubernetes Simplified: Building versatile environments with Flux and vCluster on AWS

9 min readDec 25, 2023

Let’s dive into a practical and efficient way to manage Kubernetes environments using Flux and vCluster. Our mission? To set up an adaptable, scalable environment that smoothly transitions through various stages of application development – from development to QA, and all the way to production. The magic happens by harnessing virtual Kubernetes clusters, all neatly housed within a single, physical cluster on Amazon Web Services (AWS). This setup is not just about making things work; it's about making them work smartly and seamlessly.

Introduction

The Foundation: Creating a Kubernetes Management Cluster

Our setup starts with constructing a single physical Kubernetes cluster, known as the management cluster. While we choose Amazon Web Services (AWS) for its well-established reliability and scalability, this approach is flexible and can be adapted to other cloud providers that support Kubernetes and vCluster requirements, particularly in terms of persistent volume claims and Kubernetes distribution compatibility.

We utilize AWS’s Elastic Kubernetes Service (EKS) and manage it through eksctl, a user-friendly command-line interface (For a real world project, a proper terraform setup would be more suitable).

The choice of AWS is guided by its industry-wide adoption, which provides a level of assurance and community support. However, the principles we employ are applicable across various cloud environments. The critical aspects, such as support for persistent volume claims and vCluster compatibility, should be the guiding factors in the selection process.

The Power of Flux: Centralized Management and Automation

At the heart of our setup is Flux, a tool that epitomizes the GitOps methodology. Flux continuously monitors a Git repository containing our Kubernetes resource definitions and ensures that the cluster’s state matches the configurations stored in this repository. This setup offers several advantages:

Automated Synchronization: Changes made in the Git repository are automatically applied to the cluster, streamlining the deployment process.
Audit Trails and Rollbacks: Every change is tracked and version-controlled, allowing for easy auditing and rollbacks if needed.
Consistency and Reliability: Ensuring that the cluster’s state always reflects the configuration in Git reduces discrepancies and errors.

Virtual Clusters for Isolated Environments

vCluster comes into play by enabling us to create virtual Kubernetes clusters within the physical AWS cluster. These virtual clusters are lightweight, isolated environments that mimic standalone Kubernetes clusters. This approach offers several benefits:

Resource Efficiency: Virtual clusters share the underlying resources of the physical cluster, making them more resource-efficient than running multiple separate physical clusters.
Isolation of Environments: Each virtual cluster (development, QA, production) operates in isolation, ensuring that activities or issues in one environment do not impact the others.
Simplified Management: Managing multiple environments becomes easier as they all reside within the same physical cluster.

Streamlining Virtual Cluster Management with Flux

In our Kubernetes setup, the Flux Controllers, running in the management cluster, is pivotal for managing both the deployment of virtual clusters and the applications within them. This dual functionality enhances the efficiency and effectiveness of our environment management.

Deploying Virtual Clusters

The Helm Controller with the help of the other Flux controllers automates the creation of virtual clusters. This approach ensures automated, version-controlled setup of the development, QA, and production environment.

Managing Infrastructure and Applications

Once the virtual clusters are operational, the Kustomize Controller oversees application deployment within these environments. By utilizing remote kubeconfigs, it seamlessly manages and synchronizes application states across all virtual clusters, aligning them with our Git-defined configurations.

Let’s go…

Step 1: Setting Up a Kubernetes Cluster on AWS with eksctl

The journey begins with installing a Kubernetes cluster on AWS. For this, eksctl is our tool of choice, known for its efficiency and simplicity in managing EKS clusters. We'll be using Kubernetes version 1.28 to ensure compatibility with the vCluster version we are using. The installation process involves configuring the cluster settings to match our specific requirements, including node types, sizes, and the number of nodes.

After setting up the cluster, we need to enable Amazon EBS CSI (Container Storage Interface) support. This is essential for handling persistent volume claims in Kubernetes, particularly for stateful applications. To integrate the EBS CSI Driver as an EKS add-on, we’ll configure IAM Roles for Service Accounts (IRSA). IRSA is a critical component for securely managing permissions between Kubernetes service accounts and AWS resources, and it’s especially necessary for the EBS CSI Driver to function correctly.

The entire process of installing the cluster, setting up EBS CSI support via EKS add-on, and configuring IRSA can be thoroughly understood through a detailed Medium article. This article provides a comprehensive guide, offering step-by-step instructions and insights for a successful setup.
Make sure to name the cluster “management-cluster” and to use kubernetes version 1.28.

With this foundational step complete, we’re ready to move on to integrating Flux and setting up our virtual clusters.

Step 2: Installing and Configuring Flux in the AWS Kubernetes Cluster

Having set up our physical Kubernetes cluster with eksctl, we now move to the task of installing and configuring Flux. This will lay the groundwork for our GitOps-driven approach to manage the virtual environments.

Begin by installing the Flux CLI tool, which is essential for setting up Flux in our cluster. The installation process varies slightly depending on your operating system, and detailed instructions can be found in the official Flux documentation.

With the CLI tool installed, the next step is to bootstrap Flux into your Kubernetes cluster. Bootstrapping is the process of installing Flux components and setting up the necessary permissions and configurations. Run the following command, replacing `[your-git-repository]` with the URL of your Git repository where your Kubernetes configurations will be stored:

flux bootstrap github \
 -- owner=[your-github-username] \
 -- repository=[your-git-repository] \
 -- branch=main \
 -- path=./clusters/management-cluster \
 -- personal

This command configures Flux to synchronize with your specified Git repository, effectively linking your cluster’s state to your version-controlled configurations.

After bootstrapping, it’s good practice to verify that Flux has been successfully installed and is operational. You can check the status of Flux components by running:

flux check

This command will provide a status report, indicating if Flux is properly installed and running in your cluster.

Step 3: Installing the Environments with Flux

In this chapter, we focus on deploying our three virtual Kubernetes environments — development, QA, and production — using Flux. This setup leverages Flux’s capabilities to manage complex deployments, ensuring that each environment is configured and maintained efficiently.

In the folder /clusters/management-cluster we create a helm repository with a file helm-repository.yaml:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: loft-sh
  namespace: flux-system
spec:
  interval: 30m
  url: https://charts.loft.sh

In the same folder, we add Kustomizations, one for each environment by adding 3 files vcluster-<envionment>.yaml. For the dev environment, the file looks like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: vcluster-<environment>
  namespace: flux-system
spec:
  interval: 10m0s
  prune: true 
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: "./vcluster-template"
  wait: true
  postBuild:
    substitute:
      env: dev # Change this value for each environment

In the folder ./vcluster-template we create the template referenced above, called vcluster-template.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: env-${env}
  namespace: flux-system
spec:
  interval: 1m
  releaseName: env-${env}
  targetNamespace: env-${env}
  chart:
    spec:
      chart: vcluster-eks
      version: 0.18.1
      sourceRef:
        kind: HelmRepository
        name: loft-sh
  install:
    createNamespace: true
  values:
    syncer:
      extraArgs:
        - --tls-san=env-${env}.env-${env}.svc.cluster.local
        - --out-kube-config-server=https://env-${env}.env-${env}.svc.cluster.local

After pushing these changes to our gitops repository, let’s open a second terminal and observe flux doing the changes.
This can be done with this command:

flux get kustomizations --watch

Step 4: Deploy some workload to the environments

Now, we tell Flux to deploy some sample workloads to our virtual clusters.
To do so we add a folder env with the following subfolders to the root of our repository:

The test.yaml files deploys a forever-running container into a namespace `testi`:

kind: Namespace
apiVersion: v1
metadata:
  name: testi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: testi
spec:
  replicas: 1
  selector:
    matchLabels:
      app: perpetual
  template:
    metadata:
      labels:
        app: perpetual
    spec:
      containers:
        - name: forever-running-container
          image: alpine
          command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]

In the folder /clusters/management-cluster, we add Kustomizations, one for each environment by adding 3 files env-<envionment>.yaml (change <environment> to `dev`, `qa`and `prod`).

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: env-<environment>
  namespace: env-<environment>
spec:
  interval: 10m0s
  prune: true # remove stale resources from cluster
  sourceRef:
    kind: GitRepository
    namespace: flux-system
    name: flux-system
  path: "./env/<environment>"
  dependsOn:
    - name: vcluster-dev
      namespace: flux-system
  kubeConfig:
    secretRef:
      name: vc-env-<environment>
      key: config

After pushing these changes to our gitops repository, let’s again head over to the terminal where this command is running:

flux get kustomizations --watch

We should see all Kustomizations in state `True` or `Applied` after some time.

In practical projects, tools like Kustomize overlays or Flux HelmReleases are often employed for more organized and automated deployments across different environments. If you’re curious about exploring this topic further, feel free to drop me a note, and I could delve into it in another blog post.

Accessing the Virtual Clusters

Navigating and managing virtual Kubernetes clusters in a physical cluster environment works best with the use of the vcluster CLI. This chapter explains how to install and use the vcluster CLI to access our development environment, including the essential steps for listing virtual clusters and establishing a connection for direct access with tools like kubectl.

Installing the Vcluster CLI

To begin, install the vcluster CLI on your local machine. It's a straightforward process:

Download the vcluster CLI from the official vcluster downloads page.
Follow the instructions specific to your operating system to install the CLI.

Listing Available Virtual Clusters

With the vcluster CLI installed, you can list all virtual clusters:

vcluster list

This command shows all virtual clusters, helping you identify the specific one you need, such as the development environment.

Connecting to a Development Environment

To access the development environment, you need to set up port forwarding and then connect using vcluster. Open two terminals for this process.

Set Up Port Forwarding:
In the first terminal, execute the following kubectl command

kubectl port-forward -n env-dev svc/env-dev 8443:443

Connect to the Virtual Cluster:
In the second terminal, use the vcluster CLI to connect to the virtual cluster:

vcluster connect env-dev -n env-dev --server=https://localhost:8443

This command establishes a connection to the env-dev virtual cluster through the port-forwarded address.

Conclusion

In this guide, we’ve navigated the intricacies of setting up and managing multi-environment Kubernetes clusters using Flux and vcluster on AWS. From the initial steps of establishing a robust physical Kubernetes cluster with eksctl, to the advanced techniques of managing and accessing virtual clusters, we covered a range of essential topics for modern DevOps practices.

Key Takeaways:

Setting Up the Physical Cluster: We began by creating a physical Kubernetes cluster on AWS using eksctl, ensuring it supports Kubernetes version 1.28 for vCluster compatibility, along with EBS CSI for persistent storage and IRSA for secure AWS service integration.
Installing and Configuring Flux: Flux was introduced as the cornerstone for our GitOps approach, allowing us to automate deployment and manage Kubernetes resources effectively. We explored how to bootstrap Flux into our cluster and use it to synchronize our cluster state with our Git repository configurations.
Creating and Managing Virtual Clusters with vCluster: We delved into creating isolated virtual environments within our physical cluster using vCluster. This setup facilitated efficient resource utilization and environment separation, essential for managing different stages like development, QA, and production.
Accessing Virtual Clusters: The final piece of our setup involved accessing and interacting with these virtual clusters. We outlined the process of installing the vcluster CLI, setting up port forwarding, and connecting to virtual clusters, ensuring seamless management and interaction with each environment.

Throughout this journey, we emphasized the importance of a streamlined, scalable, and efficient Kubernetes setup. By leveraging the capabilities of AWS, eksctl, Flux, and vcluster, we demonstrated how to create a dynamic, GitOps-driven Kubernetes environment that caters to various operational needs.

This guide serves as a blueprint for DevOps professionals and Kubernetes enthusiasts looking to enhance their cluster management strategies and embrace the full potential of cloud-native technologies.

Additional Resources

For further exploration and deeper understanding of the tools and concepts covered in this guide, here’s a collection of resources and links.

Example Code Repository

flux-vcluster-example — github repository containing example code we have written in this article

Tools and Documentation

AWS EKS and eksctl:
AWS EKS — AWS Elastic Kubernetes Service
eksctl — The official CLI for Amazon EKS
Flux:
Flux Documentation — Official Flux CD Documentation
Flux GitHub Repository — Source code and more details on Flux
Vcluster:
vcluster Documentation — Comprehensive guide and documentation on vcluster on GitHub — GitHub repository for vcluster
Kubectl:
Kubernetes CLI (kubectl) — Documentation for Kubernetes CLI, kubectl
Kustomize:
Kustomize — Official Kustomize website