Bootstrap Flux CD on EKS Fargate using AWS CodeCommit

architecture

Hello AWS Enthusiasts, today we’ll be deploying Flux CD on EKS Fargate using AWS CodeCommit the GitOps way!

Before we deep dive into the above architecture and its implementation, let me quickly explain the 2 new keywords that we just came across i.e. “GitOps” & “Flux CD”.

GitOps is a way of managing your infrastructure and applications so that whole system is described declaratively and version-controlled (most likely in a Git repository or even a S3-compatible bucket), and having an automated process that ensures that the deployed environment matches the state specified in a repository.

Flux CD is one of the tools that make GitOps possible by running a continuous reconciliation loop through its various controllers and syncing your current state with the desired state and even vice-versa.

And now without further ado, let’s jump straight into the architecture implementation.

Prerequisites:

We would need an EKS Fargate cluster to run the Flux components on, refer to my medium article on how to create one which also includes creating a Fargate profile for Flux. Make sure to export the kubeconfig file as it will be used by the Flux provider.

We would also need AWS CLI for our setup. You can follow this installation guide.

Create the below the Terraform files.

variables.tf declares the necessary input variables that will be used by the various resources in our setup.

variables/local.tfvars populates the above variables with the user-supplied values. Update these variable assignments with your choice of relevant values.

data.tf has the necessary data sources to fetch information from external sources for eg. your AWS account. We have added a depends_on to the aws_secretsmanager_secret_version data source because without it terraform can try to read the secret version before it gets created and execution will fail.

vpc-endpoint.tf as the name suggests defines an interface VPC endpoint for the “git-codecommit” service. We didn’t go with the “codecommit” service for our VPC endpoint because “git-codecommit” service type is responsible for Git operations with CodeCommit repositories for example using a Git client to run commands such as git pull, git commit, and git push , while the “codecommit” service type is responsible for CodeCommit API operations for example using the AWS CLI, the CodeCommit API, or the AWS SDKs to interact with CodeCommit for operations such as CreateRepository, ListRepositories, and PutFile. Flux CD will only be performing Git operations and not CodeCommit API operations when reconciling Kubernetes resources thus the aforementioned service type choice. VPC Endpoints help keep their service-specific traffic within the AWS network thus increasing security. We have also created a security group for the VPC endpoint and are allowing ingress from our private subnet's CIDRs. Note that we have set private_dns_enabled as truethis is done so that we can use the service’s default public endpoint DNS name while still leveraging Amazon’s private network connectivity. AWS behind the scenes associates a private hosted zone with the VPC that contains a record set matching the service’s default public endpoint DNS name. Otherwise, we’ll have to use the endpoint-specific DNS hostnames to stay within the AWS network. To use this feature, ensure that the attributes ‘Enable DNS hostnames’ and ‘Enable DNS support’ are enabled for your VPC.

kms.tf defines a Customer Managed KMS key which will be used by CodeCommit to encrypt data at rest. We have also defined a key policy with the AWS account root user ARN as the principal that doesn’t automatically grant any permissions on the key to any of the account identities but at the same time enables IAM users/roles to use identity-based policies as well as key policies to get the required access on this key thus following the principle of least privilege here. deletion_window_in_days is set for 7 days during which you can cancel the key deletion to prevent accidental loss of data that was encrypted with this key. You are not charged for KMS keys that are in a pending deletion stage.

repo.tf defines a CodeCommit repo and uses the KMS key defined above for at-rest encryption. We have also defined a null_resource to run a local provisioner that initializes the repo for Flux by simply pushing a file with base64 encoded data to the specified branch. This step is required because our newly created CodeCommit repo is empty and Flux throws an error if we try to bootstrap it on an empty repo. More details on this open issue.

credentials.tf defines an IAM user who has read and write access to our CodeCommit repo and also Encrypt & Decrypt access on the KMS key configured on the repo. We have also added a time_sleepresource of 30 seconds to give enough soak time for our policies to take effect. To make sure this 30-second delay always happens whenever there is a change in the policy, we have added a triggers block and added the policy document as a parameter to check for changes to run the time_sleep resource. And finally to authenticate we are creating Git credentials for HTTPS connections to our CodeCommit repo. These credentials will be associated with our IAM user and will have CodeCommit and KMS permissions as defined.

secrets.tf defines a secret that stores the HTTPS Git credentials we created above. We have set recovery_window_in_days to 0 because this helped in deleting secrets quickly during testing but you may want to set it to a value between 7 & 30 as this prevents accidental secret deletion by giving you a buffer time to restore the secret before it gets permanently deleted.

providers.tf declares the necessary terraform providers to create the resources. In the flux provider, we are passing in the CodeCommit repo URL, the branch name to clone from and push the Flux yamls to, and the Git credentials created above. Flux provider will look for a kubeconfig file at the path ./kubeconfig/config. We have also defined a required_providers section that specifies the provider's sources and their version constraints. The ~> operator allows only the rightmost version component to increment.

flux.tf does all the magic of installing the flux controllers, creating the necessary CRDs, namespaces, network policies, etc, configuring the GitRepository and Kustomization resources with appropriate values as specified in the Flux provider and the flux_bootstrap_git resource, reconciling Kubernetes resources and Flux itself, cloning, and pushing the Flux yamls to the repo. We have specified a path i.e. eks-fargate/applications under which Flux will look for its desired state and reconcile the current state running on the cluster with it. We have specified the namespace i.e. fluxcd in which Flux controllers will be deployed. We have defined an interval of 30 seconds for Flux to check for changes in the desired state. We have set watch_all_namespaces to true because this lets Flux monitor its CRDs in all the namespaces otherwise it will only look in the namespace where the Flux controllers are installed. We have added a bunch of resources to the depends_on meta-argument to make sure flux gets bootstrapped only after the repo has been initialized, VPC endpoints and KMS key policies have been created and the necessary identity-based policies have gotten enough soak time otherwise Flux will error out.

This wraps up creating all the terraform files needed for our setup. Now let’s deploy our architecture. Run the below command. Wait for terraform apply to succeed.

terraform apply  --var-file variables/local.tfvars --refresh=false

Now let’s take a look at the below screenshots, to get a better understanding of our entire setup!

Deployments have been created for all 4 Flux controllers in the fluxcd namespace.

Flux yamls have been pushed to the CodeCommit repo in the flux-branch branch.

CodeCommit Git credentials have been stored in the Secrets Manager.

KMS Key has been created for at-rest CodeCommit encryption.

This key has been set as the repository encryption key for our CodeCommit repo.

GitRepository resource is created which checks the CodeCommit repo every 30 seconds in the flux-branch branch for new revisions and creates a tarball (.tar.gz) artifact with the fetched data and stores it in the source controller deployment when the current GitRepository revision differs from the latest fetched revision

Kustomization resource is created which defines a pipeline for fetching, decrypting, building, validating, and applying Kustomize overlays or plain Kubernetes manifests. sourceRef point to the fluxcd GitRepository object which has the required Artifact containing the YAML manifests. It looks under the path eks-fargate/applications in the Source Artifact containing the kustomization.yaml file.

Let’s test the reconciliation flow by dropping a file to CodeCommit.

Run the below command to push a file to CodeCommit. Parent commit ID you can get from the commits section of the repo in the console. Remember file content has to be base64 encoded, so encode the nginx deployment yaml below. Take note of the commit IDs produced from this command.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: fluxcd
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

We can see the current state has been reconciled and nginx deployment was created in the EKS cluster.

The same can be observed from the source controller and kustomize controller logs. The artifact was created and reconciled successfully for our latest commit ID.

To keep your AWS bills in check, destroy the infrastructure as soon you are done creating and testing the entire setup by running the below command

terraform destroy --auto-approve -var-file variables/local.tfvars

yay!

This marks the end of our “Bootstrap Flux CD on EKS Fargate using AWS CodeCommit” blog.
If you have any questions/suggestions please add them in the comments.
If you learned anything new today, please consider giving a clap👏, it keeps me motivated to write more AWS content. 😀