End-to-End DevOps: Kubernetes Project Management with GitLab CI/CD on AWS EKS

This article provides a guide on how to implement DevOps principles and GitLab CI/CD tools at each stage of the software development process. Within this title, strategies, best practices, and tips for effectively managing projects on Kubernetes are addressed.

What is GitLab CI/CD?

GitLab CI/CD is the part of GitLab that you use for all of the continuous methods (Continuous Integration, Delivery, and Deployment). With GitLab CI/CD, you can test, build, and publish your code with no third-party application or integration needed.

Read more about GitLab CI/CD here.

Prerequisite:

• AWS & GitLab Account
• Basic understanding of AWS & GitLab CI/CD
• An access key & secret key created in AWS

Lets, start with the configuration of the project

Step 1:- Create a Repository

• If working independently, create a project/repository, or if within a company, establish a project/repository under a designated group. Ensure that your project/repositories are set to private. This is crucial because your access key and secret key will be stored in the variables section.

Step 2: After cloning created repository to your local environment, add the following files to your repo. https://gitlab.com/mehmetkanus17/app.git

Step 3:- Store AWS keys

• In order to create the resources in AWS account, we must need to have AWS Access Key & AWS Secret Key
• Now, we need to store AWS Access Key, AWS Secret Key & Region in the secrets section of the repository
• Under group name or username go to settings -> CI/CD -> Under the variable section create the below variables and store your AWS access keys, AWS secret keys & Region

Step 4:- Add deploy tokens

• To enable Kubernetes applications to pull images from GitLab Container Registry, you should add a token at the Settings/Repository/Deploy tokens menu.

Step 4:- Create a workflow file

• For your application to be automatically deployed to AWS EKS, you need to create a workflow file.
• Create .gitlab-ci.yml file and add the below code to it
• The below job will run on every push and pull request that happens on the main branch.

stages:
  - creating-cluster
  - build
  - deploy

variables:
  WEB_IMAGE: $CI_REGISTRY_IMAGE/web
  RESULT_IMAGE: $CI_REGISTRY_IMAGE/result
  
creating-cluster:
  stage: creating-cluster
  image:
    name: alpine/k8s:1.28.4
  script:
    - cd create-cluster
    - eksctl create cluster -f clusterconfig.yaml

    - aws eks update-kubeconfig --name m6a-cluster --region us-east-1

    - kubectl create secret docker-registry gitlab -n default --docker-server="registry.gitlab.com" --docker-username=XXXXXX --docker-password"XXXXXXX"

result-build:
  stage: build
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - cd result_image
    - docker build -t "${RESULT_IMAGE}:latest" .
    - docker build -t "${RESULT_IMAGE}:${CI_COMMIT_SHORT_SHA}" .
    - docker push "${RESULT_IMAGE}:latest"
    - docker push "${RESULT_IMAGE}:${CI_COMMIT_SHORT_SHA}"

web-build:       # This job runs in the build stage, which runs first.
  stage: build
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - cd web_image
    - docker build -t "${WEB_IMAGE}:latest" .
    - docker build -t "${WEB_IMAGE}:${CI_COMMIT_SHORT_SHA}" .
    - docker push "${WEB_IMAGE}:latest"
    - docker push "${WEB_IMAGE}:${CI_COMMIT_SHORT_SHA}"

mysql-deploy:
  stage: deploy
  image:
    name: alpine/k8s:1.23.7
  script:
    - aws eks update-kubeconfig --name m6a-cluster --region us-east-1
    - cd mysql_deployment
    - kubectl apply -f .

web-deploy:
  stage: deploy
  needs:
    - mysql-deploy
  image:
    name: alpine/k8s:1.28.4
  script:
    - aws eks update-kubeconfig --name m6a-cluster --region us-east-1
    - cd web_server
    - kustomize edit set image "${WEB_IMAGE}=:${CI_COMMIT_SHORT_SHA}"
    - kubectl apply -k .

result-deploy:
  stage: deploy
  needs:
    - mysql-deploy
  image:
    name: alpine/k8s:1.28.4
  script:
    - aws eks update-kubeconfig --name m6a-cluster --region us-east-1
    - cd result_server
    - kustomize edit set image "${RESULT_IMAGE}=:${CI_COMMIT_SHORT_SHA}"
    - kubectl apply -k .

• In GitLab CI/CD workflow file, a container will be created for each job under every stage, using the required image to execute the necessary commands. Therefore, you should create each job with an appropriate image to run the required commands.
• For example, a container will be created with the image alpine/k8s:1.28.4 for the “creating-cluster” job. This is because alpine/k8s:1.28.4 image includes tools such as kubectl, eksctl, helm, aws cli, making it suitable for the required commands in this job.

• Exactly, as you mentioned, a container for the “web-build” job will be created with docker:24.0.5 image. This choice is made because docker:24.0.5 image includes the necessary components to execute commands like docker build and docker push. It’s important to select the appropriate image for each job to ensure that the required tools and dependencies are available during the CI/CD process.

• You can define and use variables according to your needs.

• To ensure that all jobs are executed when the pipeline is initially triggered, you should add “run-all” to the commit message. Subsequently, when you only want to run application jobs without executing the “creating-cluster” job, you should include the appropriate parameter in the commit message. For instance, for all jobs, use “run-all,” and for only the web application job, you can use “run-web” in the commit message. This way, you can control the execution based on the specified parameters in the commit messages.

....
web-build:
  rules:
    - if: '($CI_COMMIT_MESSAGE =~ /.*run-web.*/ || $CI_COMMIT_MESSAGE =~ /.*run-all.*/)'
  stage: build
....
web-deploy:
  rules:
    - if: '($CI_COMMIT_MESSAGE =~ /.*run-web.*/ || $CI_COMMIT_MESSAGE =~ /.*run-all.*/)'
  stage: deploy

• The provided Kustomize block serves to dynamically set the image tag for a Kubernetes deployment. kustomize edit set image "${WEB_IMAGE}=:${CI_COMMIT_SHORT_SHA}" line modifies the image tag in the Kustomize configuration, replacing ${WEB_IMAGE} with the specified image name and appending :${CI_COMMIT_SHORT_SHA} to use the short SHA of GitLab CI/CD commit. This ensures that each deployment gets a unique image tag based on the corresponding commit, aiding in versioning and traceability.

....
script:
    - aws eks update-kubeconfig --name m6a-cluster --region us-east-1
    - cd web_server
    - kustomize edit set image "${WEB_IMAGE}=:${CI_COMMIT_SHORT_SHA}"
    - kubectl apply -k .
....

Step 5: Check the output

• Now, As soon as you commit your workflow file GitLab will trigger the action and the resources will be going to create on AWS EKS and Gitlab Container Registry.
• After running the job you will see that all the steps run perfectly and there was no error. So you will have passedwritten in the green color as each step job successfully.

• You can also check the output of each step by clicking on it

AWS EKS resources.

• cluster and nodes

• add-ons

• pods

• gitlab container registry

• Don’t forget to delete all resources.

eksctl delete cluster m6a-cluster — region us-east-1

Bonus: Consider using Kaniko for the build stage instead of Docker. Kaniko is a tool to build container images from a Dockerfile inside a container or Kubernetes cluster. It allows you to perform container builds without the need for Docker daemon privileges. This can enhance security and flexibility in your CI/CD pipeline. To integrate Kaniko into your GitLab CI/CD workflow, you can update your CI script accordingly, replacing Docker commands with the equivalent Kaniko commands. Ensure to adjust your pipeline configuration to accommodate Kaniko’s usage and take advantage of its benefits in container image building.”

web-build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.14.0-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker && echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - CONTEXT="${CI_PROJECT_DIR}/web_image"
    - /kaniko/executor
      --context "${CONTEXT}"
      --dockerfile "${CONTEXT}/Dockerfile"
      --destination "${WEB_IMAGE}"
      --destination "${WEB_IMAGE}:${CI_COMMIT_SHORT_SHA}"

result-build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.14.0-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker && echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - CONTEXT="${CI_PROJECT_DIR}/result_image"
    - /kaniko/executor
      --context "${CONTEXT}"
      --dockerfile "${CONTEXT}/Dockerfile"
      --destination "${RESULT_IMAGE}"
      --destination "${RESULT_IMAGE}:${CI_COMMIT_SHORT_SHA}"