Open in app

Sign in

Write

Sign in

Title: Comprehensive CI/CD Pipeline with Terraform, Packer, Jenkins, SonarQube, Maven, Docker, Datree, and ArgoCD

Saad Ullah Khan Warsi
19 min readOct 9, 2023

--

Introduction

In today’s fast-paced development landscape, creating a robust and automated CI/CD pipeline is a critical component for delivering high-quality software efficiently. In this comprehensive guide, we will walk you through the process of setting up a full-fledged CI/CD pipeline using Terraform, Packer, Jenkins, SonarQube, Docker, Datree, and ArgoCD. Each step is designed to provide a deep dive into the various aspects of building a reliable and efficient development pipeline.

Architecture Diagram of Microservices CICD and AWS Infrastructure

Step-1 : Infrastructure as Code with Terraform

Before we embark on the journey of creating a CI/CD pipeline, it’s essential to establish a solid infrastructure foundation. Terraform, an Infrastructure as Code (IaC) tool, empowers us to define and provision infrastructure in a consistent and repeatable manner.

To create the EKS cluster, we will define its infrastructure as code using Terraform. This includes specifying the desired AWS region, node groups, and other critical settings. We’ll execute Terraform commands to provision the EKS cluster.

Terraform code is available at: https://github.com/saad946/Jenkins-SonarQube-FluxCD-CICD/tree/main/terraform

terraform init
terraform plan
terraform apply

Connect to the EKS cluster:

aws eks update-kubeconfig --name CLUSTERNAME --region REGIONNAME

Step-2 :Custom Jenkins and SonarQube Images with Packer

To ensure control and consistency in your CI/CD pipeline, custom images for Jenkins and SonarQube are vital. Packer enables us to create tailored images that include specific plugins and configurations.

Create Packer Templates for Jenkins and SonarQube Images:

  • Craft Packer templates that define the desired configurations for Jenkins and SonarQube.

Build Custom Images with Packer:

  • Execute Packer to build custom images for Jenkins and SonarQube.
  • Incorporate necessary Jenkins plugins and the SonarQube scanner for SonarQube integration.
#jenkins.pkr.hcl

# Variables
# variable "aws_access_key" {
#   type = string
# }

# variable "aws_secret_key" {
#   type = string
# }

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

# Sources (Builders)
source "amazon-ebs" "demo" {
  # access_key = var.aws_access_key
  # secret_key = var.aws_secret_key
  region                  = "ap-southeast-1"
  source_ami              = "ami-0df7a207adb9748c7"
  instance_type           = "t2.large"
  temporary_key_pair_type = "ed25519"
  ssh_username            = "ubuntu"
  ami_name                = "packer-jenkins-ami-${local.timestamp}"
}

# Builds
build {
  sources = [
    "source.amazon-ebs.demo",
  ]

  # Provisioners
  provisioner "shell" {
    inline = [
      "sleep 30",
      "sudo apt update -y",      
      "sudo apt upgrade -y",   
      "sudo apt install openjdk-17-jre -y",
      "curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null",
      "echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null",
      "sudo apt-get update -y",
      "sudo apt-get install jenkins -y",
      "sudo apt update -y",
      "sudo apt-get install -y ca-certificates curl gnupg",
      "sudo install -m 0755 -d /etc/apt/keyrings",
      "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg",
      "sudo chmod a+r /etc/apt/keyrings/docker.gpg",
      "echo \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null",
      "sudo apt-get update -y",
      "sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin",
      "sudo chmod 777 /var/run/docker.sock",
      "sudo systemctl start docker",
      
    ]
  }

  # Post-processors
  post-processor "manifest" {
    output = "manifest.json"
    strip_path = true
  }
}
#sonarqube.pkr.hcl

# Variables
# variable "aws_access_key" {
#   type = string
# }

# variable "aws_secret_key" {
#   type = string
# }

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

# Sources (Builders)
source "amazon-ebs" "demo" {
  # access_key = var.aws_access_key
  # secret_key = var.aws_secret_key
  region                  = "ap-southeast-1"
  source_ami              = "ami-0df7a207adb9748c7"
  instance_type           = "t2.large"
  temporary_key_pair_type = "ed25519"
  ssh_username            = "ubuntu"
  ami_name                 = "packer-sonarqube-ami-${local.timestamp}"
}

# Builds
build {
  sources = [
    "source.amazon-ebs.demo",
  ]

  # Provisioners
  provisioner "shell" {
    inline = [
      "sudo apt update -y",
      "sudo apt-get install -y ca-certificates curl gnupg",
      "sudo install -m 0755 -d /etc/apt/keyrings",
      "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg",
      "sudo chmod a+r /etc/apt/keyrings/docker.gpg",
      "echo \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable\" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null",
      "sudo apt-get update -y",
      "sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin",
      "sudo chmod 777 /var/run/docker.sock",
      "sudo systemctl start docker",
      "docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 sonarqube",
      
    ]
  }

  # Post-processors
  post-processor "manifest" {
    output = "manifest.json"
    strip_path = true
  }
}

# Commands
To init:
packer init <packer template file>
e.g. packer init jenkins.pkr.hcl
     packer init sonarqube.pkr.hcl

To validate packer template:
packer validate <packer template file>
e.g. packer validate jenkins.pkr.hcl
     packer validate sonarqube.pkr.hcl

to start image build:
packer build <packer template file>
e.g. packer build jenkins.pkr.hcl
     packer build sonarqube.pkr.hcl
AMIs for Jenkins and SonarQube via Packer

Step-3 :Setting Up GitHub Repository

Central to the CI/CD pipeline is the source code residing in a GitHub repository. Ensure your code is organized and follows best practices for efficient CI/CD integration.

Create a GitHub Repository for Your Project:

  • Establish a dedicated repository for your application code.
  • Commit and push your code to this repository for version control.

Step-4 :Jenkins CI Setup

With the infrastructure and custom images in place, it’s time to set up Jenkins for continuous integration:

Install Jenkins on a Server Using Custom Jenkins Image:

  • Deploy Jenkins on a server using the custom Jenkins image created earlier.

Add Necessary Jenkins Plugins:

  • Enhance Jenkins functionality by installing essential plugins, including SonarQube and SonarQube Quality Gate, Docker and Docker Pipeline.
  • Also make sure to install AWS CLI, Docker and Helm CLI on Jenkins Server under jenkins user with switching sudo su -jenkins.

Create a Role for Jenkins:

  • Define a role for Jenkins with appropriate permissions.
  • Grant this role full access to the Elastic Container Registry (ECR) and SSM Manager.

Attach the Jenkins Role to the Jenkins Server:

  • Associate the Jenkins role with the Jenkins server to enable interaction with ECR.
Jenkins UI at port 8080
Docker Plugins
Jenkins erver role

Step-5 :SonarQube Integration with Jenkins:

Ensure that your SonarQube server is up and running, and it is accessible on port 9000. You can access it via a web browser by navigating to http://your-sonarqube-server:9000.

Initial Configuration:

  • Upon logging in for the first time, SonarQube may prompt you to change the admin password for security reasons. Follow the on-screen instructions to update the password.
SonarQube UI login

Generate an Authentication Token:

  • To enable Jenkins to interact with SonarQube, it’s a good practice to generate an authentication token. This token will be used by Jenkins to authenticate with SonarQube.
  • Navigate to your SonarQube server by clicking on your username (admin) in the top right corner and selecting “My Account.”
  • In the “Security” section, click on “Security” again to access the security settings.
  • Click on “Generate Token” to create an authentication token. Provide a name for the token (e.g., “Jenkins Integration”) and click “Generate.”
Auth step 1 server
Auth step 2 in SonarQube server

Configure Jenkins Integration with SonarQube:

  • In your Jenkins pipeline or job configuration, you’ll need to configure the integration with SonarQube. This typically involves using the SonarQube Scanner for Jenkins.
  • Install the SonarQube Scanner plugin in Jenkins if not already installed.
  • In your Jenkins job configuration, you should find a section related to SonarQube analysis. Configure it with the following details:
  • SonarQube server URL: http://your-sonarqube-server:9000
  • SonarQube authentication token: Use the token generated in previous step.
  • Other relevant settings, such as project key, sources directory, etc.
Auth step 1 in Jenkins server
Auth step 2 in Jenkins server

Step-6 :Maven Build and SonarQube Integration

To ensure code quality and seamless integration, configure Jenkins to build your code with Maven and integrate it with SonarQube:

Set Up Maven for Building:

  • Configure Jenkins to build your code using Maven as the build tool.
  • Place your Maven file in git root folder which includes mnvw, mnvw.cmd and pom.xml.
#mnvw
#!/bin/sh
# ----------------------------------------------------------------------------
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#    https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
# ----------------------------------------------------------------------------

# ----------------------------------------------------------------------------
# Maven Start Up Batch script
#
# Required ENV vars:
# ------------------
#   JAVA_HOME - location of a JDK home dir
#
# Optional ENV vars
# -----------------
#   M2_HOME - location of maven2's installed home dir
#   MAVEN_OPTS - parameters passed to the Java VM when running Maven
#     e.g. to debug Maven itself, use
#       set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000
#   MAVEN_SKIP_RC - flag to disable loading of mavenrc files
# ----------------------------------------------------------------------------

if [ -z "$MAVEN_SKIP_RC" ] ; then

  if [ -f /usr/local/etc/mavenrc ] ; then
    . /usr/local/etc/mavenrc
  fi

  if [ -f /etc/mavenrc ] ; then
    . /etc/mavenrc
  fi

  if [ -f "$HOME/.mavenrc" ] ; then
    . "$HOME/.mavenrc"
  fi

fi

# OS specific support.  $var _must_ be set to either true or false.
cygwin=false;
darwin=false;
mingw=false
case "`uname`" in
  CYGWIN*) cygwin=true ;;
  MINGW*) mingw=true;;
  Darwin*) darwin=true
    # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home
    # See https://developer.apple.com/library/mac/qa/qa1170/_index.html
    if [ -z "$JAVA_HOME" ]; then
      if [ -x "/usr/libexec/java_home" ]; then
        export JAVA_HOME="`/usr/libexec/java_home`"
      else
        export JAVA_HOME="/Library/Java/Home"
      fi
    fi
    ;;
esac

if [ -z "$JAVA_HOME" ] ; then
  if [ -r /etc/gentoo-release ] ; then
    JAVA_HOME=`java-config --jre-home`
  fi
fi

if [ -z "$M2_HOME" ] ; then
  ## resolve links - $0 may be a link to maven's home
  PRG="$0"

  # need this for relative symlinks
  while [ -h "$PRG" ] ; do
    ls=`ls -ld "$PRG"`
    link=`expr "$ls" : '.*-> \(.*\)$'`
    if expr "$link" : '/.*' > /dev/null; then
      PRG="$link"
    else
      PRG="`dirname "$PRG"`/$link"
    fi
  done

  saveddir=`pwd`

  M2_HOME=`dirname "$PRG"`/..

  # make it fully qualified
  M2_HOME=`cd "$M2_HOME" && pwd`

  cd "$saveddir"
  # echo Using m2 at $M2_HOME
fi

# For Cygwin, ensure paths are in UNIX format before anything is touched
if $cygwin ; then
  [ -n "$M2_HOME" ] &&
    M2_HOME=`cygpath --unix "$M2_HOME"`
  [ -n "$JAVA_HOME" ] &&
    JAVA_HOME=`cygpath --unix "$JAVA_HOME"`
  [ -n "$CLASSPATH" ] &&
    CLASSPATH=`cygpath --path --unix "$CLASSPATH"`
fi

# For Mingw, ensure paths are in UNIX format before anything is touched
if $mingw ; then
  [ -n "$M2_HOME" ] &&
    M2_HOME="`(cd "$M2_HOME"; pwd)`"
  [ -n "$JAVA_HOME" ] &&
    JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`"
fi

if [ -z "$JAVA_HOME" ]; then
  javaExecutable="`which javac`"
  if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then
    # readlink(1) is not available as standard on Solaris 10.
    readLink=`which readlink`
    if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then
      if $darwin ; then
        javaHome="`dirname \"$javaExecutable\"`"
        javaExecutable="`cd \"$javaHome\" && pwd -P`/javac"
      else
        javaExecutable="`readlink -f \"$javaExecutable\"`"
      fi
      javaHome="`dirname \"$javaExecutable\"`"
      javaHome=`expr "$javaHome" : '\(.*\)/bin'`
      JAVA_HOME="$javaHome"
      export JAVA_HOME
    fi
  fi
fi

if [ -z "$JAVACMD" ] ; then
  if [ -n "$JAVA_HOME"  ] ; then
    if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
      # IBM's JDK on AIX uses strange locations for the executables
      JAVACMD="$JAVA_HOME/jre/sh/java"
    else
      JAVACMD="$JAVA_HOME/bin/java"
    fi
  else
    JAVACMD="`\\unset -f command; \\command -v java`"
  fi
fi

if [ ! -x "$JAVACMD" ] ; then
  echo "Error: JAVA_HOME is not defined correctly." >&2
  echo "  We cannot execute $JAVACMD" >&2
  exit 1
fi

if [ -z "$JAVA_HOME" ] ; then
  echo "Warning: JAVA_HOME environment variable is not set."
fi

CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher

# traverses directory structure from process work directory to filesystem root
# first directory with .mvn subdirectory is considered project base directory
find_maven_basedir() {

  if [ -z "$1" ]
  then
    echo "Path not specified to find_maven_basedir"
    return 1
  fi

  basedir="$1"
  wdir="$1"
  while [ "$wdir" != '/' ] ; do
    if [ -d "$wdir"/.mvn ] ; then
      basedir=$wdir
      break
    fi
    # workaround for JBEAP-8937 (on Solaris 10/Sparc)
    if [ -d "${wdir}" ]; then
      wdir=`cd "$wdir/.."; pwd`
    fi
    # end of workaround
  done
  echo "${basedir}"
}

# concatenates all lines of a file
concat_lines() {
  if [ -f "$1" ]; then
    echo "$(tr -s '\n' ' ' < "$1")"
  fi
}

BASE_DIR=`find_maven_basedir "$(pwd)"`
if [ -z "$BASE_DIR" ]; then
  exit 1;
fi

##########################################################################################
# Extension to allow automatically downloading the maven-wrapper.jar from Maven-central
# This allows using the maven wrapper in projects that prohibit checking in binary data.
##########################################################################################
if [ -r "$BASE_DIR/.mvn/wrapper/maven-wrapper.jar" ]; then
    if [ "$MVNW_VERBOSE" = true ]; then
      echo "Found .mvn/wrapper/maven-wrapper.jar"
    fi
else
    if [ "$MVNW_VERBOSE" = true ]; then
      echo "Couldn't find .mvn/wrapper/maven-wrapper.jar, downloading it ..."
    fi
    if [ -n "$MVNW_REPOURL" ]; then
      jarUrl="$MVNW_REPOURL/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar"
    else
      jarUrl="https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar"
    fi
    while IFS="=" read key value; do
      case "$key" in (wrapperUrl) jarUrl="$value"; break ;;
      esac
    done < "$BASE_DIR/.mvn/wrapper/maven-wrapper.properties"
    if [ "$MVNW_VERBOSE" = true ]; then
      echo "Downloading from: $jarUrl"
    fi
    wrapperJarPath="$BASE_DIR/.mvn/wrapper/maven-wrapper.jar"
    if $cygwin; then
      wrapperJarPath=`cygpath --path --windows "$wrapperJarPath"`
    fi

    if command -v wget > /dev/null; then
        if [ "$MVNW_VERBOSE" = true ]; then
          echo "Found wget ... using wget"
        fi
        if [ -z "$MVNW_USERNAME" ] || [ -z "$MVNW_PASSWORD" ]; then
            wget "$jarUrl" -O "$wrapperJarPath" || rm -f "$wrapperJarPath"
        else
            wget --http-user=$MVNW_USERNAME --http-password=$MVNW_PASSWORD "$jarUrl" -O "$wrapperJarPath" || rm -f "$wrapperJarPath"
        fi
    elif command -v curl > /dev/null; then
        if [ "$MVNW_VERBOSE" = true ]; then
          echo "Found curl ... using curl"
        fi
        if [ -z "$MVNW_USERNAME" ] || [ -z "$MVNW_PASSWORD" ]; then
            curl -o "$wrapperJarPath" "$jarUrl" -f
        else
            curl --user $MVNW_USERNAME:$MVNW_PASSWORD -o "$wrapperJarPath" "$jarUrl" -f
        fi

    else
        if [ "$MVNW_VERBOSE" = true ]; then
          echo "Falling back to using Java to download"
        fi
        javaClass="$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.java"
        # For Cygwin, switch paths to Windows format before running javac
        if $cygwin; then
          javaClass=`cygpath --path --windows "$javaClass"`
        fi
        if [ -e "$javaClass" ]; then
            if [ ! -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then
                if [ "$MVNW_VERBOSE" = true ]; then
                  echo " - Compiling MavenWrapperDownloader.java ..."
                fi
                # Compiling the Java class
                ("$JAVA_HOME/bin/javac" "$javaClass")
            fi
            if [ -e "$BASE_DIR/.mvn/wrapper/MavenWrapperDownloader.class" ]; then
                # Running the downloader
                if [ "$MVNW_VERBOSE" = true ]; then
                  echo " - Running MavenWrapperDownloader.java ..."
                fi
                ("$JAVA_HOME/bin/java" -cp .mvn/wrapper MavenWrapperDownloader "$MAVEN_PROJECTBASEDIR")
            fi
        fi
    fi
fi
##########################################################################################
# End of extension
##########################################################################################

export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"}
if [ "$MVNW_VERBOSE" = true ]; then
  echo $MAVEN_PROJECTBASEDIR
fi
MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS"

# For Cygwin, switch paths to Windows format before running java
if $cygwin; then
  [ -n "$M2_HOME" ] &&
    M2_HOME=`cygpath --path --windows "$M2_HOME"`
  [ -n "$JAVA_HOME" ] &&
    JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"`
  [ -n "$CLASSPATH" ] &&
    CLASSPATH=`cygpath --path --windows "$CLASSPATH"`
  [ -n "$MAVEN_PROJECTBASEDIR" ] &&
    MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"`
fi

# Provide a "standardized" way to retrieve the CLI args that will
# work with both Windows and non-Windows executions.
MAVEN_CMD_LINE_ARGS="$MAVEN_CONFIG $@"
export MAVEN_CMD_LINE_ARGS

WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain

exec "$JAVACMD" \
  $MAVEN_OPTS \
  $MAVEN_DEBUG_OPTS \
  -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \
  "-Dmaven.home=${M2_HOME}" \
  "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \
  ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@"
#mnvw.cmd 

@REM ----------------------------------------------------------------------------
@REM Licensed to the Apache Software Foundation (ASF) under one
@REM or more contributor license agreements.  See the NOTICE file
@REM distributed with this work for additional information
@REM regarding copyright ownership.  The ASF licenses this file
@REM to you under the Apache License, Version 2.0 (the
@REM "License"); you may not use this file except in compliance
@REM with the License.  You may obtain a copy of the License at
@REM
@REM    https://www.apache.org/licenses/LICENSE-2.0
@REM
@REM Unless required by applicable law or agreed to in writing,
@REM software distributed under the License is distributed on an
@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@REM KIND, either express or implied.  See the License for the
@REM specific language governing permissions and limitations
@REM under the License.
@REM ----------------------------------------------------------------------------

@REM ----------------------------------------------------------------------------
@REM Maven Start Up Batch script
@REM
@REM Required ENV vars:
@REM JAVA_HOME - location of a JDK home dir
@REM
@REM Optional ENV vars
@REM M2_HOME - location of maven2's installed home dir
@REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands
@REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a keystroke before ending
@REM MAVEN_OPTS - parameters passed to the Java VM when running Maven
@REM     e.g. to debug Maven itself, use
@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000
@REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files
@REM ----------------------------------------------------------------------------

@REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on'
@echo off
@REM set title of command window
title %0
@REM enable echoing by setting MAVEN_BATCH_ECHO to 'on'
@if "%MAVEN_BATCH_ECHO%" == "on"  echo %MAVEN_BATCH_ECHO%

@REM set %HOME% to equivalent of $HOME
if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%")

@REM Execute a user defined script before this one
if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre
@REM check for pre script, once with legacy .bat ending and once with .cmd ending
if exist "%USERPROFILE%\mavenrc_pre.bat" call "%USERPROFILE%\mavenrc_pre.bat" %*
if exist "%USERPROFILE%\mavenrc_pre.cmd" call "%USERPROFILE%\mavenrc_pre.cmd" %*
:skipRcPre

@setlocal

set ERROR_CODE=0

@REM To isolate internal variables from possible post scripts, we use another setlocal
@setlocal

@REM ==== START VALIDATION ====
if not "%JAVA_HOME%" == "" goto OkJHome

echo.
echo Error: JAVA_HOME not found in your environment. >&2
echo Please set the JAVA_HOME variable in your environment to match the >&2
echo location of your Java installation. >&2
echo.
goto error

:OkJHome
if exist "%JAVA_HOME%\bin\java.exe" goto init

echo.
echo Error: JAVA_HOME is set to an invalid directory. >&2
echo JAVA_HOME = "%JAVA_HOME%" >&2
echo Please set the JAVA_HOME variable in your environment to match the >&2
echo location of your Java installation. >&2
echo.
goto error

@REM ==== END VALIDATION ====

:init

@REM Find the project base dir, i.e. the directory that contains the folder ".mvn".
@REM Fallback to current working directory if not found.

set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR%
IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir

set EXEC_DIR=%CD%
set WDIR=%EXEC_DIR%
:findBaseDir
IF EXIST "%WDIR%"\.mvn goto baseDirFound
cd ..
IF "%WDIR%"=="%CD%" goto baseDirNotFound
set WDIR=%CD%
goto findBaseDir

:baseDirFound
set MAVEN_PROJECTBASEDIR=%WDIR%
cd "%EXEC_DIR%"
goto endDetectBaseDir

:baseDirNotFound
set MAVEN_PROJECTBASEDIR=%EXEC_DIR%
cd "%EXEC_DIR%"

:endDetectBaseDir

IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig

@setlocal EnableExtensions EnableDelayedExpansion
for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a
@endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS%

:endReadAdditionalConfig

SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe"
set WRAPPER_JAR="%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.jar"
set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain

set DOWNLOAD_URL="https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar"

FOR /F "usebackq tokens=1,2 delims==" %%A IN ("%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.properties") DO (
    IF "%%A"=="wrapperUrl" SET DOWNLOAD_URL=%%B
)

@REM Extension to allow automatically downloading the maven-wrapper.jar from Maven-central
@REM This allows using the maven wrapper in projects that prohibit checking in binary data.
if exist %WRAPPER_JAR% (
    if "%MVNW_VERBOSE%" == "true" (
        echo Found %WRAPPER_JAR%
    )
) else (
    if not "%MVNW_REPOURL%" == "" (
        SET DOWNLOAD_URL="%MVNW_REPOURL%/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar"
    )
    if "%MVNW_VERBOSE%" == "true" (
        echo Couldn't find %WRAPPER_JAR%, downloading it ...
        echo Downloading from: %DOWNLOAD_URL%
    )

    powershell -Command "&{"^
  "$webclient = new-object System.Net.WebClient;"^
  "if (-not ([string]::IsNullOrEmpty('%MVNW_USERNAME%') -and [string]::IsNullOrEmpty('%MVNW_PASSWORD%'))) {"^
  "$webclient.Credentials = new-object System.Net.NetworkCredential('%MVNW_USERNAME%', '%MVNW_PASSWORD%');"^
  "}"^
  "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; $webclient.DownloadFile('%DOWNLOAD_URL%', '%WRAPPER_JAR%')"^
  "}"
    if "%MVNW_VERBOSE%" == "true" (
        echo Finished downloading %WRAPPER_JAR%
    )
)
@REM End of extension

@REM Provide a "standardized" way to retrieve the CLI args that will
@REM work with both Windows and non-Windows executions.
set MAVEN_CMD_LINE_ARGS=%*

%MAVEN_JAVA_EXE% ^
  %JVM_CONFIG_MAVEN_PROPS% ^
  %MAVEN_OPTS% ^
  %MAVEN_DEBUG_OPTS% ^
  -classpath %WRAPPER_JAR% ^
  "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" ^
  %WRAPPER_LAUNCHER% %MAVEN_CONFIG% %*
if ERRORLEVEL 1 goto error
goto end

:error
set ERROR_CODE=1

:end
@endlocal & set ERROR_CODE=%ERROR_CODE%

if not "%MAVEN_SKIP_RC%"=="" goto skipRcPost
@REM check for post script, once with legacy .bat ending and once with .cmd ending
if exist "%USERPROFILE%\mavenrc_post.bat" call "%USERPROFILE%\mavenrc_post.bat"
if exist "%USERPROFILE%\mavenrc_post.cmd" call "%USERPROFILE%\mavenrc_post.cmd"
:skipRcPost

@REM pause the script if MAVEN_BATCH_PAUSE is set to 'on'
if "%MAVEN_BATCH_PAUSE%"=="on" pause

if "%MAVEN_TERMINATE_CMD%"=="on" exit %ERROR_CODE%

cmd /C exit /B %ERROR_CODE%
#pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.7.0</version>
  <relativePath/> <!-- lookup parent from repository -->
 </parent>
 <groupId>com.javatechie</groupId>
 <artifactId>devops-integration</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <name>devops-integration</name>
 <description>Demo project for Spring Boot</description>
 <properties>
  <java.version>1.8</java.version>
 </properties>
 <dependencies>
  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
  </dependency>

  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-test</artifactId>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
 </dependencies>

 <build>
  <plugins>

   <plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
   </plugin>

   <plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>
     <execution>
      <goals>
       <goal>build-info</goal>
      </goals>
     </execution>
    </executions>
   </plugin>
   <plugin>
    <groupId>pl.project13.maven</groupId>
    <artifactId>git-commit-id-plugin</artifactId>
   </plugin>
  </plugins>
  <finalName>devops-integration</finalName>
 </build>

</project>

Create Project Configuration in Jenkins:

  • Configure a project in Jenkins that links to your Git repository and integrates with SonarQube using the token.
  • Define the retention period for build artifacts in Jenkins.
Jenkins Project Configuration step 1
Jenkins Project Configuration step 2
Create pipeline syntux for SonarQube token
Use given syntax in Jenkins Pipeline to authenticate jenkins with SonarQube

Jenkinsfile:

#Jenkinsfile

pipeline{
    
    agent any 

    environment {
        AWS_ACCOUNT_ID="983291512478"
        AWS_DEFAULT_REGION="ap-southeast-1"
        IMAGE_REPO_NAME="jenkins-pipeline"
        VERSION = "${env.BUILD_ID}"
        REPOSITORY_URI = "983291512478.dkr.ecr.ap-southeast-1.amazonaws.com/jenkins-pipeline"
    }
    
    stages {
        
        stage('Git Checkout'){
            
            steps{
                
                script{
                    
                    git branch: 'main', url: 'https://github.com/saad946/Jenkins-SonarQube-FluxCD-CICD.git'
                }
            }
        }
        stage('UNIT testing'){
            
            steps{
                
                script{
                    
                    sh 'mvn test'
                }
            }
        }
        stage('Integration testing'){
            
            steps{
                
                script{
                    
                    sh 'mvn verify -DskipUnitTests'
                }
            }
        }
        stage('Maven build'){
            
            steps{
                
                script{
                    
                    sh 'mvn clean install'
                }
            }
        }
        stage('Static code analysis'){
            
            steps{
                
                script{
                    
                    withSonarQubeEnv(credentialsId: 'sonars-tokens') {
                        
                        sh 'mvn clean package sonar:sonar'
                    }
                   }
                    
                }
            }
            stage('Quality Gate Status'){
                
                steps{
                    
                    script{
                        
                        waitForQualityGate abortPipeline: false, credentialsId: 'sonars-tokens'
                    }
                }
            }
   
        // Uploading Docker images into AWS ECR
            stage('Pushing to ECR') {
                steps{  
                    script {
                        sh "aws ecr get-login-password --region ap-southeast-1 | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
                        sh "docker build -t ${IMAGE_REPO_NAME}:${VERSION} ."
                        sh "docker tag ${IMAGE_REPO_NAME}:$VERSION ${REPOSITORY_URI}:${VERSION}"
                        sh "docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}:${VERSION}"
                        sh "docker rmi ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}:${VERSION}"
         }
        }
      }
   }

        
}
Jenkins CI Pipeline before docker part

Step-7 :Docker Image Creation and push to AWS ECR

Once your code passes SonarQube testing, build a Docker image and push it to the Elastic Container Registry (ECR):

Configure Jenkins to Build a Docker Image:

  • Instruct Jenkins to create a Docker image from your application code.

Push the Docker Image to ECR:

  • Store the Docker image in the ECR repository for deployment.

Remove the Docker Image from the Jenkins Server:

  • Safely remove the Docker image from the Jenkins server to save space and resources.
#Dockerfile 
FROM maven as build
WORKDIR /app
COPY . .
RUN mvn install

FROM openjdk:11.0
WORKDIR /app
COPY --from=build /app/target/devops-integration.jar /app/
EXPOSE 8080
CMD ["java","-jar","devops-integration.jar"]
ECR Registry

Step-8 :GitHub Actions CI with Datree

To ensure code quality and adherence to best practices, integrate Datree into your GitHub Actions workflow:

Integrate Datree into GitHub Actions Workflow:

  • Include Datree as a step in your GitHub Actions workflow.
  • Configure Datree to scan your codebase for issues related to Kubernetes Helm charts, YAML files, and other configuration files.

Offline Mode for Datree:

  • Utilize Datree in offline mode to check for compliance with best practices even without an internet connection.
  • This ensures your CI/CD pipeline remains robust and secure in environments with limited connectivity.
#.github/workflows/main.yaml

# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the action will run. 
on:
  # Triggers the workflow on push or pull request events but only for the master branch
  push:
    branches: [ main ]
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      # Runs a single command using the runners shell
      - name: Run a one-line script
        run: echo Hello, world!

      # Runs a set of commands using the runners shell
      - name: Run a multi-line script
        run: |
          echo Add other actions to build,
          echo test, and deploy your project
      - name: Checkout
        uses: actions/checkout@v2
      - name: Install Helm-Datree
        run: |
          helm plugin install https://github.com/datreeio/helm-datree
          helm plugin update datree
          helm datree version
          helm datree config set offline local
      - name: Run Datree Tests
        run: 
          
          helm datree test kubernetes/myapp/
Datree with GitHub Actions for Helm charts tracking

By integrating Datree into your CI/CD pipeline, you ensure that your Kubernetes configurations and Helm charts are in compliance with best practices, enhancing the quality and security of your deployments.

Step-9 :ArgoCD Deployment to EKS Cluster

With Datree integrated into your GitHub Actions workflow, you can proceed with the deployment using ArgoCD as previously outlined:

Apply the ArgoCD Helm Chart with Custom Values:

  • Utilize custom Helm chart values tailored to your deployment requirements.

Configure ArgoCD Application Sync:

  • Specify the Git repository and branch that ArgoCD should monitor for changes.
  • Configure the ArgoCD application to automatically sync whenever changes are detected.
# in EKS cluster,apply ArgoCD helm chart
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd -n argocd --create-namespace argo/argo-cd --version 3.35.4 -f argocd.yaml
#argo-values.yaml
---
global:
  image:
    tag: "v2.6.7"

server:
  extraArgs:
    - --insecure
# Apply ArgoCD application and secret (github PAT) in EKS Cluster, so ArgoCD 
  can sync with GitHub repo.

#application.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/saad946/Jenkins-SonarQube-FluxCD-CICD.git
    targetRevision: main
    path: kubernetes/myapp
   
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - Validate=true
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true


#repo-secret.yaml

---
apiVersion: v1
kind: Secret
metadata:
  name: github-sync
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: https://github.com/saad946/Jenkins-SonarQube-FluxCD-CICD.git
  sshPrivateKey: |
    
    -----BEGIN OPENSSH PRIVATE KEY-----
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
    QyNTUxOQAAACChdeffghjkoiytee3+xqfyz/nu8GvE/s/TBVOdduqRNQAAAKDueOG17njh
    tQAAAAtzc2gtZWQ4ffswrtyuioolnbvcd3+xqfyz/nu8GvEyRj/9Th9p/s/TBVOdduqRNQ
    AAAEDVoY65Z+DDYiAsdWcRCjySz1k+hShW3dw4txkXFLi3a6Hwxjxrf7Gp/LP+e7wa8TJG
    P/1OH2n+z9MFU5126pE1AAAAFnNhYWR1bGxhaDk0NkBnbWFpbC5jb20BAgMEBQYH
    -----END OPENSSH PRIVATE KEY-----
  insecure: "false"
  enableLfs: "false"
#Install ArgoCD CLI in cluster and use these commands to interact with argoCD.

argocd port forward svc/argocd-server -n argocd 8080 &
# get argocd secret from repo-server named secret and decode it to use in 
# follwoing step
argocd login localhost:8080
user: admin
password: from previous step

argocd app list
argocd app get posting-service
argocd app sync posting-service

By combining Datree’s code analysis capabilities with ArgoCD’s automated deployment, you ensure that only high-quality, compliant Kubernetes configurations and Helm charts are deployed to your Elastic Kubernetes Service (EKS) cluster, maintaining a secure and reliable production environment.

Conclusion

With a comprehensive CI/CD pipeline that encompasses Terraform, Packer, Jenkins, SonarQube, Docker, Datree, and ArgoCD, you have not only automated the building, testing, and deployment of your software but also enforced best practices for Kubernetes configurations and Helm charts. This approach significantly enhances code quality, security, and compliance throughout your development and deployment process, ultimately leading to more reliable and efficient software delivery.

References:

GitHub Repo: https://github.com/saad946/Jenkins-SonarQube-FluxCD-CICD

Terraform Documentation: https://www.terraform.io/docs/index.html

Packer Documentation: https://www.packer.io/docs

Jenkins Documentation: https://www.jenkins.io/doc/

SonarQube Documentation: https://docs.sonarqube.org/latest/

Docker Documentation: https://docs.docker.com/

Datree Documentation: https://docs.datree.io/

ArgoCD Documentation: https://argoproj.github.io/argo-cd/

Jenkins
Sonarqube
Argo Cd
Kubernetes
Terraform

--

--

Saad Ullah Khan Warsi

Written by Saad Ullah Khan Warsi

52 Followers

Kubernetes Certified Developer CKAD | AWS Certified Solutions Architect | Terraform Certified Associate

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams