Nov 8, 2022

19 min read

Build an End-to-End DevSecOps CI/CD Pipeline and deploy it to EKS

Goals & Objectives:

Like every time, the development teams encounter huge pressure to build new and better applications faster and more performant than before. With the thought of creating a DevSecOps CI/CD pipeline, the main goal is to minimize risk and prevent new threats from entering the DevOps pipeline.

The main objective is to improve the agile framework by surrounding it with security. In essence, this makes the speed of development and the security of code equivalent in priority. Whereas, before they were opposites, and you could not get both at once.

The good news is the DevSecOps engineers can help you solve this problem by developing a custom DevSecOps solution.

What is DevSecOps:

DevSecOps introduced security earlier in the software development life cycle (SDLC). The goal is to develop more secure applications by making everyone involved in the SDLC responsible for security.

Having a business, tech, and security work together to produce secure products seems like a dream come true. Maybe too good to be true? Let’s investigate more and see if DevSecOps can be the silver bullet we all need in building secure solutions. Among multiple benefits of adding DevSecOps mindest into the DevOps pipeline:

Secure software is developed early on.
Testing early often reduces development costs for remediation.
Change in the group's mindset, now security is a priority.
Long-term confidence and satisfaction.

Requirements

Before you start, I assume you have the following:

Docker.
AWS account.
Gitlab and docker hub accounts.

Pipeline Architecture

The pipeline that we will create is represented by the following diagram:

The developers push their code changes into GitLab.
Jenkins downloads the project in the workspace.
In this step, Jenkins will perform in parallel the compilation of the Maven project (generation of * .class) and the static analysis of the code with the CheckStyle plugin using Jenkins Docker agents.
Running Unit Tests using JUnit.
Running Integration Tests using JUnit.
In this step, we find the generation of documentation and the parallel execution of a set of code analyses using SonarQube, PMD, checkstyle, and FindBugs.
Storing artifacts in the Nexus repository.
Building, scanning the docker image, and pushing it to the docker hub.
Provisioning with Terraform the EKS AWS Kubernetes CLUSTER.
Provisioning with Ansible. That is the preparation of the environment to ensure the installation of JDK and tomcat on our servers.
Deploy the artifact in the staging server which is a Docker container with the Ubuntu OS.
Deploying the artifact in the production server of our EKS on AWS cloud.

Section 1: Build, test and push java dockerized application to Docker registry :

DevSecOps tools encompass the entire software development lifecycle from code reviews and version control to deployment and monitoring. The main goals of DevOps are to make frequent software releases possible, and automate as many tasks and processes as possible.

1. Setting up Jenkins

Jenkins is an open-source automation tool written in Java with plugins built for Continuous Integration purposes. Jenkins is used to building and testing your software projects continuously making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. It also allows you to continuously deliver your software by integrating with a large number of testing and deployment technologies.

With Jenkins, organizations can accelerate the software development process through automation. Jenkins integrates development life-cycle processes of all kinds, including build, document, test, package, stage, deploy, static analysis and much more.

During this project, we will use a Jenkins container created by Docker and devops network :

docker network

line 7: The image we are going to use is jenkinsci / blueocean. Docker and the blue Ocean plugin are installed in this image.

Why do we need the docker in the Jenkins image?

Simply to allow the Jenkins server to build images locally and use docker agents in pipeline for running the stages. For more info Docker-in-Docker for CI

Line # 8: The default user of this image is Jenkins so he does not have the right to execute Docker commands. To solve this problem, we have two solutions:

Launch the container with the root user
Add Jenkins user to the docker group.

If we try to run a docker command as jenkins user, we get the errors shown below:

# 16: Expose the docker host daemon in the Jenkins container so that it can create containers and images in the host machine.

We run this command to create the Jenkins container

Open the browser with this address “docker-host-IP: 8081” (localhost for Linux), and you will get the following page:

In our case you will find the password in “./jenkins-data/secrets/initialAdminPassword” since we created the jenkins-data volume in docker-compose.

Select install suggested plugins:

After installation, we will get the dashboard of Jenkins and we notice the appearance of Blue Ocean.

SCM Stage

In this stage, we will define the Gitlab repository of our project which will be cloned at each launch of the pipeline.

In the Blue Ocean Dashboard, click on create a pipeline, then select the Gitlab and the repository of our project.

Blue Ocean gives us the possibility to create our pipeline from scratch using pipeline-editor or to retrieve the declaration of the pipeline from Jenkinsfile in the repository.

Note: To authenticate to Gitlab, I advise you to use a token instead of an email and password. This link shows how to generate a token in Gitlab.

Compile Stage

In order to verify that Jenkins has created a Maven container for the compilation of the project, we can execute the following command to visualize our running containers in real time:

$ docker stats

We note that a container with a random name ‘practical thompson’ in my case is compiling our project.

Persist the Maven dependencies:

Now if we rerun the pipeline, we note that Jenkins has found the docker image of Maven since it has already been downloaded previously. However, Maven will re-download all the dependencies again. Why ?

Just because the dependencies downloaded by Maven are stored inside the container and will be destroyed at the end of the execution.

The solution is very simple, just cache the dependencies of Maven locally. To do this we create a docker volume with the -v option.

Checkstyle Stage

CheckStyle is a development tool to help programmers write Java code that adheres to a coding standard. It automates the process of checking Java code to spare humans from this boring (but important) task. This makes it ideal for projects that want to enforce a coding standard.

We will use the Maven Checkstyle plugin as follows:

$ mvn checkstyle:checkstyle

After the execution of the command, you can verify that the ‘checkstyle-result.xml’ file has been generated in the target directory:

Integration of Jenkins and Checkstyle:

Installing the Checkstyle plugin from Jenkins:

Given that we are going to use Maven’s Checkstyle plugin, so the docker image to be used by docker agent to generate checkstyle-result.xml is Maven.

#51 : reuseNode a boolean, false by default. If true, run the container on the node specified at the top-level of the Pipeline, in the same workspace, rather than on a new node entirely.

#56–63: Jenkins checkstyle plugin configuration. The most important thing is to specify the path of the checkstyle-result.xml file in the ‘pattern’ argument.

Unit Test Stage

#69–71 :This stage will be executed only on the develop or master branches.You can change them, just it’s my choice to speed up the development time and not run this stage at every feature branch.

#80 : Maven command to run the unit tests.

#82–86 : Always after running tests, the results will be exposed to Jenkins/Blue Ocean Dashboard using the JUnit plugin of Jenkins. UT results can be found under the target/surfire-reports directory with the xml extension.

Integration Tests Stage

#88–91 : This stage will be executed only on the develop or master branches.

#99–101 : Maven command to generate artifact and run the integration tests.

Note: -Dsurefire.skip=true : argument to skip a unit test (not rerun UT)

#102–105 : Always after running tests, the results will be exposed to Jenkins/Blue Ocean Dashboard using the JUnit plugin of Jenkins. The IT results are located under the target/failsafe-reports directory with the XML extension.

#107–108 : save the artifacts and pom file for future uses such as deployment. This is useful if we are in a distributed environment. For example, if Jenkins has a node for deployment and another node for build and testing, then instead of rerunning all steps ( compile-> UT-> IT-> Artifact ) in the deployment node, just recover the artifacts generated by the other node.

#110 : Expose the artifacts to Jenkins/Blue Ocean Dashboard.

Code Quality Analysis stage

In this part of the pipeline, several code analysis stages will be executed in parallel. We also note the generation of the Javadoc documentation.

PMD:

PMD is a tool that detects programming flaws such as “unused variables, empty catch blocks, unnecessary object creation, etc”.

FindBugs:

FindBugs is a tool that allows you to analyze compiled files (bytecode)/ (*.class) to detect bugs such as “identify unused private methods, execution of a method on a null object (NullPointerException),…”

NB :
If the project is uncompiled, FindBugs will report nothing. So you have to compile the project before running FindBugs (mvn clean compile findbugs:findbugs)/(findbugs:gui => graphical interface).

Installation of Findbugs de Jenkins plugin.

JavaDocs

Warning Next Generation plugin:

If you have noticed that when installing the Pmd, Checkstyle, and Findbugs plugins, it is noted that they are obsolete (deprecated) and that they are all integrated with the Warning Next Generation plugin. So no need to install each plugin separately.

SonarQube

SonarQube (formerly Sonar)[1] is an open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs and security vulnerabilities in 20+ programming languages. SonarQube offers reports on duplicated code, coding standards, unit tests, code coverage, comments, bugs, and security vulnerabilities.

Note:
SonarQube uses H2 as the default database

#19: we find in this folder the configuration files of SQ (i.e sonar.properties, etc).
#20: data files (i.e H2 database, Elasticsearch indexes, etc.)
#21: here we find the installed plugins.

To launch the SQ container, type the following command:

$ docker-compose up -d sonarqube

To access to SQ Dashboard, just open the browser with this address ‘docker_host-ip: 9000’.
login: admin and password: admin123

If we try to run the sonar plugin of Maven on our project with the following command:

$ mvn sonar:sonar -Dsonar.host.url=http://192.168.99.100:9000

We get the following error because no code analysis plugin is installed in SQ

To solve this problem, you need to install the Java code analysis plugin in SQ Marketplace

After installation, we restart our server.

Or we can put the plugin directly in the folder ‘/extensions/plugins’.

In order to verify that the plugin has been installed, we should find it in the extensions/plugins folder.

rerun this command:

$ mvn sonar:sonar -Dsonar.host.url=http://192.168.99.100:9000

And finally, our project has been analyzed by SQ

Integration of SonarQube with Jenkins

The first step is to specify the SonarQube address and port in the environment variables.

In the environment section just put the address and port of the SQ server.

For those using Linux, just put localhost as the address. For Windows users put the address of the docker’s host in my case 192.168.99.100.

The following command returns your IP address

$ docker-machine ip
$ 192.168.99.100

#166: to run the sonar plugin of Maven.

Building and scanning Docker images:

After each build of the docker image using the dockerfile, we will be going to automate trivy in this CICD pipeline. Trivy will scan these docker images in every build and publish the report in HTML format so that it’s easy to access the report by developers.

We will be using a separate stage to scan the docker images. Let’s add a stage Trivy Scan just after the build stage(Where docker images is built).

stage('Building and scanning image') {
            steps {
                script {
                    sh """trivy image --format template --template "@html.tpl" --output trivy_report.html IMAGE_NAME
"""
                    
                }
                
            }
        }

Publishing Trivy Scan Reports

After the successful execution of the Trivy Scan stage. The stage created an HTML report of the vulnerabilities present in the docker image and saved it to the file trivy_report.html. Let’s publish the Trivy HTML reports in Jenkins. At the end of the pipeline, script add

post {
        always {
            archiveArtifacts artifacts: "trivy_report.html", fingerprint: true
                
            publishHTML (target: [
                allowMissing: false,
                alwaysLinkToLastBuild: false,
                keepAll: true,
                reportDir: '.',
                reportFiles: 'trivy_report.html',
                reportName: 'Trivy Scan',
                ])
            }
        }

This will publish Trivy HTML reports in Jenkins in each build.

You can open the report for each build and can look for the vulnerabilities in it. So that you start fixing it to make your Docker images more secure.

Nexus 3 Repository

Nexus is a software tool designed to optimize the download and storage of binary files used and produced in software development. It centralizes the management of all the binary artifacts generated and used by the organization to overcome the complexity arising from the diversity of binary artifact types, their position in the overall workflow, and the dependencies between them.

Open the browser with port 8081 to access Nexus UI. login: admin and password: admin123.

Default repositories

maven-snapshots repository:

In our example, we will use the maven2 hosted repository (created by default) for storing our artifacts. Otherwise, you can create one.

Nexus integration with Jenkins

installation of Nexus plugin under Jenkins

Create credentials to connect to the Nexus server

Installation of the pipeline-utility-steps plugin.

This plugin allows us to read and extract information from the pom.xml file. For more info click here and here.
Add the address, port, version, protocol, repository (where we are going to store our artifact), and the id of the credentials (that we already created before) of Nexus in the Jenkinsfile environment block.

Notice : I put as URL nexus: 8081 but I could also put 192.168.99.100 (localhost for Linux) since in our case Jenkins and Nexus are in the same network named “devops”, so the resolution of DNS between the containers will be made automatically by Docker. i.e we can use either @ip, hostname (default service name) or the id of the container to ping/connect to the containers.

Nexus Stage:

#7–8: To retrieve the artifact that is already generated in the “Integration Tests” step and to avoid rerunning the mvn package again in the case where we have several slave nodes in Jenkins. For example, if node_1 is responsible for running Integration Tests and generating the artifact, and node2 is responsible for running “Deploy Artifact”. Instead of rerunning mvn package, use the unstash function to retrieve the artifact

We check that our artifact has been stored in our Nexus repository.

Ansible

Ansible is an open-source software provisioning, configuration management, and application-deployment tool. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows. It includes its own declarative language to describe system configuration.

Staging Server

In our example, we will use as a development server a Docker container based on an image that I created in Dockerhub named ansible-target. Below is the Dockerfile of this image.

#1: ubuntu-sh-server is an ubuntu image with an ssh server installed. For more details ubuntu-ssh-server.

#8–9 : the creation of an “ansible” user and assigned him to the “ansible” group.

#10 : set the password of the ansible user to ansible

#11 : add ansible user in sudoers because Ansible uses sudo to execute commands.

#12 : Ansible is based on Python so you have to install python in ansible_target and by default Ansible checks for the existence of python under /usr/ bin/python. Or python3 is installed by default in Ubuntu under /bin/python3, so you can just create a symbolic link.

The following command will launch our dev server:

In order to connect to this server using ssh, it is necessary to add the public key of Ansible master in Ansible target. to do this there are several methods but I will use the simplest (unsecured).

Execute the following command in Ansible target:

echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGHBsoki/RIm9uMwp+c1LcxHOo46YWYNjypGWpNWlsVB7S+Kibh+73LiPRRxwFRhSCkUwYyi4EEG6cstd8vELA4Mggv5A2uS/siciNcMCmF7Lr28yPfJMt3yX9LjDkHRDz9W28ncaeTLE0vuGphjx8kKG8h+zc5maLEcFwzbMv31ULbd3qCqhK35rgBP/OQT/bww4TikUprgdYX6+wkx5f3QflmaVTsM1jtmeTm8ME+XqWml8Nm8mZlxmzos2Pz84F3ilxrc41eStQk/FXaGaxlLihd8LFoFoqiYO4KlIdszOTd3jq6oMrj6Fy0HSE1gqe6hW+RQqN69mH3SRPDbwX root@7aecbf9c557f" > /home/ansible/.ssh/authorized_keys

Or using the cmd below

ssh-copy-id -i ~/.ssh/id_rsa.pub ansible@ip_machine_ansible_target

Notice: I retrieved the public key from the Docker image Ansible_management, just launch a container and you will find the key in /root/.ssh/id_rsa.pub
You can also use your own keys or regenerate a key pair with the ssh-keygen command.

Check inside your ansible-target that Java has been installed and our application has been deployed on Tomcat.

Deploy to AWS EKS Cluster using Helm charts and Terraform

Amazon Web Services offers a fully managed Kubernetes service called Amazon Elastic Kubernetes Service (Amazon EKS). According to Cloud Native Computing Foundation, Kubernetes is the most widely used container orchestration platform and 63% of Kubernetes workloads run on AWS.

When you want to use Kubernetes it brings its own challenges. You have to deal with the network, cluster security, storage, and scalability challenges. Successful deployment of Kubernetes mostly lies in picking the right environment.

Deploy AWS EKS via a Jenkins job using Terraform. The idea here is to easily deploy EKS to AWS, specifying some settings via pipeline parameters.

eksctl has now come along since I wrote this repo, and that is a simpler way of deploying EKS. Thus I created an eksctl-based deployment. Both the eksctl and this deploy have similar setups, so where there is a duplicate, refer to the eksctl docs.

For each cluster, the deploy creates a vpc, 3 subnets, and some networking infra allowing connection out onto the internet so you can access the cluster remotely.

Use of EC2 instances via node groups

EC2 instances are used as EKS workers via a node group. An autoscaling group is defined so the number of EC2 instances can be scaled up and down using the Cluster Autoscaler.

Changes made to the AWS provider example

Some changes to the AWS provider example:

A lot of the settings have been moved to terraform variables, so we can pass them from Jenkins parameters:

aws_region: you specify the region to deploy to (default eu-west-1).
cluster-name: see below (default demo).
vpc-network: network part of the vpc; you can have different networks for each of your PVC eks clusters (default 10.0.x.x).
vpc-subnets: number of subnets/az’s (default 3).
inst-type: Type of instance to deploy as the worker nodes (default m4.large).
num-workers: Number of workers to deploy (default 3)
The cluster name has been changed from terraform-eks-demo to <your-name>; this means multiple eks instances can be deployed, using different names, from the same Jenkins pipeline.
The security group providing access to the k8s API has been adapted to allow you to pass CIDR addresses to it, so you can customize how it can be accessed.

Jenkins pipeline

The pipeline uses a terraform workspace for each cluster name, so you should be safe deploying multiple clusters via the same Jenkins job. The state is maintained in the Jenkins job workspace.

terraform tool installation:

You need to install the Jenkins terraform plugin, and then define it as a tool in Manage Jenkins->Tools. Check the Jenkinsfile for the version required; for example I setup the tool version as 1.0 for all 1.0.x releases available; just update the minor version used as newer versions become available. The second digit (eg 1.x) is considered functionality change with terraform so best use labels like 1.0,1.1, etc.

IAM roles required

Several roles are required, which is confusing. Thus decided to document these in simple terms.

Since EKS manages the Kubernetes backplane and infrastructure, there are no masters in EKS. When you enter kubectl get nodes you will just see the worker nodes that are either implemented via node groups.

Note that as well as using node groups, you can now use fargate, which also shows up as worker nodes via the kubectl get nodes command.

Required roles:

Cluster service role: this is associated with the cluster (and its creation). This allow the Kubernetes control plane to manage AWS resources on behalf of the cluster. The policy AmazonEKSClusterPolicy has all the required permissions.
The service eks.amazonaws.com needs to be able to assume this role (trust relationship).
We also attach policy AmazonEKSVPCResourceController to the role, to allow security groups for pods.
Node group role: This allows worker nodes to be created for the cluster via an auto scaling group (ASG). The more modern node group replaces the older method of having to create all the resources manually in AWS (ASG, launch configuration, etc). There are three policies that are typically used:
AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryReadOnly

What is HELM?

Helm is the application package manager running atop Kubernetes. It allows describing the application structure through convenient helm-charts and managing it with simple commands.

Why is Helm needed?

It’s a huge shift in the way server-side applications are defined, stored, and managed. Adoption of Helm might well be the key to the mass adoption of microservices, as using this package manager simplifies their management greatly.

Deploy the application as a deployment

We will deploy our application as kubernetes deployment and we will expose it as a LoadBalancer service as described below by the YAML definition:

Monitoring with Prometheus:

Prometheus is an open-source monitoring and alerting metric collector with a time series database. It was taken in by the Cloud Native Computing Foundation (CNCF) as the second hosted project after Kubernetes.

Each server, container, and pod produce logs as it executes various processes, programs, and applications. In Linux machines, they can be found in the /var/log directory. Prometheus fetches these logs and converts them to readable formats (text).

And that is all Folks.
Thanks for reading, hope you learned something new. Do clap and share if you like.

Terraform Tool configurations:

Credentials used in the project:

You can find the source code of the project ( the Spring boot project with “Swagger, Unit Test, Integration Test, etc” and all the Docker images “Jenkins, SonarQube, Nexus, Ansible, etc” necessary for our example) on my Gitlab.

Resources

Terraform docs are here.

AWS docs on EKS are here.

Amazon EBS CSI driver

The Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver allows Amazon Elastic Kubernetes…

docs.aws.amazon.com

Securing Container Image using trivy in CICD Pipeline

Hello Everyone I hope you guys are doing well. In this blog, I am going to explain how we can free our docker images…

vijetareigns.medium.com