Open in app

Sign in

Write

Sign in

Fouad Djidjeli
9 min readNov 4, 2019

--

Get started with platform CI/CD Jenkins-Nexus- Gitlab PART 1

In this devops-oriented article, we will see how to install and get started with the platform CI/CD.

First, we start with the Jenkins tool, specifically Library-shared, JenkinsFile, master/slave (PART1).

After, Nexus like repository manager and Gitlab like Git repository hosting service (PART2).

Finally, we will package all in one project Ansible in order to deploy and install on a target server (PART3).

Jenkins

Jenkins is an open source automation tool written in Java with plugins build for Continuous Integration purpose.

Is used to build and test your software projects continuously making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build.

It also allows you to continuously deliver your software by integrating with a large number of testing and deployment technologies.

Installing

In this article, we will install Jenkins on CentOS system, for other environment you can consult https://jenkins-x.io/docs/getting-started/setup/install/

As a Java tool, Jenkins only needs Java 8 version pre-installed.

Install Jenkins through yum:

Add the Jenkins repository to the yum repos, and install it from here.

Install Jenkins (standalone) without internet:

If you have an internet constraint to install Jenkins on a server, you can install it standalone with plugins. Upstream from install Jenkins, prepare war and plugins locally (machine access to internet)

  • download jenkins.war http://mirrors.jenkins.io/war/latest/jenkins.war
  • download all the plugins you need https://updates.jenkins-ci.org/download/plugins/
  • upload jenkins.war and plugins to the target server ( we will use Ansible in this step) create path to install ${ JENKINS_PATH } (mkdir -p /my_path_jenkins_installation)
  • create ${JENKINS_PATH}/plugins (mkdir -p /my_path_jenkins_installation/plugins)
  • copy all plugins to ${JENKINS_PATH}/plugins
  • copy jenkins.war to ${JENKINS_PATH}
  • nohup java -jar ${JENKINS_PATH}/jenkins.war –httpPort=8080 &

…Note: nohup and & for to running jenkins in background

JenkinsFile

The pipeline under Jenkins is a script that defines a build workflow. The pipeline was introduced via a dedicated plugin that you will find here https://wiki.jenkins.io/display/JENKINS/Pipeline+Plugin. Many functions are integrated as:

1- echo “your message”

2- sh “your command”

3- etc …

In addition to the functions, we can use all the power of the Groovy language and especially the DSL (Domain Specific Language). The best known are:

1- node

2- stage (successive stages)

3- etc …

Suggested Plugins

Jenkins as one of the best tools of automation of the sector it is thanks to the flexibility of the plugins. The plugins create the ideal scenario to answer most of your needs, even to all, among the main plugins we find:

Pipelines and Continuous Delivery

  • Pipeline: Orchestrate automation workflows regardless its complexity.
  • Multi-branch Pipeline: create a set of pipeline projects according to detected branch in scm repository.

Source Code Management

This section basically handles the checkout of your project from various Source Control Management (SCM) like Bitbucket, GitHub, GitLab, and even the old CVS. Jenkins suggest Git.

User Management and Security

  • Matrix Authorization Strategy: Enables a deep security level control to the Jenkins administrator of each available task. Allows enabling, disabling, and configuring key features of Jenkins.

Getting Started

The project configuration is a bit complicated at the beginning. There are so many options that you may be able to block, but you do not worry. You only need a few of them to start. Over time, you will gain experience and understand the purpose of each option.

Create first project with multi-branch pipeline

Using the simplest example possible, let’s look at a HelloWorld project from GitHub and print “helloworld”.

JenkinsFile is a file of entry points, inside the stage Test we use a simple command shell << echo “hello world” >> It could also be a local Git server or GitHub.

To get there, there are only two steps to configure:

Source Code Management

This section let you configure the checkout from any SCM. Click the Git option and paste this url on the repository url field. Done! Jenkins will make a simple clone of this project.

src/main/java

target/classes

Jenkinsfile

pom.xml

Build configuration

In this step you choose the build by JenkinsFile mode and Relative location within the checkout of your Pipeline script (Jenkinsfile by default)

Our script path is JenkinsFile in project HelloWorld

Console output

Each build has a specific page showing everything that has happened, how long it has taken and a lot of other information.

Master/Slave

Jenkins master/slave architecture is used for distributed build environments, where the workload of building projects is distributed to multiple agent nodes, and we can use different environments for each build.

Jenkins Master

Your main Jenkins server is the master machine. The tasks performed by the master are:

  • Scheduling build jobs.
  • Dispatching builds to the slaves for the execution.
  • Monitor the slaves.
  • Recording and presenting the build results.
  • Can also execute build jobs directly.

Jenkins Slave

  • It hears requests from the Jenkins Master instance.
  • Slaves can run on a variety of operating systems.
  • The job of a Slave is to do as they are told to, which involves executing build jobs dispatched by the Master.
  • We can configure a project to always run on a particular Slave machine or a particular type of Slave machine, or simply let Jenkins pick the next available Slave.

Setting Up A Jenkins Slave

For this example, I have used a slave machine and master machine in the same machine (local).

  • Go to Jenkins dashboard → Manage Jenkins ->Manage Nodes.
  • Give a name to the node and then click OK. For the first time user, only one option of “Permanent Agent”.
  • Then, a page will open in which we have to specify the configuration of the slave machine. All of them are not required in the configuration:
  • Name: Name of the Slave which should be unique.
  • Description: Description of this slave. It is optional
  • # of executors: The maximum number of concurrent builds that Jenkins may perform on this agent. I have used 1 executor for testing purpose, but a good practice would be the number of CPU cores on the machine.
  • Remote root directory: An agent needs to have a directory dedicated to Jenkins. Specify the path to this directory on the agent. you can also juste specify in JenkinsFile.
  • Launch method: It Controls how Jenkins starts this agent.

→ Launch agent agents via ssh

→ Launch agent agents by connecting it to the master (Jnlp)

In this post i will use Launch agent agents by connecting it to the master via Java Network Launching Protocol (Jnlp)

  • Download agent.jar in your workDir
  • Copy the command line
  • Run command line

How to run Jenkins jobs on multiple slaves with a single job?

For example use case:

I have three different environments each containing a Jenkins Slave,

STAGE1-slave1
STAGE2-slave2
STAGE3-slave3

I don’t want to create three different build jobs for them.

I have three files jars (service, router, front), i want to deploy each jar on a target server. Service on slave 1, router on slave 2, front on salve 3

You will need install NodeLabel Parameter Plugin

You can inspire from this example on GitHub

Shared Library

We will tackle one aspect in this article the Shared Library.

What is a Shared Library?

A shared library is a collection of independent Groovy scripts which you pull into your Jenkinsfile at runtime.

The best part is, the Library can be stored, like everything else, in a Git repository.

You can use the Shared Library in several cases, usually it’s used in case you have Jenkins pipelines to automate your generations, you quickly realize that you have to copy and paste similar code between different pipelines.

In this case you can use Shared Library which can centralize all pipelines in single script file.

  • First you create your Groovy scripts (project Library-shared), and add them into your Git repository.
  • Then, you add your Shared Library into Jenkins from the Manage Jenkins screen.
  • Finally, you pull the Shared Library into your pipeline using this annotation (usually at the top of your Jenkinsfile your projects): @Library('demo-library-shared')

src/org.global/GlobalVars.groovy

test/org/global

vars/myPipeline.groovy

Inside your Library you’ll probably have two types of common code:

Steps:

These are called Global Variables in Jenkins terminology. The custom steps that you want to be available to all your Jenkins pipelines.

For example:

You can define your Jenkinsfile for your projects. To do this, add your code in vars/myPipeline.groovy, then implement a def call function:

import org.global.GlobalVars

def call(String pipe) {

    GlobalVars globalVars = new GlobalVars()

    if (pipe == 'my_pipeline_1') {

        echo "The build specific for my_pipeline_1"
        echo ".............my_pipeline_1 ........."
        pipeline {
            agent {
                node {
                    label "slave1"
                    customWorkspace "${env.WORKSPACE}/${env.JOB_NAME}_${env.BUILD_ID}"
                }
            }
            stages {
                stage('BUILD') {
                    steps {
                        script{

                            def name = globalVars.getName(pipe)
                            echo "build name --->:"+name
                        }
                    }
                }
            }
            
        }

    }
    else if (pipe == 'my_pipeline_2') {
        echo "The build specific for my_pipeline_2"
        echo ".............my_pipeline_2 ........."
        pipeline {
            agent {
                node {
                    label "slave2"
                    customWorkspace "${env.WORKSPACE}/${env.JOB_NAME}_${env.BUILD_ID}"
                }
            }
            stages {
                stage('BUILD') {
                    steps {
                        script{
                            def name = globalVars.getName(pipe)
                            echo "build name --->:"+name
                        }
                    }
                }

            }
        }
    }
    else if (pipe == 'my_pipeline_3') {
        echo "The build specific for my_pipeline_3"
        echo ".............my_pipeline_3 ........."
        pipeline {
            agent {
                node {
                    label "slave3"
                    customWorkspace "${env.WORKSPACE}/${env.JOB_NAME}_${env.BUILD_ID}"
                }
            }
            stages {
                stage('BUILD') {
                    steps {
                        script{
                            def name = globalVars.getName(pipe)
                            echo "build name --->:"+name

                        }
                    }
                }
            }
        }
    }
    else {
        currentBuild.result = 'ABORTED'
        echo('Error  cannot run this build ')
        echo('you must choose name of myPipeline in your JenkinsFile into your project')
        error('Aborting the build .....')
    }
}

Other common code:

This may include auxiliary classes or a common code that you can include in the pipeline steps themselves.

Code like this needs to go in the src/your/package/name directory, and then you can use normal Groovy syntax, e.g:

in each pipeline i call method getName

 package org.global;
class GlobalVars {
    def getName(name) {
        return name
    }
}

You can the import this class into your Jenkinsfile and reference the static variable like GlobalVars.getName():

You can use TestUnit with Junit and mockito to test your src code groovy.

public class GetSlaveTest {    

     @Test
    public void getNameTest() {
        //GIVEN
        String pipe = "my_name_pipeline";
        String expected = "my_name_pipeline" ;
        GlobalVars globalvars = new GlobalVars();
        // WHEN (execute)
        String name = (String) globalvars.getName(pipe);
        // THEN ( verify)
        assert name.equals(expected);
    }
 }

See the example repository GitHub

Configure library-shared in Jenkins

Once you have created your library with custom steps, you must tell Jenkins.

In Jenkins, go to Manage Jenkins → Configure System. Under Global Pipeline Libraries, add a library with the following settings:

  • Name: demo-library-shared
  • Default version: Specify a Git reference (branch or commit SHA), e.g. master
  • Retrieval method: Modern SCM
  • Select the Git type
  • Project repository GitHub

How to use the library in Jenkinsfile?

To use the shared library in a pipeline, you just add @Library(‘your-library-name’) to the top of your pipeline definition, or Jenkinsfile. Then call your step by name. e.g:

For Project 1 → myPipeline_1:

@Library('demo-library-shared')_
stage('Get pipeline') {
    echo 'Choose pipeline '
    myPipeline 'my_pipeline_1'
}

For Project 2 → myPiepline_2:

@Library('demo-library-shared')_
stage('Get pipeline') {
    echo 'Choose pipeline '
    myPipeline 'my_pipeline_2'
}

For Project 3 → myPipeline_3:

@Library('demo-library-shared')_
stage('Get pipeline') {
    echo 'Choose pipeline '
    myPipeline 'my_pipeline_3'
}

NOTE: The underscore ( _ ) is not a typo! You need this underscore if the line immediately after the @Libary annotation is not an import statement.

  • Run the pipeline project 1 above, and the output should look something like this.
Jenkins
Master Slave
Shared Libraries
Groovy
Ansible

--

--

Fouad Djidjeli

Written by Fouad Djidjeli

22 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams