Taming CI pipeline with Jenkins, Kallithea, Nexus and more.
Repetition — that is the actuality and the earnestness of existence.
-Søren Kierkegaard
Scrum is an iterative way of software development. We’ve heard a lot of buzz regarding configuration management, continuous integration and other “automate all the things” approach, but what will it take to implement them today? Latest approach from class leader CI software Jenkins is to use Jenkinsfile pipeline, a groovy based domain-specific language, to fulfill the most complicated scenarios out there.
The task was to develop fully operational model for three tier application, which uses MariaDB as a backend, springbooted Java application as a middleware and static, client side Reactjs application as a frontend. The only customer requirement was to use mercurial, as VCS, so all other parts we was free to decide on our own:
The logic was decided, as follows:
- each service, including ci configuration, has each own repository;
- both frontend and middleware has their own pipeline, which will be triggered on each commit, also;
- during each build we will analyze version (in pom.xml for java or in package.json for javascript);
- depending on that we will push artifacts to proper repositories and servers.
Now, when the logic design is complete we are ready to start building our Jenkins files. They are pretty much the same, so I’ll be focused on frontend pipeline just for sake of briefness.
So first things first, we need to decide on notification. The only way (for now) to get notification from the pipeline is wrap your stages in try…catch…finally blocks and write your own e-mail body generator. I’ve decided to go an easy way and borrowed most of the code from Liam Newman’s blog , only added an additional function to export changeset authors and hashes (this requires Email-ext plugin to be installed):
node {
try {
notifyBuild('STARTED')
/* ... existing build steps ... */
} catch (e) {
// If there was an exception thrown, the build failed
currentBuild.result = "FAILED"
throw e
} finally {
// Success or failure, always send notifications
notifyBuild(currentBuild.result)
}
}
def notifyBuild(String buildStatus = 'STARTED') {
// build status of null means successful
buildStatus = buildStatus ?: 'SUCCESSFUL'
// Default values
def changes = getChangeString()
def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
def summary = "${subject} (${env.BUILD_URL})"
def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
<p>Check console output at "<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>"</p>
${changes}"""
emailext (
subject: subject,
body: details,
recipientProviders: [[$class: 'DevelopersRecipientProvider']]
)
}
def getChangeString() {
MAX_MSG_LEN = 100
def changeString = ""
echo "Gathering SCM changes"
def changeLogSets = currentBuild.rawBuild.changeSets
for (int i = 0; i < changeLogSets.size(); i++) {
def entries = changeLogSets[i].items
for (int j = 0; j < entries.length; j++) {
def entry = entries[j]
truncated_msg = entry.msg.take(MAX_MSG_LEN)
changeString += "<p> - ${truncated_msg} [${entry.author}]\n</p>"
}
}
if (!changeString) {
changeString = " - No new changes"
}
return changeString
}Now we are ready to build and deploy our application. Let’s divide our process into three stages: Preparation, Build, and Deploy. To be able to implement all those steps we need another couple of plugins to be installed, and they are Pipeline Maven Plugin, Ansible Plugin, Config File Provider Plugin, NodeJS Plugin.
In Preparation step we will checkout clear workspace from previous build and checkout current version of CVS, also we will call our notification method:
stage('Preparation') {
notifyBuild('STARTED')
deleteDir()
checkout changelog: true, poll: true, scm: [$class: 'MercurialSCM', browser: [$class: 'Kallithea', url: 'http://192.168.240.250:5000/corero-frontend/'], credentialsId: '74a096d1-9650-481c-88ff-cd7f4e99ad9f', source: 'http://jenkins@kallithea-server:5000/frontend', revision: 'ddefault']
}Pretty easy huh? By the way credentialsId field is presented by Credentials Plugin, it is accessible from Jenkins front page:
Let’s get back to our pipeline again. For the next step we will use NodeJS Plugin to install node and npm to our workplace, install javascript dependencies with npm install, build our artifacts with npm run build, than we will pack them in zip archive and push to maven repository. By default maven requires pom.xml to be present in order to place artifact properly, but it also supports automatic pom generation via parameters. We will go the second way, but there are some obstacles to overtake.
As we need to extract NAME and VERSION of the app from package.json file. Things appears to be easy with groovy.json.JsonSlurper, but the class is non serializable, which is restricted in Jenkinsfile. To omit this lets write a methods with “@NonCPS” annotation, which will return serializable string class to our main method:
import groovy.json.JsonSlurper
import java.util.Map@NonCPS
def getBuildName() {
def buildText = new File("${env.WORKSPACE}/package.json")
def buildJson = new JsonSlurper().parseText(buildText.text)
Map jsonResult = (Map) buildJson
getBuildName = jsonResult.get("name")
}@NonCPS
def getBuildVersion() {
def buildText = new File("${env.WORKSPACE}/package.json")
def buildJson = new JsonSlurper().parseText(buildText.text)
Map jsonResult = (Map) buildJson
getBuildVersion = jsonResult.get("version")
}
I also will be using env class in order to store NAME and VERSION, as it is consistent throughout all the stages
stage('Build') {
def nodeHome = tool name: 'NodeJS6.9.0', type: 'jenkins.plugins.nodejs.tools.NodeJSInstallation'
env.PATH = "${nodeHome}/bin:${env.PATH}"
sh "npm install"
sh "npm run build"
env.PACKAGE_VERSION = getBuildVersion()
sh "echo ${env.PACKAGE_VERSION}"
env.PACKAGE_NAME = getBuildName()
withMaven(jdk: 'JDK', maven: 'Maven3', mavenLocalRepo: '.repository', mavenOpts: '', mavenSettingsConfig: 'bf5ddece-edbb-465b-a1d2-8143d0351831', mavenSettingsFilePath: '') {
sh "cd ./build && zip -r ${env.PACKAGE_NAME}-${env.PACKAGE_VERSION}.zip * && mvn -X deploy:deploy-file -Durl=http://NEXUS:8082/nexus/content/repositories/snapshot/ -DrepositoryId=password -DgroupId=com.corero -DartifactId=${env.PACKAGE_NAME} -Dversion=${env.PACKAGE_VERSION} -Dpackaging=zip -Dfile=${env.PACKAGE_NAME}-${env.PACKAGE_VERSION}.zip"
}
}mavenSettingsConfig — provided through Config File Provider Plugin, -DrepositoryId should be declared in content according to Maven specs for settings.xml file:
Now, let’s move on to the last “Deploy” step. We will clear the jenkins’s workspace, download latest version of the package from NEXUS repository, using its API and deploy it via prepared ansible playbook.
For the playbook, I will use geerlingguy.apache role from ansible-galaxy and add additional post tasks in order to upload artifacts and restart the server:
post_tasks:
- name: Copy Frontexd app
copy: src={{ build_path }} dest=/var/www/vhosts/rx owner=apache group=apache mode=0755
- name : Restart apache
service: name=httpd state=restartedJenkins stage step itself looks as follows:
stage('Deploy') {
sh "wget -O rx-latest.zip \"192.168.240.249:8082/nexus/service/local/artifact/maven/content?g=com.corero&a=${env.PACKAGE_NAME}&v=LATEST&r=coreroSnapshot&p=zip\""
sh "unzip rx-latest.zip -d ./build"
ansiblePlaybook credentialsId: 'sshroot', installation: 'ansible 2.2.0.0', inventory: "${env.WORKSPACE}@script/jenkins/ansible/frontend/hosts", playbook: "${env.WORKSPACE}@script/jenkins/ansible/frontend/webservers.yml", extraVars: [build_path: "${env.WORKSPACE}/build/"], sudoUser: null
}${env.WORKSPACE}@script is a default folder, where Jenkins downloads Configuration CVS (see the first schema).
This is all folks!
P.S. Here in Jenkinsfile we are using lots of classes, that should be added to list of approvals. It can be done through “Manage Jenkins”->” In-process Script Approval”, mine looks like this:
