DevSecOps in OpenShift with SonarQube and Custom Scanning images

Dale Bingham

You have OpenShift (or Minishift locally) running to get up to speed with containers, kubernetes, and OpenShift as it pertains to modern software development. You deployed SonarQube and some projects (like this). You have your SonarQube project and project key setup to match your project deploy for scanning. Now you want to up your game with the “Sec” part of DevSecOps and at least scan your local images to see what is wrong with them. That is what this blog post is about. Don’t have OpenShift and SonarQube setup? Read this blog and follow along to quickly get up and running. Do not have any projects setup to scan? I have one for you here and its corresponding blog. Then come back here and get rolling. This has a few steps to fully set it up however, I want you to understand what goes into a good starting pipeline with security built in from the start.


First things First, get Jenkins prepped with SonarQube

To do our DevSecOps thing here we will need a Jenkinsfile to run through. I am going to use my PeopleAPI project and its Jenkinsfile as my example here however any project that you have in Minishift will work really. To get going first I am going to log into Jenkins in my OpenShift project and then click Manage Jenkins → Manage Plugins. (I really need to do a custom Jenkins setup but will have to wait for that.) There is a plugin that I need you to add under the “Available” tab. The “SonarQube Scanner” plugin is the one you definitely need to install. You can install others if you wish. With it checked/selected click the “Download and Install after Reboot” button on Jenkins and wait for it to download and install. It should not take too long however you could go get some coffee or see what the score of the last Manchester United game was. Click the “Restart Jenkins” checkbox on the Installing Plugins page and when it reboots verify those two plugins are listed in the same area under “Installed” plugins.

Now go to the Manage Jenkins → Configure System and scroll down to the SonarQube section. We need three things here: a name, a key, and the URL to SonarQube. I named mine “SonarQube-Server” to keep it simple and because I do not like spaces in names. The URL you can use to keep this within OpenShift is http://sonarqube.sonarqube.svc:9000 based on the service information.

SonarQube Internal URL for our Jenkinsfile

To get the Token you need from SonarQube click the link in your SonarQube project that is the Route for the SonarQube application like https://sonarqube-sonarqube.x.x.x.. Log in with admin/admin (just for this purpose, you would NOT use an Admin token for this) and click your far right icon for your account dropdown. Click the My Account link on the menu and then the Security tab (see the image below). Enter a name in the Generate Tokens field that will match your project (I used peopleapi to match the project name) and click the Generate button. You will get a token that you must copy right now for use in the SonarQube Servers plugin area of Jenkins. Copy that token because once you leave this page it goes away and you have to regenerate.

Generating a token in SonarQube for scanning

Enter the Name, URL, and the Token you just generated and your screen should look something like this below. Make sure the “Enable injection…” checkbox is checked as we use those environment variables in our Jenkinsfile later. Click the Apply button to save this.

SonarQube Setup in Jenkins Configure System area

Load the Jenkins Slave images for SonarQube

To have your Jenkinsfile spawn off a SonarQube scan within OpenShift, I made a custom Jenkins Slave images based off the CentOS7 Jenkins Slave Base in DockerHub. The one we need for this is the .NET Core global tool one liked. This adds the .NET Core 2.1 SDK to a Jenkins Slave as well as the sonarscanner global tool to perform a SonarQube scan easily on .NET Core projects. To quickly make this, use the image-stream-template.yaml file linked and copy/paste that into the “openshift” project in OpenShift (logging in with admin/admin) via the Import YAML/JSON button. This will create an image stream definition and a build definition. Go into the Builds → Builds in the OpenShift project in OpenShift and build the “jenkins-slave-sonarqube-dotnet” configuration. When complete it should make a new image under the Builds → Images area in the OpenShift project similar to 172.30.1.1:5000/openshift/jenkins-slave-sonarqube-dotnet. We will use this image in a later step.

Custom image for Jenkins and SonarQube Scanner for .NET Core

This lets your Jenkinsfile spawn off pods to do sections of your Jenkins pipeline within your OpenShift project and then the containers die off when they are done. This is NOT JenkinsX. It is just Jenkins in OpenShift. So now you have Jenkins, you have the two additional plugins, and you have setup your access tokens. (The GH repo for this Jenkins Slave image has other good templates. Feel free to peruse.)

Setup Jenkins to use the Jenkins Slave Images

The Jenkinsfile has things listed in there like agent { label ‘name’ } where name are things like ‘sonar-dotnet’, ‘base’, ‘dotnetcore21’, and the like. These are labels of Kubernetes pods in Jenkins under Manage Jenkins → Configure System. It is a little daunting until you understand WTF it means. Scroll down to where it says “Kubernetes Pod Template” and you will have all of these we need in here. This pod template corresponds to the ‘label’ part of the Jenkinsfile. We need to match the name, label, container name, and docker image of each Pod Template to the labels in our Jenkinsfile: dotnetcore, base, sonar-dotnet. (The sonar-dotnet one is the one we build above with the sonarscanner global tool!) So we will use 3 of these Kubernetes Pod Templates. Thankfully in my PeopleAPI project I added these as ConfigMaps so they come up automatically!

ConfigMaps for Jenkins Slave images in OpenShift used by our Jenkinsfile

If you do this by hand the developer experience (DX) here sucks I know. Trust me! That is why I did ConfigMaps in my PeopleAPI .NET project I linked above and you can find in my GH repo. I have them here as well so we can reuse over and over. Feel free to use, reuse, adjust for your own needs.

Important Note: if you have never setup the access.registry login/pwd for using Red Hat images within OpenShift you need to setup a secret in OpenShift to pull from the RH Registry as it requires a Red Hat Developer account. We use the .NET Core Jenkins Slaves from RH in this. Perform the below commands inside the “openshift” namespace (oc project openshift) specifically after you login with your OC command from the Username → Copy Login Command in the top right corner of the OpenShift Web Console. Replace the “user-name” and “password” in the below commands with your Red Hat Developer login and password.

oc project openshift

oc create secret docker-registry redhat-registry \
--docker-server=registry.redhat.io \
--docker-username=<user-name> \
--docker-password=<password> \
--docker-email=unused

oc secrets link default redhat-registry --for=pull

oc secrets link builder redhat-registry

oc create -f https://raw.githubusercontent.com/redhat-developer/s2i-dotnetcore/master/dotnet_imagestreams.json

oc replace -f https://raw.githubusercontent.com/redhat-developer/s2i-dotnetcore/master/dotnet_imagestreams.json

Update your Jenkinsfile to point to SonarQube

The Jenkinsfile in my PeopleAPI project is in the develop branch of the repo. That is what the PeopleAPI deployment YAML points to. That setup in there is what runs the Jenkins slave images, does the build, does the deploy, and scans the codebase. Make sure the withSonarQubeEnv(‘SonarQube-Server’) area of the Jenkinsfile has the correct SonarQube Server setup from the Jenkins plugin steps above. If you followed along with the sample PeopleAPI project you should be all set.

Also you need to run this command oc adm policy add-role-to-user edit system:serviceaccount:peopleapi:default logged in with the “oc” command line interface so that the Jenkinsfile can run and make the builds and such. You can do that again by going to the top of your Minishift window, clicking your login, and clicking the menu “Copy Login Command”. You need to be the admin/admin user I spoke of earlier to do this. Otherwise your build will hang at the “Build Image” step indefinitely (so I learned!).

Optional: Add the language profile packs to SonarQube

In SonarQube as the admin user, go to the Administrator → Marketplace screen. I always make sure I have these languages loaded to use: SonarC#, SonarGo, SonarJava, SonarJS, SonarPHP, SonarPython. You at least need the language pack for whatever project you are going to scan. You can use the search bar under SonarQube Administration → Marketplace to find them quickly. If they are not there for your languages you wish to scan, add them and then click the Restart button at the top of the page to load them up. Then click Restart again to confirm and your SonarQube is all set for this and other projects in your OpenShift platform or even outside it if you wish!

SonarQube language plugins

Run the Pipeline and see what happens!

Go to the Builds → Pipelines screen in your Minishift project and click the Start Pipeline button. This will engage the Jenkins application in your project and run the Pipeline. You can click the “View Logs” to open the logs in Jenkins itself. You also should see the steps of your pipeline start to take shape and show blue for running, green for successfully finished, and red for OH CRAP!

Jenkinsfile Pipeline running in Minishift

While the pipeline is running you can follow the Logs, you can click the Applications → Pods in the OpenShift project and see the container images for the Jenkins slaves coming up and down. You also can follow along on the Pipeline screen for the pieces to take shape.

Pods running during the Pipeline execution

There are more ways to add Jenkins plugins, trigger builds with webhooks, and a host of other things to automate and further secure your software process. I hope this gave you a glimpse into starting that and getting rolling. Happy coding!

Advanced: Setup SonarQube automatically with the correct plugins

Want a challenge? Setup a Dockerfile in a public GH repo you can use to point to. And then similar to the templates for the SonarQube Jenkins slave images, have a YAML to points to the Dockerfile to create a custom SonarQube image that you can deploy. The plugins have to be copied into the /opt/sonarqube/extensions/ directory when you create the image. I did this for my one project and it saves you a lot of time, helps on configuration management, and automates as much as possible!

Dale Bingham

Written by

CTO of Cingulara. Software Geek by trade. Father of three daughters. Husband. Lover of newer tech where it fits. Follow at https://www.cingulara.com/ @cingulara

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade