Twistlock: Secure Serverless Kubernetes CI/CD at Scale

In this blog, we will demonstrate how running CI/CD at scale can be done easily with Kubernetes and Jenkins, as well as keeping your application secured and your infrastructure scaled to fit the workloads running on K8s, thus in order to maximize security, performance, and cost reduction.

We’ll use Jenkins and the Twistlock Jenkins plugin to perform the essential function of scanning newly built container images in a busy, bursty development environment. Spotinst Ocean will be used to manage the K8s nodes, providing the user with a Serverless Containers experience by fully managing all the scaling activities on the cluster and right-sizing the nodes’ flavors to the running pods on the cluster. Spotinst Ocean also helps to reduce infrastructure costs by up to 80%, by leveraging Spot Instances.

Before we get started, here are some tools and services you should be familiar with:

• Serverless ContainersServerless containers are a new way of managing and scaling container infrastructure, especially in Kubernetes. By collecting information about the cluster resources and state, we can right-size our instances to the most efficient state by size and price in order to run container workloads of the cluster.
• Spotinst OceanSpotinst Ocean is a Serverless Containers Platform that takes away the pains of scaling and managing containers and nodes in a Kubernetes cluster. Spotinst Ocean also operates a Kubernetes Auto-scaler which launches the specific instances for the workload that is running on the cluster. The professional term for this method is “POD driven auto-scaling”, as our auto-scaler scales the cluster according to the most important entity of the cluster — the K8s POD.
• Though the appeal of serverless Kubernetes is undeniable, we still need a plan around securing these workloads. Security threats such as crypto miners, misconfigurations/compliance violations, vulnerable libraries and packages, and exploits pose a huge risk to organizations, as these threats can harm, cripple and terminate business operations.Twistlock Cloud Native Cyber Security
  Twistlock essentially created the container security space about five years ago, and with 35% of the Fortune 100 companies as their customers, it is arguably the leading solution in cloud-native cybersecurity with extensive support to Kubernetes, containers, serverless, and VMs. Though it’s not in the scope of this blog, it is worth mentioning that Twistlock’s solution is comprised of vulnerability scanning, advising customers on cloud-native firewalling, runtime security, and deep forensics to protect everything from containers to serverless functions to VM hosts automatically.

Step-by-step guide

Technical deep dive, step-by-step on how to configure this solution

Prerequisites:

The operations we’ll demonstrate in this blog:

• Install Twistlock console on K8s
• Install Jenkins master on K8s
• Install Twistlock Jenkins plugin
• Creating a pipeline using Twistlock vulnerabilities scanning product
• Fix the dependency which will cause the scan to fail
• Rebuild with the new version of our image

Preparing your Twistlock platform

1. Install Jenkins master on K8s with Twistlock plugin
   The easiest way to install Jenkins on is to usehelm. Just use the following command:

• helm install stable/jenkins --set rbac.create=true --name cicd

1. By default, Jenkins helm chart installs Kubernetes plugin — you can find it here. Kubernetes Jenkins plugin will be used to run CI/CD pipelines as K8s pods.
   You can find more information about Jenkins helm chart here
2. Install Twistlock Jenkins plugin
   Usethis guide to install Twistlock Jenkins plugin
3. Creating a pipeline using Twistlock vulnerabilities scanning product
   Now that we’re all set with our application, let’s go ahead and connect to our Jenkins UI.
   Use these commands to get the endpoint for your Jenkins master:

• export SERVICE_IP=$(kubectl get svc — namespace default cicd-jenkins — template “{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}”)

1. In order to retrieve the Jenkins password, use this command:

• printf $(kubectl get secret --namespace default cicd-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

1. Now we can log in to our Jenkins using the URL and the password we’ve retrieved from our cluster. In Jenkins, click on “New Item” → select a Pipeline project, give it a name and click “OK”. In the next window, scroll down to the “Pipeline” section, and enter the following POD template. In the end, click on “Save”

Pod template Jenkinsfile for running slaves as build pods

Ocean scaling kicks-in

Executing your pipeline

Now we’re ready to initiate our pipeline build. Let’s overview the groovy script of our pipeline.

• The first part is the PodTemplate declaration which defines how our Jenkins slave pod will be configured:
• 2 containers are used
• jenkinsci/jnlp — this will handle all the communication from the Master to the slave pod
• docker — in this pod we will simulate a build process of an image which will be scanned for vulnerabilities in the next steps
• Mapping the docker.sock from the K8s node (host) to the container in order to perform DinD (Docker in Docker) builds
• YAML configuration is done to create resource limits to the slave pod — this is best practice and mandatory for Spotinst Ocean to know how about the K8s cluster and which instance is required to be launched. There is also the configuration of “securityContext” for the pod to use the host Docker group id. You can get your host’s docker group id with the following command: `grep docker /etc/group`
• The second part is our pipeline stages:
• Creating a custom image (out of a regular alpine image) — this is done to simulate the process of a CI build stage
• Scanning the new image using the Twistlock Jenkins plugin.
• Publish the scan result to the Twistlock console for tracking and visibility

By clicking on “Build now” on the pipeline job main dashboard, our pipeline will be triggered.

Executing this pipeline should trigger the following events:

1. Jenkins will launch a Jenkins pod slave in K8s using the Kuberntes Jenkins plugin with the specification we’ve covered above
2. Spotinst Ocean will identify that a pod needs to be scheduled, and that it is in a “pending” state
3. Jenkins slave pod will execute the build
4. Twistlock Jenkins plugin will scan the Docker image for vulnerabilities and publish the results to the Twistlock console
5. Spotinst Ocean will identify that the cluster can be scaled down as the pod that caused the scale-up finished its processing

This is a screenshot of our Ocean cluster just before building our pipeline:

As you can see, the cluster utilization is about 79% and that there are 2 nodes on our cluster.

After initiating the build, we see that a new slave pod has been launched. Spotinst Ocean launched a new instance to fulfill the desired capacity needed by the “pending” pods on the cluster.
This is the cluster state before Ocean scaling up (pending pod on the cluster)

This is the cluster state after Ocean scaled up. As you can see, the average CPU went down as another node joined the cluster. The pending pod goes from “Pending” state to “Running”, and the Jenkins pipeline initiated.

As you can see here, the pipeline built a new image using the pipeline we’ve used above. As soon as the image got built, Twistlock plugin scans the image for security vulnerabilities and compliance. In this case, it found 2 issues:

1. Package “ libjpeg-turbo” exists and was reported on CVE-2018–14498
2. Image created with a root user

On the Twistlock console, we can see that the scanning results were recorded to the console. Under Vulnerabilities → Jenkins Jobs we can click on the right build name/number and we’ll get the outputs from the scanning that was performed in our build.

We can even get information about the Docker image layer that caused the security error:

Finally, after our pod has been terminated, the cluster utilization is back to what it was before our pipeline build was initiated. Ocean identifies this cluster state and issues a scale down event — meaning, it sends a “drain” request to the K8s API-server for the node it finds that is most suitable for scaling down and terminates it as soon as it gets the information that the node is drained.

We can also check the logs to see more information about the scale-down event

In this blog, we’ve demonstrated how we can enhance our container security by scanning container images in the early stages of the CI/CD pipeline, in order to provide the developers feedback in regards to vulnerabilities and compliance issues using Twistlock. We’ve also covered how we can use Spotinst Ocean to run K8s (EKS) nodes without having to manage infrastructure or operate any scaling or right-sizing activity for our cluster. Together, Twistlock and Spotinst allow you to scale rapidly when needed while reducing complexity and minimizing security risks.

Do you have any questions/feedback? feel free to contact me directly @ tsahi.duek@spotinst.com

Originally published at https://spotinst.com on May 15, 2019.