How to Speed Up Jenkins Deployment using Portworx

Photo by Patrick Lindenberg on Unsplash

In software testing environment, deployment is one of the most important things which influence the performance of testing process. Which mean if deployment process especially using Jenkins takes 60–120 minutes in normal circumstances, it is too long and of course wasting time. After waiting for a long time for deployment process it can fail cause of some reasons, and imagine that if 1 squad only has 1 staging server for doing the test, it can be bottleneck in testing process.

For the example, let’s take the minimum average time of deployment process using Jenkins (i.e., 60 minutes) and if you have 5 branches/tasks which you want to test today you must allocate about 300 minutes = 5 hours only for deployment process, and then if you work for 8 hours/day you only have 3 hours to doing the test exclude if the branch have some bugs/problems that you must report back to developer for fixing and need to retest. I think it is not efficient for testing process.

Background on Jenkins architecture

So now i want to explain about the characteristics of Jenkins deployment process in general. “Jenkins architecture revolves around master / slave roles. Standard Jenkins deployment models were based initially on bare-metal servers and VMs, and like many aspects of today’s DevOps environments are now moving to containers.” (https://portworx.com/speed-up-jenkins-builds/)

The conventional use of storage in Jenkins has both pros and cons:

Pros:

• Jenkins has a simple and stateless model.
• Jenkins makes no assumptions regarding initial state.
• Jenkins initiates every build from scratch.
• Jenkins can create n slaves when you need to scale.

Cons:

• Jenkins has relatively poor utilization in scaleout models involving lots of slaves.
• Each slave instance in Jenkins must build its pipeline from scratch, which typically involves a complete repository clone, followed by a complete compilation/build.
• If you don’t build your pipeline from scratch, incremental builds can fail due to subtle changes in starting state.

Here is the picture of Jenkins architecture in general :

Source: https://portworx.com/speed-up-jenkins-builds/

And here is the picture of Architecture of Bukalapak Staging — Jenkins :

Notes: Bukalapak Staging — Jenkins server has 8 executors to doing deployment process.

Where Portworx fits and why

Portworx is storage for containers. Get data services for stateful containers including HA(High Availability), snapshots-backup & encryption. “Portworx provides remarkable benefits for containerized Jenkins models. Following are Portworx-specific benefits for common use cases:” (https://portworx.com/speed-up-jenkins-builds/)

Persistent/Shared Volumes

Source: https://portworx.com/products/introduction/

-When using native Docker, when a container moves, its volume doesn’t move with it.

-When using Portworx managing your persistent containers, no matter where your containers are scheduled, they always have their data.

Faster Incremental Builds

Typically both slaves and their data are ephemeral, with data and state discarded when the slave exits. However, if the slave makes use of two different volumes (“build” and “artifact”), then a slave can exit while preserving its artifact repository, to help accelerate subsequent, or incremental builds.

Monolithic Master

Slaves are optional. Smaller Jenkins environments with fewer resources and requirements may opt to simply run only a single monolithic master. In the conventional model, all data/state on the master will be lost when the master exits. With Portworx, data persists in this model, allowing the master to exit, yet quickly restart when needed. When monolithic masters run in an AWS/Cloud, compute charges are only paid for (for example on “m4.16xlarge”) when needed.

Highly Parallel / Fully Distributed

Without Portworx, the master delegates, and each slave does its own complete build and test cycles. With Portworx, only the master performs the build cycle, followed by multiple test cycles all performed in parallel by multiple slaves. In this most powerful model, the master takes multiple volume snapshots when it completes its build cycle, and then assigns a read/writeable snapshot for each slave to use as its own private volume for their respective test cycles.

Easily scale out test cycles

Without Portworx, each slave would need its own private build volume, which typically performs slowly when doing container-to-container copies. With Portworx, test cycles can easily scale out, by having the master create private read/writeable snapshots and spawning new slaves dynamically on demand.

Here is the picture plan for Jenkins architecture after using Portworx in Bukalapak Staging server:

Proof Pudding

This proof was taken from the research article of Global Solutions Architect Portworx — Jeff Silberman. “Actual performance matters much more than any theoretical discussion. So to demonstrate, we used the Jenkins “Game of Life” demo to compare the conventional way of running Jenkins, versus using a combination of the “Faster Incremental Builds” and “Highly Parallel / Fully Distributed” models, both described above.” (https://portworx.com/speed

-up-jenkins-builds/)

Each “baseline” feature build job include the following steps

1. Create slave build container on ECS cluster.
2. Git clone the project and checkout the particular feature branch.
3. Copy the Library folder contents from Jenkins master or NFS location to the slave container workspace. Requires copying 5–6 GiB files from library folder on Jenkins master to slave containers.
4. Run the build process.
5. Archive the build output artifact back to Jenkins master.

On the other hand, using snapshots for your Jenkins slaves, you can accelerate your builds dramatically.

To speed up Jenkins builds you need to:

1. Create snapshot for the Library folder
2. Create slave build container and use the created snapshot volume as slave Jenkins workspace.
3. Git clone the project and checkout the particular feature branch
4. Run the build process
5. Archive the build output artifact back to the Build artifact shared volume

*A dedicated snapshot volume is created for each feature build process.

*The snapshot volumes will be deleted after build process completed.

*The build artifact PX shared volume is mounted to each slave build agent during the container start

The savings in time is simply astounding:

Source: https://portworx.com/speed-up-jenkins-builds/

-Total time for 10 parallel jobs built with PX snapshots and saved artifacts: 0:07:14

-Total time for 10 parallel jobs built without PX snapshots: 3:29:00

Here is the diagram of “Build time savings using Portworx” :

Source: https://portworx.com/speed-up-jenkins-builds/

This is an article used for research, author hopes this information can be made for improvement in deployment process especially in Bukalapak staging server. So it can reduce the obstacles in testing process.

References:

• https://portworx.com/products/introduction/ (Accessed: 2018–08–12)
• https://portworx.com/speed-up-jenkins-builds/ (Accessed: 2018–08–12)