[Jenkins] Rescheduling failed builds with Naginator plugin
I think everyone in software development knows about Jenkins[1] which is an awesome tool for build automation. At WSO2 we use Jenkins as a build automation tool for all of our projects. As you can see in our Jenkins dashboard [2] there are about 500+ build jobs. Here comes the problem.
Sometimes some builds are failing due to infrastructure related issues like network connectivity problems.
So in such situations do we need to trouble the product development team about the build failure and ask them to take the necessary measures? or do we have an alternative? Luckily we do have an alternative.
There are few jenkins plugins which enable us to automatically reschedule the failed builds. Basically we can configure these plugins to reschedule all the unstable/failed builds or to look for a specific pattern in build output log and reschedule the build. In this post I’ll be discussing about one of the leading build rescheduling plugin called Naginator[3].
Configuring Naginator plugin for a project
After you have successfully installed the Naginator plugin, there will be a “Post-Build” action called “Retry build after failure” on your project’s configuration page. Select that option.
Then you’ll see the following configuration options.
As can be seen from the above image, Naginator provides following options.
- Rerun build for unstable builds as well as failures — This will enable us to automatically reschedule the build for unstable builds
- Delay before retrying build — This defines the delay in which the Naginator should wait before rescheduling the build. Delay can either be a fixed interval (retries every n seconds time)or progressive interval. When we set it to Progressive mode, the delay calculation between retry builds works according to the below formula.
next-delay = previous-delay + increment * number-of-consecutive-failures
Whenever the progressive delay hits the maximum value it will remain at the same value. For example, for increment of 5 seconds and maximum of 50 seconds, the delay between builds will be (5, 15, 30, 50, 50, 50, …) seconds. - Maximum number of successive failed builds — This specifies the maximum number of retries the Naginator plugin should retry to run a failed build.
The above mentioned configurations would be sufficient for retrying all the failed builds. However in our case we had a different problem which is only to retry failed builds due to connectivity problems. In such cases we need to enable the advanced configuration options by ticking the “Only rerun build if regular expression is found in output” option.
So with the above configuration I will look if the failed build output contains the string “Connection failure” and if found Naginator will reschedule that particular build. Since the Naginator plugin is seeking for a regex pattern in the output, we can combine multiple scenarios by concatenating the regex expression using ‘|’ operator which allows us to handle different types of build failure causes.
.*Return code is: 50[23], ReasonPhrase: (Bad Gateway|Service Temporarily Unavailable).*|.*Error (cloning|fetching) remote repo.*References
[1]. https://jenkins.io/
[2]. https://wso2.org/jenkins
[3]. https://wiki.jenkins.io/display/JENKINS/Naginator+Plugin