Keptn automates release readiness validation for Austrian Online Banking Software
Raiffeisen Software has been one of the early adopters of Keptn, as the SLO-based Quality Gate capability allows them to automate the production release readiness validation of their online banking software.
To fulfill the requirement of high level of quality, Raiffeisen Software has built a continuous test tool stack allowing them to automate the quality evaluation of new software releases before releasing them to their customers. The following illustration gives an overview of that tool stack, which includes tools such as Jenkins, JMeter, Digital.ai Release and others. Keptn’s Quality Gate capability was added in 2020 to provide automated SLO-based release readiness evaluation based on application performance and infrastructure SLIs (Service Level Indicators) monitored by Dynatrace.
Let’s have a quick look into how Keptn integrates into that process, how it evaluates SLOs and how this data helps the responsible release managers from making their production release decisions.
Step 1: Automated Testing and SLO Scoring through Jenkins
Once a new release candidate comes out of engineering, a Jenkins pipeline automates the task of ensuring that Keptn is properly configured for the upcoming load test. Jenkins then executes the load test which simulates peak traffic conditions against the online banking software. After the test is finished, Jenkins queries Keptn’s SLO Score for the timeframe of the test execution. That score (ranging from 0 to 100) gets translated into a Jenkins pass, warning or fail (as specified in the Keptn Quality Gate), as shown in the following screenshot:
Raiffeisen Software is using the Jenkins Shared Library for Keptn which makes it easy to integrate Keptn into existing Jenkins pipelines. Besides quality gate evaluations the Jenkins Shared Library for Keptn provides additional capabilities such as triggering any automation sequence, uploading resources to the Keptn configuration repository, creating or updating projects or services, and more.
Step 2: Detailed SLO Quality Gate Results in Keptn
Through Jenkins, Keptn gets automatically configured with a list of SLIs (Service Level Indicators) and SLOs (Service Level Objectives) that should be evaluated when Jenkins triggers Keptn’s Quality Gate. The configuration happens through YAML files that include the queries against the data source, as well as the objectives for each SLI. Those files are version controlled and Jenkins makes sure that Keptn always has the latest SLI/SLO before a new release gets evaluated.
Currently at Raiffeisen Software, Keptn is configured to pull out end user response times captured by Dynatrace while the load test is running. Dynatrace has the capability to capture these metrics per test case and Keptn is then able to pull these individual response time metrics and compares them against the objective Raiffeisen Software has, e.g.: login must complete within 100ms.
Besides the SLO comparison against absolute values, such as 100ms, Raiffeisen Software uses the capability of using relative criteria. Relative criteria means that a result from the current evaluation is compared with the baseline across previous evaluations. For Raiffeisen Software this is a timesaving feature as its not mandatory to define thresholds for every SLI as automatic baselining does a great job in detecting regressions.
The following is a screenshot showing the individual SLIs Keptn analysis for every release, how it aggregates up to a total SLO score and how there is cross-linking between the quality gate visualization in the Keptns UI back to Jenkins:
If you want to learn more about the inner workings of SLI/SLO-based quality gates check out these resources such as the Keptn documentation or a YouTube tutorial on Quality Gates in 5 Minutes.
Step 3: Release Decisions and Error Analysis
Every performance test run and analysis makes it to the release managers, where the status of the quality gate reflects the recommendation: GREEN, YELLOW or RED. Here is an example from their Confluence page:
In case something is not GREEN, the detailed information on which SLOs failed are also published to that Confluence page for everyone to see and analyze. Like in the following example, where Keptn detected a degradation in application performance caused by a replacement of hard drives that were significantly slower than those used before:
By bringing the automated SLO based analysis by Keptn not only to Jenkins but also into Confluence, decision makers can better understand why certain releases are good to go — or not good to go — to production!
Helping to improve Keptn
Raiffeisen Software has been operating Keptn on OpenShift for more than a year now. Thanks to their collaboration and active contribution within the Keptn community, Keptn is where it is today and can truly provide value to software organizations like Raiffeisen Software.
Thanks to our close collaboration, we can soon expect new capabilities for managing many SLOs or allowing multi SLI-providers in a single Quality Gate. Both are requirements from Raiffeisen Software, as they want to include more technical and business metrics into their quality gates, which require new SLI filter capabilities that the Keptn team is already working on. Raiffeisen Software also wants to pull metrics from further log and monitoring platforms as part of a single Quality Gate, which is why they are also looking forward to the summer where multi SLI-provider support is scheduled to be released.
