Part II: Tekton and Gerrit. How to make things voting
This is the second article in the series of publications describing our migration path to Tekton for the EPAM Deliver Platform (EDP). The first part introduces our platform's requirements, baseline architecture, and target state. One of the requirements discussed earlier is to keep the integration of the existing EDP toolset. Today EDP supports three version control systems (VCSs) that Tekton Pipelines work with: Gerrit, GitHub, and GitLab. This article covers the topic of Tekton integration with Gerrit VCS since this is our default Git repository management and code review tool on the platform.
Deployment overview
To better understand Tekton and Gerrit integration, let’s review their configuration and deployment approaches. There are no specific requirements for Tekton deployment; you can follow their official documentation.
We perform Gerrit configuration on three different levels:
- Container level (Dockerfile). EDP runs its own Gerrit container image, which enables additional configuration as a part of the entrypoint.sh script.
- Deployment level (Helm chart). Kubernetes objects are deployed with the chart templates.
- Kubernetes or runtime level (Gerrit operator). The operator supports Custom Resources (CRs) to manage Gerrit components, such as projects, groups, group membership, and permissions.
We plan to switch to the official upstream Gerrit container and Helm chart and consolidate all the configuration steps in our gerrit operator.
We use the Gerrit Webhook plugin to emit events from Gerrit and pass them to Tekton’s EventListner. The plugin is configured on two levels: global and project (with inheritance enabled). To define global values, we put the configuration section in the gerrit.config
file (see the code snippet below). Consult our entrypoint.sh
file for more details.
[plugin "webhooks"]
connectionTimeout = 3000
socketTimeout = 2500
# It is mandatory to set 1 for Tekton EventListener
maxTries = 1
retryInterval = 5000
threadPoolSize = 3
maxTries = 1
is a mandatory value because Tekton'sEventListener
returns a202 ACCEPTED
HTTP response when it can process the request. At the same time, the Gerrit webhook plugin expects one of the following success codes:sc == SC_CREATED || sc == SC_NO_CONTENT || sc == SC_OK
, which are201
,204
,200
. This leads to resending events withmaxTries
count each time.
We use the All-Projects
refs/meta/config
branch and inheritance to populate webhook project-level configuration. See the code snippet below:
cat << EOF > "webhooks.config"
[remote "changemerged"]
url = http://el-gerrit-listener:8080
event = change-merged
[remote "patchsetcreated"]
url = http://el-gerrit-listener:8080
event = patchset-created
EOF
We catch two types of events, changemerged
and patchsetcreated,
for all Gerrit projects and send them to theel-gerrit-listener
Kubernetes Service
on port 8080
. See the conceptual view diagram below.
Network Load Balancer routes external traffic: Ingress — for HTTP, NodePort — for TCP (run Git over SSH). The Gerrit operator performs Gerrit instance configuration. It also creates the gerrit-ci-user-sshkey
secret, which contains the SSH private key for the CI user — EDP CI Bot
. This key is mounted by the Tekton task gerrit-ssh-cmd
and is used for CI voting over SSH.
Work with Gerrit events
This section describes Gerrit's event processing flow with Tekton. We need to configure EventListener
and respective Pipelines
with Tasks
. In EDP, Gerrit EnvetListener
catches two events: changemerged
and patchsetcreated
. To better understand the event payload structure, you can consult official documentation or use the gerrit stream-events
command:
# check that we have access to Gerrit
ssh -p 30004 admin@127.0.0.1
**** Welcome to Gerrit Code Review ****
Hi admin, you have successfully connected over SSH.
Unfortunately, interactive shells are disabled.
To clone a hosted Git repository, use:
git clone ssh://admin@gerrit-example.com:30004/REPOSITORY_NAME.git
Connection to 127.0.0.1 closed.
# Run command to listen for patchsetcreated event
# and then create Patchset in Gerrit either thought UI or command-line
bash-5.1# ssh -p 30004 admin@127.0.0.1 gerrit stream-events -s patchset-created
The JSON below presents a patchset-created event intercepted by the stream-events command:
{
"uploader": {
"name": "Sergiy Kulanov",
"email": "sergiy_kulanov@example.com",
"username": "sergiy_kulanov@example.com"
},
"patchSet": {
"number": 1,
"revision": "81a59313f180e0996b7e59484d50d439b8fb41c8",
"parents": [
"e0a6d6ab4c0d0b70758a94dcb4434162311e8db1"
],
"ref": "refs/changes/21/21/1",
"uploader": {
"name": "Sergiy Kulanov",
"email": "sergiy_kulanov@example.com",
"username": "sergiy_kulanov@example.com"
},
"createdOn": 1672061412,
"author": {
"name": "Sergiy Kulanov",
"email": "sergiy_kulanov@example.com",
"username": "sergiy_kulanov@example.com"
},
"kind": "REWORK",
"sizeInsertions": 9,
"sizeDeletions": 0
},
"change": {
"project": "petclinic",
"branch": "master",
"id": "I18908c3df9229aa6b944dc9d6153e0e5b9a6bf5d",
"number": 21,
"subject": "Demo change",
"owner": {
"name": "Sergiy Kulanov",
"email": "sergiy_kulanov@example.com",
"username": "sergiy_kulanov@example.com"
},
"url": "https://gerrit.example.com/c/petclinic/+/21",
"commitMessage": "Demo change\n\nChange-Id: I18908c3df9229aa6b944dc9d6153e0e5b9a6bf5d\n",
"createdOn": 1672061412,
"status": "NEW",
"wip": true
},
"project": "petclinic",
"refName": "refs/heads/master",
"changeKey": {
"id": "I18908c3df9229aa6b944dc9d6153e0e5b9a6bf5d"
},
"type": "patchset-created",
"eventCreatedOn": 1672061412
}
We use the payload above to configure the EventListener
, define Cel
filters, and consume values in Tekton Pipelines
by extracting them with Tekton’s TriggerBinding
custom resource. See the example of the EDP-specific configuration of EventListener for the changemerged
event:
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: gerrit-listener
spec:
triggers:
- name: gerrit-listener-app-build
interceptors:
# we trigger this only on MERGE events
- ref:
name: "cel"
params:
- name: "filter"
value: "body.change.status in ['MERGED']"
# we run our own EDP interceptor to populate the payload with EDP-specific data
- ref:
name: "edp"
kind: NamespacedInterceptor
# we do some magic on data from edp interceptor
- ref:
name: "cel"
params:
- name: "filter"
value: "extensions.spec.type in ['application']"
- name: "overlays"
value:
- key: cbtype_short
expression: "extensions.spec.type.truncate(3)"
bindings:
# we extract the required data from the payload
- ref: gerrit-binding-build
template:
# and finally, create a PipelineRun instance with proper parameters
ref: gerrit-build-app-template
gerrit-listener-app-build
triggers on MERGE event, then we enrich the original Gerrit event payload with the EDP codebase spec (see the EDP interceptor) and finally do filtering and transformation before instantiating Tekton PipelineRun
.
CI code review
For those who haven’t worked with Gerrit, we recommend getting acquainted with Gerrit workflow. One of the enablers to merging code into the main trunk on EDP is to have several mandatory approvals:
+2 Code-Review
from a team lead or a person with such permissions.+1 Verified
from theCI System
(Code-Review pipeline of Tekton).
To work with Gerrit review (voting), we’ve created a separate Tekton Task
named gerrit-ssh-cmd
. This task mounts the SSH private key of the EDP CI Bot
user created in Gerrit and runs the command passed in the SSH_GERRIT_COMMAND
parameter.
Today, there are two scenarios of how we reference this task from the pipelines:
- Start and Stop of Code-Review pipeline on the
patchsetcreated
event. We reset theVerified
value to zero when the pipeline starts, then put+1
if it succeeds, and-1
if it fails. - Start of Build pipeline on the
changemerged
event. We put the link to the build job.
For the above scenarios, we put the link with the Tekton Dashboard PipelineRun
in the Gerrit UI.
See below the examples of the Code-Review pipeline start section:
- name: gerrit-notify
taskRef:
name: gerrit-ssh-cmd
params:
- name: GERRIT_PORT
value: {{ .Values.global.gerritSSHPort }}
- name: SSH_GERRIT_COMMAND
value: review --verified 0 --message 'Build Started $(params.pipelineUrl)' $(tasks.fetch-repository.results.commit)
workspaces:
- name: ssh-directory
workspace: ssh-creds
And the finally section of the Code-Review pipeline:
finally:
- name: gerrit-vote-success
when:
- input: "$(tasks.status)"
operator: in
values: ["Succeeded"]
taskRef:
name: gerrit-ssh-cmd
params:
- name: GERRIT_PORT
value: {{ .Values.global.gerritSSHPort }}
- name: SSH_GERRIT_COMMAND
value: "review --verified +1 --message 'Build Successful $(params.pipelineUrl)' $(tasks.fetch-repository.results.commit)"
workspaces:
- name: ssh-directory
workspace: ssh-creds
- name: gerrit-vote-failure
when:
- input: "$(tasks.status)"
operator: in
values: ["Failed"]
taskRef:
name: gerrit-ssh-cmd
params:
- name: GERRIT_PORT
value: {{ .Values.global.gerritSSHPort }}
- name: SSH_GERRIT_COMMAND
value: "review --verified -1 --message 'Build Failed $(params.pipelineUrl)' $(tasks.fetch-repository.results.commit)"
workspaces:
- name: ssh-directory
workspace: ssh-creds
The image below is an example of pipeline failure because of a sonar check. The gerrit-vote-failure
task puts Verified -1
for a patch (merge request), and the gerrit-vote-success
is skipped.
As a result, we have -1
from the EDP CI bot
in the Gerrit UI and merge block.
To simplify pipeline management and scale, we’ve implemented CI review primitives (a composition of Tekton tasks) in the separate common-library Helm repository. This approach avoids code repetition and keeps charts DRY. You can concentrate on new logic and re-use gerrit-review-start, gerrit-review-vote, and gerrit-build-start primitives.
This is how you can define the Code-Review pipeline with your-custom-steps
step, using the Helm chart approach and gerrit-review-start/gerrit-review-vote
definitions:
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: gerrit-mycustom-review
labels:
{{- include "edp-tekton.labels" . | nindent 4 }}
spec:
workspaces:
- name: shared-workspace
- name: ssh-creds
params:
- name: pipelineUrl
default: https://tekton-{{ .Release.Namespace }}.{{ .Values.global.dnsWildCard }}/#/namespaces/$(context.pipelineRun.namespace)/pipelineruns/$(context.pipelineRun.name)
type: string
tasks:
{{- include "gerrit-review-start" . | nindent 4 }}
- name: your-custom-steps
taskRef:
name: custom-task
runAfter:
- fetch-repository
workspaces:
- name: source
workspace: shared-workspace
{{ include "gerrit-review-vote" . | nindent 2 }}
Further steps
While working with Tekton, we’ve faced several challenges. Most of them are already addressed with the latest releases (see CHANGELOG.md), and we plan to share more details in our publications. Find below the list of our published and planned articles on Tekton integration:
- Part 1: Tekton adoption
- Part III: The EDP Tekton Interceptor
- Part IV: Developing and Delivering Tekton Pipelines. How to scale, not fail
- Part V: Tekton Dashboard. Impersonation in a nutshell
Follow us to stay informed.