Open in app

Sign in

Write

Sign in

Part II: Tekton and Gerrit. How to make things voting

Sergiy Kulanov
EPAM Delivery Platform
Published in
7 min readJan 4, 2023

This is the second article in the series of publications describing our migration path to Tekton for the EPAM Deliver Platform (EDP). The first part introduces our platform's requirements, baseline architecture, and target state. One of the requirements discussed earlier is to keep the integration of the existing EDP toolset. Today EDP supports three version control systems (VCSs) that Tekton Pipelines work with: Gerrit, GitHub, and GitLab. This article covers the topic of Tekton integration with Gerrit VCS since this is our default Git repository management and code review tool on the platform.

Deployment overview

To better understand Tekton and Gerrit integration, let’s review their configuration and deployment approaches. There are no specific requirements for Tekton deployment; you can follow their official documentation.

We perform Gerrit configuration on three different levels:

  • Container level (Dockerfile). EDP runs its own Gerrit container image, which enables additional configuration as a part of the entrypoint.sh script.
  • Deployment level (Helm chart). Kubernetes objects are deployed with the chart templates.
  • Kubernetes or runtime level (Gerrit operator). The operator supports Custom Resources (CRs) to manage Gerrit components, such as projects, groups, group membership, and permissions.

We plan to switch to the official upstream Gerrit container and Helm chart and consolidate all the configuration steps in our gerrit operator.

We use the Gerrit Webhook plugin to emit events from Gerrit and pass them to Tekton’s EventListner. The plugin is configured on two levels: global and project (with inheritance enabled). To define global values, we put the configuration section in the gerrit.config file (see the code snippet below). Consult our entrypoint.sh file for more details.

[plugin "webhooks"]
  connectionTimeout = 3000
  socketTimeout = 2500
# It is mandatory to set 1 for Tekton EventListener
  maxTries = 1
  retryInterval = 5000
  threadPoolSize = 3

maxTries = 1 is a mandatory value because Tekton's EventListener returns a 202 ACCEPTED HTTP response when it can process the request. At the same time, the Gerrit webhook plugin expects one of the following success codes: sc == SC_CREATED || sc == SC_NO_CONTENT || sc == SC_OK, which are 201, 204, 200. This leads to resending events with maxTries count each time.

We use the All-Projects refs/meta/config branch and inheritance to populate webhook project-level configuration. See the code snippet below:

cat << EOF > "webhooks.config"
[remote "changemerged"]
  url = http://el-gerrit-listener:8080
  event = change-merged
[remote "patchsetcreated"]
  url = http://el-gerrit-listener:8080
  event = patchset-created
EOF

We catch two types of events, changemerged and patchsetcreated, for all Gerrit projects and send them to theel-gerrit-listener Kubernetes Service on port 8080. See the conceptual view diagram below.

Conceptual view diagram of Gerrit and Tekton integration

Network Load Balancer routes external traffic: Ingress — for HTTP, NodePort — for TCP (run Git over SSH). The Gerrit operator performs Gerrit instance configuration. It also creates the gerrit-ci-user-sshkey secret, which contains the SSH private key for the CI user — EDP CI Bot. This key is mounted by the Tekton task gerrit-ssh-cmd and is used for CI voting over SSH.

Work with Gerrit events

This section describes Gerrit's event processing flow with Tekton. We need to configure EventListener and respective Pipelines with Tasks. In EDP, Gerrit EnvetListener catches two events: changemerged and patchsetcreated. To better understand the event payload structure, you can consult official documentation or use the gerrit stream-events command:

# check that we have access to Gerrit
ssh -p 30004 admin@127.0.0.1

  ****    Welcome to Gerrit Code Review    ****

  Hi admin, you have successfully connected over SSH.

  Unfortunately, interactive shells are disabled.
  To clone a hosted Git repository, use:

  git clone ssh://admin@gerrit-example.com:30004/REPOSITORY_NAME.git

Connection to 127.0.0.1 closed.

# Run command to listen for patchsetcreated event
# and then create Patchset in Gerrit either thought UI or command-line
bash-5.1# ssh -p 30004 admin@127.0.0.1 gerrit stream-events -s patchset-created

The JSON below presents a patchset-created event intercepted by the stream-events command:

{
    "uploader": {
        "name": "Sergiy Kulanov",
        "email": "sergiy_kulanov@example.com",
        "username": "sergiy_kulanov@example.com"
    },
    "patchSet": {
        "number": 1,
        "revision": "81a59313f180e0996b7e59484d50d439b8fb41c8",
        "parents": [
            "e0a6d6ab4c0d0b70758a94dcb4434162311e8db1"
        ],
        "ref": "refs/changes/21/21/1",
        "uploader": {
            "name": "Sergiy Kulanov",
            "email": "sergiy_kulanov@example.com",
            "username": "sergiy_kulanov@example.com"
        },
        "createdOn": 1672061412,
        "author": {
            "name": "Sergiy Kulanov",
            "email": "sergiy_kulanov@example.com",
            "username": "sergiy_kulanov@example.com"
        },
        "kind": "REWORK",
        "sizeInsertions": 9,
        "sizeDeletions": 0
    },
    "change": {
        "project": "petclinic",
        "branch": "master",
        "id": "I18908c3df9229aa6b944dc9d6153e0e5b9a6bf5d",
        "number": 21,
        "subject": "Demo change",
        "owner": {
            "name": "Sergiy Kulanov",
            "email": "sergiy_kulanov@example.com",
            "username": "sergiy_kulanov@example.com"
        },
        "url": "https://gerrit.example.com/c/petclinic/+/21",
        "commitMessage": "Demo change\n\nChange-Id: I18908c3df9229aa6b944dc9d6153e0e5b9a6bf5d\n",
        "createdOn": 1672061412,
        "status": "NEW",
        "wip": true
    },
    "project": "petclinic",
    "refName": "refs/heads/master",
    "changeKey": {
        "id": "I18908c3df9229aa6b944dc9d6153e0e5b9a6bf5d"
    },
    "type": "patchset-created",
    "eventCreatedOn": 1672061412
}

We use the payload above to configure the EventListener, define Cel filters, and consume values in Tekton Pipelines by extracting them with Tekton’s TriggerBinding custom resource. See the example of the EDP-specific configuration of EventListener for the changemerged event:

apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
  name: gerrit-listener
spec:
  triggers:
    - name: gerrit-listener-app-build
      interceptors:
        # we trigger this only on MERGE events
        - ref:
            name: "cel"
          params:
            - name: "filter"
              value: "body.change.status in ['MERGED']"
        # we run our own EDP interceptor to populate the payload with EDP-specific data
        - ref:
            name: "edp"
            kind: NamespacedInterceptor
        # we do some magic on data from edp interceptor
        - ref:
            name: "cel"
          params:
            - name: "filter"
              value: "extensions.spec.type in ['application']"
            - name: "overlays"
              value:
              - key: cbtype_short
                expression: "extensions.spec.type.truncate(3)"
      bindings:
        # we extract the required data from the payload
        - ref: gerrit-binding-build
      template:
        # and finally, create a PipelineRun instance with proper parameters
        ref: gerrit-build-app-template

gerrit-listener-app-build triggers on MERGE event, then we enrich the original Gerrit event payload with the EDP codebase spec (see the EDP interceptor) and finally do filtering and transformation before instantiating Tekton PipelineRun.

CI code review

For those who haven’t worked with Gerrit, we recommend getting acquainted with Gerrit workflow. One of the enablers to merging code into the main trunk on EDP is to have several mandatory approvals:

  • +2 Code-Review from a team lead or a person with such permissions.
  • +1 Verified from the CI System (Code-Review pipeline of Tekton).

To work with Gerrit review (voting), we’ve created a separate Tekton Task named gerrit-ssh-cmd. This task mounts the SSH private key of the EDP CI Bot user created in Gerrit and runs the command passed in the SSH_GERRIT_COMMAND parameter.

Today, there are two scenarios of how we reference this task from the pipelines:

  • Start and Stop of Code-Review pipeline on the patchsetcreated event. We reset the Verified value to zero when the pipeline starts, then put +1 if it succeeds, and -1 if it fails.
  • Start of Build pipeline on thechangemerged event. We put the link to the build job.

For the above scenarios, we put the link with the Tekton Dashboard PipelineRun in the Gerrit UI.

See below the examples of the Code-Review pipeline start section:

- name: gerrit-notify
  taskRef:
    name: gerrit-ssh-cmd
  params:
    - name: GERRIT_PORT
      value: {{ .Values.global.gerritSSHPort }}
    - name: SSH_GERRIT_COMMAND
      value: review --verified 0 --message 'Build Started $(params.pipelineUrl)' $(tasks.fetch-repository.results.commit)
  workspaces:
    - name: ssh-directory
      workspace: ssh-creds

And the finally section of the Code-Review pipeline:

finally:
  - name: gerrit-vote-success
    when:
      - input: "$(tasks.status)"
        operator: in
        values: ["Succeeded"]
    taskRef:
      name: gerrit-ssh-cmd
    params:
      - name: GERRIT_PORT
        value: {{ .Values.global.gerritSSHPort }}
      - name: SSH_GERRIT_COMMAND
        value: "review --verified +1 --message 'Build Successful $(params.pipelineUrl)' $(tasks.fetch-repository.results.commit)"
    workspaces:
      - name: ssh-directory
        workspace: ssh-creds

  - name: gerrit-vote-failure
    when:
      - input: "$(tasks.status)"
        operator: in
        values: ["Failed"]
    taskRef:
      name: gerrit-ssh-cmd
    params:
      - name: GERRIT_PORT
        value: {{ .Values.global.gerritSSHPort }}
      - name: SSH_GERRIT_COMMAND
        value: "review --verified -1 --message 'Build Failed $(params.pipelineUrl)' $(tasks.fetch-repository.results.commit)"
    workspaces:
      - name: ssh-directory
        workspace: ssh-creds

The image below is an example of pipeline failure because of a sonar check. The gerrit-vote-failure task puts Verified -1 for a patch (merge request), and the gerrit-vote-success is skipped.

Tekton Dashboard: Vote Gerrit failure

As a result, we have -1 from the EDP CI bot in the Gerrit UI and merge block.

Gerrit UI: Failed Pipeline report

To simplify pipeline management and scale, we’ve implemented CI review primitives (a composition of Tekton tasks) in the separate common-library Helm repository. This approach avoids code repetition and keeps charts DRY. You can concentrate on new logic and re-use gerrit-review-start, gerrit-review-vote, and gerrit-build-start primitives.

This is how you can define the Code-Review pipeline with your-custom-steps step, using the Helm chart approach and gerrit-review-start/gerrit-review-vote definitions:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: gerrit-mycustom-review
  labels:
    {{- include "edp-tekton.labels" . | nindent 4 }}
spec:
  workspaces:
    - name: shared-workspace
    - name: ssh-creds
  params:
    - name: pipelineUrl
      default: https://tekton-{{ .Release.Namespace }}.{{ .Values.global.dnsWildCard }}/#/namespaces/$(context.pipelineRun.namespace)/pipelineruns/$(context.pipelineRun.name)
      type: string

  tasks:

{{- include "gerrit-review-start" . | nindent 4 }}

    - name: your-custom-steps
      taskRef:
        name: custom-task
      runAfter:
        - fetch-repository
      workspaces:
        - name: source
          workspace: shared-workspace

{{ include "gerrit-review-vote" . | nindent 2 }}

Further steps

While working with Tekton, we’ve faced several challenges. Most of them are already addressed with the latest releases (see CHANGELOG.md), and we plan to share more details in our publications. Find below the list of our published and planned articles on Tekton integration:

Follow us to stay informed.

Tekton
Kubernetes
Cicd
DevOps
Developer Portal

--

--

Sergiy Kulanov
EPAM Delivery Platform

Written by Sergiy Kulanov

60 Followers
Editor for

Systems Architect at EPAM, Ukraine. OpenSource contributor. DevOps, Build, CI Engineer. Current Role: Architect on EPAM Delivery Platform

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams