Part 1: Tekton adoption

Published in

EPAM Delivery Platform

8 min readDec 15, 2022

We are starting a series of articles about adopting Tekton toward the EPAM Delivery Platform (EDP). Tekton is an excellent option for running Kubernetes native CI/CD workloads. We are still on this journey and have the first stable version available from our main trunk. This article describes our adoption path's overall scope and milestones: starting from the requirements and moving toward the first stable version with our new EDP 3.0 release.

Abstract

EDP enables quick project onboarding and launching of the development from sprint zero. EDP has a long history, and the first Continues Integration (CI) and Continues Delivery (CD) tool we adopted was Jenkins. Jenkins is still the de-facto standard for managing CI/CD workloads, but we are deprecating it in favor of Tekton. There are a couple of reasons for that:

Operational overhead. Jenkins scaling and upgrade delivery for core and plugins becomes one of the most time-consuming activities.
Pipelines implementation and support. A development team should have the proper expertise to write Jenkinsfile using DSL, Scripted Groovy Pipelines, or Freestyle Jobs. Scripted Pipelines require more programming skills and a higher entry threshold. Sometimes Groovy Pipelines are implemented with simple bash scripts wrapped with groovy.

This article focuses on the current and target states of the EDP CI/CD pipelines.

Requirements

Though the number of requirements is vast, we highlight some of them in this article. For EDP, we have several architecturally significant requirements (ASR):

Use Opensource solution.
The solution must follow Kubernetes native approach.

Functional requirements:

Migrate the existing pipeline functionality from two groovy Jenkins shared libraries (Stages and Pipelines) to Tekton.
Pipelines must be integrated with the existing CI/CD toolset.
Customers can upgrade from Jenkins to Tekton pipelines.

Non-functional requirements:

Maintainability and Manageability — notify the developers in case of pipeline failure; aggregate pipeline statuses onto a single dashboard.
Usability — onboard a new codebase from an existing Git repository or create a scaffold project in less than one minute; improve developers’ experience; pipelines must support versioning and be stored in code.
Scalability — run up to 500 pipelines in parallel.
Supportability — reduce time and effort for upgrade and maintenance. Critical upgrades must be delivered in less than 24 hours.

Baseline architecture

The following section describes the existing building blocks of the EDP pipelines.

EDP components

EDP is based on the following components:

Kubernetes custom resources (CRs) and EDP operators
Open-source tools
Jenkins CI/CD pipelines

All of them are integrated with Keycloak, which enables Single sign-on across the platform. In addition, we support three version control systems (VCS): Gerrit, GitLab, and GitHub. Before EDP version 3.0.0, Jenkins was a single CI/CD workload orchestrator that communicated with the integrated tools in two ways: Kubernetes API extension with custom resources and CI/CD Tools API.

The mandatory tools integrated with the pipeline are the following:

DefectDojo provides security orchestration and vulnerability management.
SonarQube manages quality gates and blocks pipelines in case of rule violations.
Nexus Repository stores artifacts.

The optional tools are:

Container registry stores build images.
ReportPortal is an automation tests aggregation and analytics platform.
Jira is a project management tool.

EDP conceptual view for versions < 3.0.0

We use Helm to install the components. Kubernetes operators perform further integration with the platform. For example, SonarQube is managed by the Sonar operator, which fulfills the following actions:

Creates necessary service accounts in SonarQube and exposes their credentials through Kubernetes secrets.
Creates the EDP Quality Gate and the Permission Template.
Enables SonarQube SSO with the help of the keycloak-operator.

EDP follows Kubernetes native approach, simplifying Tekton integration with the existing ecosystem.

EDP operators

EDP uses the operator pattern paradigm to manage platform resources. Today we have eleven operators available for installation using Helm charts, and one of them is Keycloak enabled through the OperatorHub. We plan to move all our operators there. The number of simultaneously running operators in the environment depends on the platform configuration. Some of them are mandatory, for example, the codebase operator. The others are optional, for example, Gerrit or Perf operators (see the picture below).

The admin-console operator manages the deprecated UI console, which is replaced with the edp-headlamp in the EDP v3.0.0. The edp-argocd operator is also deprecated because the latest Argo CD release (v2.5) brought the new feature, “Applications in any namespace”, earlier addressed by our operator.

The codebase-operator plays the central role and holds the state of the codebase (project) onboarded onto the EDP platform (see the code snippet below). The examples are Git repository path, language (such as Java, Python, JavaScript, Terraform, and Go), build tool (such as Apache Maven, Apache Gradle, and Npm), CI tool (Jenkins or Tekton), versioning (Default or EDP). These data form the Jenkins pipelines logic, for example: select the correct agent (pod) to build a Java code with the Maven tool, define the proper version for the artifact, push the resulting artifact into the snapshot or release Maven repository, and update the list of containers available for deployment on the environment.

apiVersion: v2.edp.epam.com/v1
kind: Codebase
metadata:
  name: demo-go
spec:
  buildTool: Go
  ciTool: Jenkins
  commitMessagePattern: >-
    ^\[EPMDEDP-\d+\]:\s(feat|fix|docs|style|refactor|test|chore)+!?:\s[A-Z][a-z]*.*
  defaultBranch: master
  deploymentScript: helm-chart
  emptyProject: false
  framework: other
  gitServer: gerrit
  jenkinsSlave: go
  jiraIssueMetadataPayload: '{"components":"EDP_COMPONENT","fixVersions":"EDP_SEM_VERSION-EDP_COMPONENT"}'
  jiraServer: jira-corp
  jobProvisioning: custom-gerrit
  lang: go
  repository:
    url: https://github.com/epam/edp-codebase-operator.git
  strategy: clone
  ticketNamePattern: EPMDEDP-\d+
  type: application
  versioning:
    startFrom: 2.6.0-SNAPSHOT
    type: edp

Pipelines

EDP supports the following technology stacks: Java, Python, JavaScript, .Net, Terraform, Jenkins Groovy Pipelines, Dockerfile, Helm, and Rego. You can also extend this list with your custom implementations. We define three pipeline types:

Code review. It starts when a new code is prepared for merge.
Build. It begins when the code review pipeline is succeeded and merged into a target branch.
Deploy. It can be automatic or manual and is used to deploy applications into the Kubernetes cluster.

The code review pipeline consists of the steps that perform unit/integration testing, SAST and Sonar scanning, compiling/building, and lint. You can also enable conventional commits and have nice changelogs generated automatically. When there are no issues with the pipeline, the team does a code review and merges the code, which triggers the build pipeline. In addition, the build pipeline ensures artifact versioning consistency across the platform. Finally, the release artifact goes to the release Nexus repository, otherwise, to the snapshot.

Once an artifact is published, we can deploy it using the deploy pipeline. In the picture below, we have three environments for testing and one for production. A user defines quality gates for artifact promotion from lower (Integration testing) to higher environments (Production). The promotion procedure makes artifacts available for the following environment only if all quality gates are passed. The quality gates can be based on integration, UI tests, and manual approvals. In addition, users can add custom ones, for example, the Deployment Risk.

CI/CD pipelines for EDP versions < 3.0.0

Target architecture

This section briefly describes the target solution on a high level. As a part of Jenkins deprecation, new components emerged on the platform:

Tekton runs CI/CD pipelines.
ArgoCD deploys applications using the GitOps approach.
EDP Headlamp (based on Headlamp) consolidates applications and pipeline management into a single place — a control plane.

EDP Headlamp

Headlamp is a simple react web UI application that represents a traditional Kubernetes Dashboard and allows natively working with resources inside the cluster. Headlamp has plugin system support, which we use to implement the EDP logic for working with custom resources (run CRUD operations):

Argo CD

Argo CD is used as a deployment tool for EDP. It has rich RBAC capabilities, Single sign-on, and multitenant support. The latest release (v2.5.0) can manage an application in arbitrary namespaces. That’s why we’ve deprecated our edp-argocd-operator. Visit our official documentation to get more information about how we perform the Argo CD teams' onboarding and management since this is a separate topic.

EDP Tekton

The edp-tekton GitHub repository consolidates all Tekton implementation logic on the platform. We use Helm charts to deploy the resources inside the Kubernetes cluster. Based on the functional and non-functional requirements, we decided to decouple Tekton logic into separate components (see the diagram below):

Common-library is the Helm chart of Library type, which stores the common logic shareable across all Tekton pipelines — CI Voting for VCSs: Gerrit, GitLab, GitHub. This library doesn’t generate specific Kubernetes resources but contains Helm templates that generate general Tekton tasks for reporting pipeline statuses. Use Helm dependency capabilities to consume the required templates.
Pipelines-library is the Helm chart of the Application type, which stores the core logic for the EDP pipelines. Tekton CRs like Pipelines, Tasks, EventListeners, Triggers, and other resources are delivered with this chart.
Custom-pipelines library is the Helm chart of the Application type, which implements custom logic running specifically for internal EDP development, for example, CI/CD and Release. It also demonstrates the customization flow on the platform.
Tekton-dashboard is a multitenancy-adopted implementation of the upstream Tekton Dashboard. It is configured to share Tekton resources across a single namespace. We have also investigated the Impersonate-Group configuration, and this is one of our next topics to share.
Interceptor is the EDP opinionated Tekton Interceptor, which enriches the payload from the VCSs events with EDP data from the Codebase CR specification. These data are used to define the logic of the Pipeline.

The schema below describes the logic behind the Tekton functionality on the platform:

1. The EventListener is exposed through the Ingress and listens for the events from the VCSs (Gerrit, GitHub, GitLab). It defines the filtering and routing rules for incoming requests.

2. Upon the Event Payload arrival, the EventListener uses interceptors to validate it. In addition, the EDP Interceptor extracts information from the codebases.v2.edp.epam.com CR and injects the received data into the Event Payload.

3. The Tekton Cel Interceptor does simple transformations of the resulting data and prepares them for the Pipeline parameters substitution.

4. The TriggerTemplate creates a PipelineRun instance with the required parameters extracted from the Event Payload. These parameters are mandatory for pipelines.

5. The PipelineRun has a mapping to the EDP Tekton pipelines using a template approach, which reduces code duplication. Each pipeline is designed for a specific VCS (Gerrit, GitLab, GitHub), technology stack (such as Java or Python), and type (code-review, build, deploy).

The Kubernetes native approach allows the creation of PipelineRun either with the kubectl tool or the EDP Headlamp UI.

Further steps

While working with Tekton, we’ve faced several challenges. Most of them are already addressed with the latest releases (see CHANGELOG.md), and we plan to share the details in our future publications:

Part II: Tekton and Gerrit. How to make things voting
Part III: The EDP Tekton Interceptor
Part IV: Developing and Delivering Tekton Pipelines. How to scale, not fail
Part V: Tekton Dashboard. Impersonation in a nutshell

Special thanks to Mariia Bochechko for helping with this publication.