Getting Started with Tekton Pipelines Part 1 — Container Image Build and Push

Tekton is an open-source project for creating CI/CD systems.
allowing developers to build, test, and deploy across cloud providers and on-premise systems.

In this Part 1, I will address the Basic Pipeline with real everyday tasks.
In the next part, I will address the Tekton Triggers.
The separation into two parts hopefully would minimize the complexity.

There is a lot of documentation both at the Tekton site and at the Tekton GitHub project.
But still, when you are about to use it for real tasks, it is confusing, and maybe an exact real day-to-day example would help, and when I was starting with Tekton, I wished for a simple example that would show me the way.

Simple Example
I have created a very simple example for Tekton Pipelines.
These examples contain the minimal entities needed to create a pipeline.
It would be better to follow them first, you can read it over https://github.com/yossicohn/tekton-pipelines/blob/main/Part-1-pipelines/simple-example/README.md
The README.md there would walk you through the implementation, step by step.

What will we do?
The article would go and try to describe the different entities and concepts in the Tekton flow.
We will go from top to bottom, meaning:
Pipelinerun ➨ Pipeline ➨Task, and in the journey, we will go over what we meet.

Our Task:
1. Pull a private git repository
2. Build the container image
3. Push the image to the private container registry

Obviously, we would use Tekton as the CI framework to define and execute the steps.
We will use Kaniko for the container image build and image push to the registry.

For more info on Kaniko, you can see my previous article kaniko-builds-with-private-repository

Note, in this article, I would assume the reader has a basic understanding of Kubernetes, moreover, I will not reference the stage of installing Tekton as you can have it over Tekton’s installation guide.

So let’s get started!

Tekton defines Kubernetes resources which are its pipeline building blocks.
Following are the resources we will use and some basic definitions for you to grasp.
● Pipeline
● Task
● PipelineResource
● Workspace
● Results
● Parameters

With the following schema, you can learn about the entities and the relationships between them.

image from: https://technologists.dev/posts/tekton-jx-pipelines/

Basic familiarity with some of the entities:

Pipeline

A Pipeline defines an ordered series of Tasks that you want to execute along with the corresponding inputs and outputs for each Task. You can specify whether the output of one Task is used as an input for the next Task using the from property. Pipelines offer the same variable substitution as Tasks.

Task

A Task defines a series of steps that run in a desired order and complete a set amount of build work. Every Task runs as a Pod on your Kubernetes cluster with each step as its own container.

PipelineResources

PipelineResources in a pipeline are the set of objects that are going to be used as inputs to a Task and can be output by a Task.

A Task can have multiple inputs and outputs.
For example:

A Task's input could be a GitHub source that contains your application code.

A Task's output can be your application container image (that was built) which can be then deployed in a cluster.

Another example for a Task's output can be a jar file to be uploaded to a storage bucket.

Workspace

A Pipeline can use Workspaces to show how storage will be shared through its Tasks. For example, Task A might clone a source repository onto a Workspace and Task B might compile the code that it finds in that Workspace. It’s the Pipeline's responsibility to ensure that the Workspace these two Tasks use is the same, and more importantly, that the order in which they access the Workspace is correct.

You can read more on workspace here

Results
A Task can emit a string result that users can view and pass to other Tasks in a Pipeline.
These results have a wide variety of potential usages.

To define a Task's results, use the results field. Each results entry in the Task's YAML corresponds to a file that the Task should store the result in. These files should be created by a Task in the /tekton/results directory. The directory itself is created automatically if the Task has a results field but it’s the responsibility of the Task to generate its contents.

You can read more on results here

Parameters

You can specify parameters, such as compilation flags or artifact names, that you want to supply to the Task at execution time. Parameters are passed to the Task from its corresponding TaskRun
Parameters are passed to the Pipeline from its corresponding PipelineRun and can replace template values specified within each Task in the Pipeline.

Our Pipeline

In our example, we will have a Pipeline that would be finalized by pulling, building, and pushing an image to a container registry.
The pipeline would be composed out of 2 tasks:

1. git-pull-source-and-get-sha — which will pull the git repository and will define the image push tag out of the git last commit sha1

2. build-docker-image-from-source — which will use Kaniko to build the container image and push it to the DockerHub private registry (we will show usage for AWS ECR as well)

Below you can see the Pipeline definition via the pipeline.yaml
The Pipeline contains the following references:

• PipelineResources — which are Git and Image resources named as input-git, output-image
• Tasks — via the task-refelement under tasks element
  we will use:
  - build-docker-image-from-source
  - git-pull-source-and-get-sha
• Workspace — we will use a single workspace, to share our git source.
  The workspace in this example is a localhost volume (and is defined by pipelinerun.yaml which we will reference later), but we can use other PVC’s

Our Pipeline defines:

• PipelineResources used by the tasks
• Results to be used by the contained Tasks
• Parameters passed to the tasks
• Workspace passed to the tasks

Note, an example for results usage:
“$(tasks.pipeline-git-source.results.dockerfile-path)”
This way we get the result propagated from the task pipeline-git-source by the name dockerfile-path

All the definitions regarding the Workspace (volumes), Parameters, ServiceAccount, and PipelineResources are found at the Pipelinerun which is used by the pipeline.yaml

In the following schema, you can see the entities relationships

• Pipeline — is manifested by PODS (Pod per Task)
• Task — is manifested as a POD
• Steps — are manifested as Containers in the Task POD

image from: https://developer.ibm.com/devpractices/devops/articles/introduction-to-tekton-architecture-and-design/

Pipelinerun

In the pipelinerun.yaml you can see the defined resources that would be used in the pipeline.yaml, this way we can inject the Pipeline with different resources, having the pipeline behave as a template.

Here we define the PipelineResources which in this case are:

• Input Resource (type git) — go-api-skeleton-git
  used to define git repositories as a resource
• Output Resource (type image) — go-api-skeleton-image
  used to define container registry as a resource
• Workspace — a volume claim which is named as shared-data and is referenced from the pipeline
• ServiceAccount — with which we define the pipeline credentials
• Parameters — the parameters injected into the pipeline

Running the Pipeline

To run the pipeline, we need to create the resources in the kubernetes cluster.
To follow the instructions, it would be better to clone the repo and cd into part-1-pipelines.

Over the Part 1 README.md at https://github.com/yossicohn/tekton-pipelines/blob/main/Part-1-pipelines/README.md you can have a step by step instructions on how to create the secrets and PipelineResources.
To show the example you can refer to my private GitHub repo yossicohn/go-api-skeleton and my private DockerHub Registry.

• Creating the pipeline https://github.com/yossicohn/tekton-pipelines/blob/main/Part-1-pipelines/pipeline.yaml

• Create the PipelineResources
  Creating two resources, git and image
  Note, that you need to update the resources to reference your own!

• Create the Tasks
  Create the 2 tasks as defined above.

• Create Secrets
  After you have the secret yaml files ready, as instructed in the README.md we can create them (this way we will have the git SSH secret defined and the Container Registry secret)
  You can see the secret usage in the tasks yaml files.
• Create authentication via k8s secret of type docker-registry

• Create the secret for the git SSH Credentials

Last, we need to create the Pipelinerun

kubectl apply -f pipeline/pipelinerun.yaml

Get the Pipelinerun list and logs via the Tekton CLI (also referenced in the README)

Note, regarding the workspace,
The workspace volume would be created automatically and is dependent on the infra you are running on (for Minikube/Kind you will have hostpath volume, and in AWS and alike you would get a provisioned Block device)

Summary

In this Part, we referenced the Tekton Pipeline and saw how we can use it to build a pipeline.
In the next part, we will reference the Tekton Triggers, and learn how we can use triggers, and Github triggers to trigger a pipeline.

Thanks for assisting Regev Golan