CI/CD for Knative serverless apps on Kubernetes with Concourse
This post describes how to deploy and update scale-to-zero REST API service on Kubernetes with Concourse CI/CD pipelines.
If you haven’t heard about Knative yet, it’s a Kubernetes-based platform that allows you to build, deploy, and manage modern serverless workloads.
Installing Knative
First of all, let’s install Knative middleware in a Kubernetes cluster. We are going to give a brief example on how to set it up on GKE, but feel free to follow the official installation guide.
Knative depends on an ingress gateway which is capable of routing requests to Knative services, so let’s install it first:
$ kubectl apply -f https://github.com/knative/serving/releases/download/v0.4.0/istio-crds.yaml $ kubectl apply -f https://github.com/knative/serving/releases/download/v0.4.0/istio.yaml$ kubectl label namespace default istio-injection=enabled
Wait until Istio is ready:
$ watch kubectl -n istio-system get podsInstall Knative components:
$ kubectl apply -f https://github.com/knative/serving/releases/download/v0.4.0/serving.yaml -f https://github.com/knative/build/releases/download/v0.4.0/build.yaml -f https://github.com/knative/eventing/releases/download/v0.4.0/release.yaml -f https://github.com/knative/eventing-sources/releases/download/v0.4.0/release.yaml -f https://github.com/knative/serving/releases/download/v0.4.0/monitoring.yaml -f https://raw.githubusercontent.com/knative/serving/v0.4.0/third_party/config/build/clusterrole.yamlWait until all knative-components are ready:
$ kubectl get pods -n knative-monitoring
$ kubectl get pods -n knative-build
$ kubectl get pods -n knative-eventing
$ kubectl get pods -n knative-source
$ kubectl get pods -n knative-servingBuild a sample REST API application
Use a sample Golang application from Knative repository:
$ export GOPATH=`pwd`
$ go get -d github.com/knative/docs/docs/serving/samples/rest-api-go$ cd $GOPATH/src/github.com/Aptomi/knative-docs
$ export REPO=<DOCKER_HUB_ACCOUNT>/rest-api
$ docker build --tag "${REPO}" --file docs/serving/samples/rest-api-go/Dockerfile .
Build it:
Sending build context to Docker daemon 59.89MB
Step 1/8 : FROM golang AS builder
---> 36e5881731e4
Step 2/8 : WORKDIR /go/src/github.com/knative/docs/
---> Using cache
---> 2284037457d4
Step 3/8 : ADD . /go/src/github.com/knative/docs/
---> 535fc84bcba5
Step 4/8 : RUN CGO_ENABLED=0 go build ./docs/serving/samples/rest-api-go/
---> Running in 7215a7bcdbcf
Removing intermediate container 7215a7bcdbcf
---> 59acd91aecdf
Step 5/8 : FROM gcr.io/distroless/base
latest: Pulling from distroless/base
41d633039bbf: Pull complete
5f5edd681dcb: Pull complete
Digest: sha256:a9bc1c4720b17441d5fb95937a66616a9ae96339599e3959ae2f5db71e7e089e
Status: Downloaded newer image for gcr.io/distroless/base:latest
---> a5a1c6b2c22f
Step 6/8 : EXPOSE 8080
---> Running in c2d6fcdd6098
Removing intermediate container c2d6fcdd6098
---> 45cc32b2e3a6
Step 7/8 : COPY --from=builder /go/src/github.com/knative/docs/rest-api-go /sample
---> c06fa51dcd1f
Step 8/8 : ENTRYPOINT ["/sample"]
---> Running in cd2559eb09fe
Removing intermediate container cd2559eb09fe
---> b5f712672524
Successfully built b5f712672524
Successfully tagged <DOCKERHUB_ACCOUNT>/rest-api:latestAnd push it into a Docker repository:
$ docker push ${REPO}
The push refers to repository [docker.io/<DOCKERHUB_ACCOUNT>/rest-api]
33ab4bae26dc: Pushed
883ad731c7dd: Pushed
e9c843895906: Pushed
latest: digest: sha256:83272c0f1117abbbfa8a7f95aa8ebbcadc506e9ca70cb3022ed41755687011c8 size: 949Deploying serverless app via CI/CD pipeline
Now it’s time to build a Concourse CI/CD pipeline to deploy this REST API application in a “serverless” mode to Kubernetes.
This pipeline will check for a new application image on docker hub, and will trigger the deployment of REST API application to Kubernetes/Knative when the image is updated.
jobs:
- name: rest-api
plan:
- get: docker-image
trigger: true - put: knative-serving
params:
kubectl: cluster-info
app_name: rest-api
image: ((docker_repo))
namespace: defaultresource_types:
- name: knative-serving
source:
repository: aptomisvc/concourse-knative-serving-resource
tag: latest
type: docker-image
resources:
- name: knative-serving
type: knative-serving
source:
kubeconfig: ((kubeconfig)) - name: docker-image
type: docker-image
source:
repository: ((docker_repo))
tag: latest
The custom Concourse resource that we are using is written by Aptomi. It allows to deploy applications to Knative from Concourse pipelines:
It requires only three parameters - name of the application, docker image, and kubernetes namespace where the application will be deployed.
You can download the pipeline from our repository and import it into Concourse:
$ curl -LO https://raw.githubusercontent.com/Aptomi/concourse-pipelines/master/knative-serving/01_knative_serving_pipeline.yml$ fly -t concourse set-pipeline -p knative-serving-example -c ./01_knative_serving_pipeline.yml --var "kubeconfig=$(cat kubeconfig)" --var docker_repo="${REPO}"$ fly -t concourse unpause-pipeline -p knative-serving-example
After a few moments, the pipeline will automatically get triggered by a docker-image resource:
And application will be deployed:
Let’s check that the app has really been deployed to Knative:
$ kubectl -n default get ksvcNAME DOMAIN LATESTCREATED LATESTREADY READY REASON
rest-api rest-api.default.example.com rest-api-00001 rest-api-00001
Our REST API service landed on rest-api.default.example.com domain. Let’s check other objects like service, route and deployment:
Verifying serverless app
Now, let’s make a simple request to our deployed REST API via Kubernetes Ingress. Starting from 0.4.0 version, knative uses istio-ingress gateway to handle network requests.
$ INGRESSGATEWAY=istio-ingressgateway
$ INGRESSGATEWAY_LABEL=istio $ export INGRESS_IP=`kubectl get svc $INGRESSGATEWAY --namespace istio-system --output jsonpath="{.status.loadBalancer.ingress[*].ip}"`$ export SERVICE_HOSTNAME=`kubectl get ksvc rest-api --output jsonpath="{.status.domain}"`
In our case, the external IP address is 35.188.208.253 and the hostname is rest-api.default.example.com. Let’s make the first HTTP request:
$ time curl --header "Host:$SERVICE_HOSTNAME" http://${INGRESS_IP}
Welcome to the stock app!
real 0m8,122s
user 0m0,008s
sys 0m0,023sAs you can see, the first request handles about 8 seconds. One of the features of Knative is automatic scaling up and down to zero. In this particular case a new HTTP request was received but application pod wasn’t running, so it took time for Knative to launch an app and serve our first request.
$ kubectl get pods -l app=rest-api-00001NAME READY STATUS RESTARTS AGELet’s check pods:
$ kubectl get pods -l app=rest-api-00001NAME READY STATUS RESTARTS AGE
rest-api-00001-deployment-99c8c558d-45fs8 3/3 Running 0 1mLet’s send another HTTP request and see how long it takes:
$ time curl --header "Host:$SERVICE_HOSTNAME" http://${INGRESS_IP}
Welcome to the stock app!
real 0m0,350s
user 0m0,024s
sys 0m0,014sThe request took 0.35 seconds because the pod is already running and available to serve requests immediately.
If application won’t receive other HTTP requests for some time, Knative will scale it down to zero and stop the corresponding pod.
Updating serverless app via CI/CD pipeline
Now, let’s make some change in our application. For example, change the line with banner message in docs/serving/samples/rest-api-go/stock.go file
Then rebuild and push docker image to the repo:
$ docker build --tag "${REPO}" --file docs/serving/samples/rest-api-go/Dockerfile .
$ docker push ${REPO}
The push refers to repository [docker.io/aptomi/rest-api]
883ad731c7dd: Layer already exists
e9c843895906: Layer already exists
34aCxabe27yy: Pushed
latest: digest: sha256:2414c0faa47abxyfa8a7b95aa8ebbcadd440e9ca70cb2311ed4175514h01aa8 size: 949After pushing the updated app image, Concourse resource will automatically discover it and trigger the same CI/CD pipeline to redeploy our app to Knative:
Let’s check that the app has been updated via Concourse CI/CD pipeline:
$ curl --header "Host:$SERVICE_HOSTNAME" http://${INGRESS_IP}
Welcome to the stock v2 app! Summary
Concourse, being a powerful CI/CD framework, can be easily used to drive deployments and updates of serverless apps.
This is just a small example, but you can easily build a more complete CI/CD pipeline and automate the entire process of taking your serverless application to production— from git commit, building, testing, and publishing artifacts to deploying and updating application on Kubernetes/Knative.
If you have any questions, don’t hesitate to leave a comment below or reach out to me directly at sryabin@aptomi.io.
