Automatic Kubernetes deployment with Gaia and HashiCorp Vault
Kubernetes, the well known open source container management system, which has been widely adopted in the last years and has proven to be battle-hardened, is the de facto standard for deploying containerized applications.
Thousands of companies are already using it in production and this is only the beginning. Besides solving a lot of problems, it also brings new ones on the board which are sometimes tedious. Nowadays, when complex microservice architectures with hundreds of services are deployed with Kubernetes, the configuration management of deployment files becomes a nightmare. The amount of different YAML-Files which need to be maintained are tremendous and often has it’s own dangers.
Gaia is the perfect tool to automate Kubernetes deployments in a reproducible way, coordinate maintenance tasks for your cluster, cleanup of dangling resources in your cluster and much more.
In this guide we will setup a small local test environment including Kubernetes and HashiCorp Vault which we will use to store safely our secrets. We will create a powerful Gaia pipeline in Go which deploys an application to this Kubernetes cluster (with namespace and service).
We will also write tests for our pipeline which we can execute locally to test our jobs and make sure they are actually working.
What is Gaia?
Gaia is an open source automation platform which makes it easy and fun to build powerful pipelines in any programming language. Based on HashiCorp’s go-plugin and gRPC, gaia is efficient, fast, lightweight and developer friendly. Gaia is currently alpha! Do not use it for mission critical jobs yet!
Develop powerful pipelines with the help of SDKs and simply check-in your code into a git repository. Gaia automatically clones your code repository, compiles your code to a binary and executes it on-demand. All results are streamed back and formatted to a user-friendly graphical output.
- Docker for Mac or Docker for Windows for a local test Kubernetes cluster.
- (Optional) Golang 1.10.3 or later (to run tests locally).
2.1 Setup test environment: Docker with Kubernetes support
This is probably the easiest way to get a local Kubernetes cluster running. Open the Docker preferences window to activate Kubernetes:
After some setup time, your local Kubernetes cluster should be available and your
kubectl should be already configured. You can verify this by typing
kubectl cluster-info which should print something like this:
Kubernetes master is running at https://localhost:6443 KubeDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
That’s it for now.
2.2 (Optional): External Kubernetes Cluster
If you don’t want to start a local Kubernetes cluster you can also use an external cluster. The API-Server must be reachable from your computer and you need to have a valid Kube-Config. Additionally, we assume that all required certificates are contained in the Kube-Config file. If this is not the case, please copy the certificates plain into the Kube-Conf file. Example:
- name: docker-for-desktop
client-certificate-data: cert data here...
client-key-data: key data here...
3. Setup of Gaia and HashiCorp Vault
The first step is to download and start HashiCorp Vault as well as Gaia and join them into the same docker network.
For this tutorial we will use Docker but you can also install them manually if you prefer this installation method. Run the following commands in your terminal:
docker network create gaia-vault
docker run --cap-add=IPC_LOCK -d \
-e 'VAULT_DEV_ROOT_TOKEN_ID=root-token' \
-e 'VAULT_ADDR=http://localhost:8200' \
-e 'VAULT_TOKEN=root-token' \
-p 8200:8200 --name=vault --net=gaia-vault vault:latest
docker run -d -p 8080:8080 --net=gaia-vault --name=gaia gaiapipeline/gaia:latest
This will create a network called
gaia-vault which is used to allow communication between Gaia and HashiCorp Vault.
Then we start HashiCorp Vault with a development token (Don’t do this in production!). We expose the service on port 8200 which is optional and can be omitted if prefered.
The last command starts Gaia and exposes it on port 8080. For this tutorial we don’t mount the data directory to the host system. This means if you delete your Gaia Docker container all your data is lost. If you want to persist your data you can mount the data folder via the following parameter:
4. Store Kube-Config into Vault
The Kube-Config is particularly important for the connection to the Kubernetes API. It tells the Kube-Client (usually kubectl) where the API is located and a certificate for authentication and authorization purpose.
If you use the local Kubernetes cluster from Docker for Mac or Docker for Windows the Kube-Config should be already generated and placed on your file system (usually ~/.kube/config). Our Gaia pipeline should have later access to the Kubernetes API and therefore needs access to this Kube-Config file. The perfect place for this sensitive file is HashiCorp Vault where we will store it now.
Let’s assume that HashiCorp Vault is not locally installed. If you have HashiCorp Vault locally installed and did set VAULT_ADDR and VAULT_TOKEN correctly you can skip the next step.
Copy your Kube-Config into the Vault container and chroot into the container:
docker cp ~/.kube/config vault:/tmp/config
docker exec -it vault sh
We are now in the container and have access to the vault client. We already set VAULT_ADDR and VAULT_TOKEN during the start-up so we can now directly use the vault client to store our Kube-Config:
vault kv put secret/kube-conf conf="$(cat /tmp/config | base64)"
We encoded our config in Base64 so there are no problems with special characters.
5. Create Gaia pipeline in Go
The full pipeline code is available on github: k8s deployment tutorial code
If you have go installed locally you can run the following command in your terminal to get the pipeline code:
go get github.com/gaia-pipeline/tutorial-k8s-deployment-go
Otherwise clone the source code:
The pipeline should be quite straight forward if you are familiar with Go. It does the following steps:
- GetSecretsFromVault sets up the connection to Vault and gets our stored Kube-Config.
- PrepareDeployment sets up the connection to our Kubernetes cluster and parses all given paramets by Gaia and stores them in-memory.
- CreateNamespace sets up a new namespace in our Kubernetes cluster.
- CreateDeployment creates a Kubernetes deployment resource. It will update the resource if already exists.
- CreateService creates a Kubernetes service resource. It will update the resource if already exists.
- main is our main entry function which registers all functions and builds the dependency graph that tells Gaia in which order the functions should be executed. It also starts the Serve method which is provided by the Gaia Go-SDK.
6. Run local tests
Gaia really shines when it comes to testing your developed pipelines. The pipeline we have developed is pure Go code so we can test it like usual.
You can have a look at the main_test.go file or otherwise just run the tests in the clones pipeline folder:
go test -v ./...
This should print out something like this:
2018/07/13 23:28:40 All data has been retrieved from vault!
2018/07/13 23:28:40 Service 'nginx' has been created!
=== RUN TestCreateService
2018/07/13 23:28:40 Service 'nginx' has been created!
--- PASS: TestCreateService (0.02s)
=== RUN TestCreateDeployment
2018/07/13 23:28:40 Deployment 'nginx' has been created!
--- PASS: TestCreateDeployment (0.08s)
ok github.com/gaia-pipeline/tutorial-k8s-deployment-go 0.156s
These tests created now real resources in our test Kubernetes cluster in a few milliseconds. Run the following command to verify that:
kubectl get po -n nginx
This should print out something like this:
NAME READY STATUS RESTARTS AGE
nginx-6c5f974bd7-mlhgz 1/1 Running 0 13s
It works! For testing purpose, let us now remove the namespace (and therefore all related objects):
kubectl delete ns nginx
7. Add pipeline to Gaia
In step 3 we already started Gaia and it’s since then listening on port 8080. Open now the UI: http://localhost:8080/
You should now see the Gaia login view. Use the default credentials admin/admin to get access.
Now we will add our pipeline to Gaia by clicking on the top button “Create Pipeline”. This will forward us to the create pipeline view where we copy the URL of our pipeline (https://github.com/gaia-pipeline/tutorial-k8s-deployment-go) into the first input field.
We should give this pipeline a name to distinguish this one from others. Afterwards we click on “Create Pipeline” below.
Gaia will now automatically checkout our pipeline code, build it into a binary file, copy the binary file into the local pipelines folder and finally do some sanity checks.
You can view the status of this process below in the pipeline creation history table.
8. Add secrets to Gaia’s Vault
Gaia comes with it’s own internal Vault which can store secrets and sensitive data in a secure way. In our scenario it makes sense to store the address of our HashiCorp Vault instance as well as the access token which is needed so that our pipeline gets access to Vault.
Please keep in mind that this is just an example. In a real scenario it might be better to use different authentication methods to authenticate against HashiCorp Vault.
In the left menubar of Gaia’s UI you should be able to find the menu item Vault. Click on it to get to the Vault view. Click on Add Secrets to add a new secret to Gaia’s Vault. Create now two new secrets:
- Key: vault-token; Value: root-token
- Key: vault-address; Value: http://vault:8200
9. Start pipeline
Finally, we can go back to the Overview and should now see our pipeline. Click on Start Pipeline to start the pipeline. This should forward you to the arguments view where you have to specify the application name (e.g. myapp), full image name (e.g. nginx:latest), and the number of replicas (e.g. 1).
After the pipeline has been scheduled, you should see very quickly progress in the detailed view of your pipeline run.
Let’s recapitulate what steps we’ve taken:
- We setup a local test environment with Kubernetes, HashiCorp Vault and Gaia.
- We saved sensitive data (Kube-Config) in our HashiCorp Vault instance.
- We build a powerful pipeline in Go and wrote tests for it.
- We executed the tests.
- We created a new pipeline in Gaia. Gaia automatically compiled our pipeline.
- We added secrets to Gaia’s Vault and executed the pipeline.
With just a few steps we created a powerful pipeline which is secure, tested, includes no hidden magic (at the end it’s just Go code), fast, and the most important part: It was fun!