Install Secure Helm In Google Kubernetes Engine (GKE)
In my last post, I talked a lot about the joys of Helm and why you should spend the time installing it into your Kubernetes Cluster. If you need a refresher go read that article now.
Within the last post we did talk about the importance of securing Helm and the Tiller server that does the actual installation of containers into your Kubernetes. You wouldn’t want someone getting in between your Kubernetes Cluster through Tiller and starting to install various Containers without your knowledge. To prevent this case we need to go further with our Helm installation and add Transport Layer Security (TLS).
Don’t worry though, this isn’t going to be too painful, and fortunately for you I’ve already done the hard part. In this article we will look at how to create the necessary keys for your secure Helm installation, some pitfalls that plague many people on the internet, and then a quick example of how to install Helm Charts now that we have our TLS security setup.
If you haven’t gone through or even read the first part of this series you might be lost, have questions where the code is, or what was done previously. Remember this assumes you’re using GCP and GKE. I will always provide the code and how to test the code is working as intended.
Note: The Helm site does include many great docs about the Secure Helm Installation however I feel it is nice to see how everything ties together in a working example along with helping with some of the issues that keep getting in the way. I’d recommend reading through their docs at some point.
Creating The TLS Keys
WARNING WARNING WARNING! This is the step that almost everyone messes up. After you have all of this correct the next steps are really easy.
How do we solve the problem of messing up key creation? Easy. We truly understand the key development process and then it is really simple. Where most people get things wrong is they just copy half-explained scripts from the internet and then have no idea why things go badly. So let’s start understanding the key development process.
It all starts with using the openssl command to build our main Certificate Authority (CA) key.
openssl genrsa -out ca.key.pem 4096This generates a Certificate Authority (CA) key that will be used to sign other keys in the process. So we can think of this visually let’s mark this key as A.
With this key ready we need to create a certificate that is signed with the key and various information about your company/organization.
openssl req -key ca.key.pem -new -x509 \
-days 7300 -sha256 \
-out ca.cert.pem \
-extensions v3_ca \
-subj "${SUBJECT}"
Next we need to create a key for Tiller — the Helm Server that manages Helm Charts. You’ll need to create one key PER Tiller host. In this case we only have one so we only need one.
openssl genrsa -out tiller.key.pem 4096
Now the part that confuses people. We need to create a Certificate for the Tiller Host Key (Key “B”) that is also signed with your company/organization AND is signed with the original certificate. This is done so as you may need to expire keys in the future you can still test against the original certificate that is mathematically buried into your signed certificate.
openssl req -new -sha256 \
-key tiller.key.pem \
-out tiller.csr.pem \
-subj "${SUBJECT}"
openssl x509 -req -days 365 \
-CA ca.cert.pem \
-CAkey ca.key.pem \
-CAcreateserial \
-in tiller.csr.pem \
-out tiller.cert.pem
Finally we need also need to repeat the same exact process for all of your users that will need to access Helm. In this case we only have one user that we are going to call “Helm”. If you have multiple users due to your security needs just keep repeating this process: Creating a key for the user, followed by mixing that key with the original certificate to create a user certificate.
openssl genrsa -out helm.key.pem 4096And the code necessary for the Helm user certificate.
openssl req -new -sha256 \
-key helm.key.pem \
-out helm.csr.pem \
-subj "${SUBJECT}"
openssl x509 -req -days 365 \
-CA ca.cert.pem \
-CAkey ca.key.pem \
-CAcreateserial \
-in helm.csr.pem \
-out helm.cert.pemProducing the final Helm user certificate that is signed for the user but mathematically also contains the very first key.
With all these files ready and properly put together the next steps go very quickly.
Install Helm With TLS
Now with everything put together we just need to install Helm. We will be giving Helm the Tiller certificate and key along with the original ca.cert.pem. This is important because in the future various users will come to Helm with their own certificates (in our example the Helm user) wanting access to our Helm instance. Helm with mathematically check the user against the ca.cert.pem for mathematical acceptance and then either allow or deny the user.
helm init \
--tiller-tls \
--tiller-tls-cert tiller.cert.pem \
--tiller-tls-key tiller.key.pem \
--tiller-tls-verify \
--tls-ca-cert ca.cert.pem \
--service-account tillerNow our installed Helm instance knows all that it needs to know for security checks in the future.
Verify Your Helm With TLS Installation
You can verify that Helm is installed and using TLS properly by having the Helm user try to interact with Helm. We do this really simply by calling for a list of installed Helm Charts. If successful we will see no response. If not successful, an error.
helm ls --tls \
--tls-ca-cert ca.cert.pem \
--tls-cert helm.cert.pem \
--tls-key helm.key.pemWith a successful response you already see that this Helm command requires way too many parameters to be lazy-coder-friendly. So we will move the certificate information into our helm home directory.
cp ca.cert.pem $(helm home)/ca.pem
cp helm.cert.pem $(helm home)/cert.pem
cp helm.key.pem $(helm home)/key.pemNow we can run the same command and Helm with assume the certificate information on future calls.
helm ls --tls # we just need to add --tls to our Helm commands!Again success! Now let’s install a Helm Chart.
Install A Helm Chart The Secure Way
If you saw how to install Redis in my last install Helm (insecurely) article then this command is going to look very familiar. The only difference is that we now have to add--tls to our Helm command. Everything else is the same.
helm install stable/redis \
--tls \ # look! this is the only change!
--values values/values-production.yaml \
--name redis-systemBoom! Everything should now be working. Obviously I could make this easier for everyone if I include a working example. Let’s see that next.
Quickly Install A Secure Helm With Provided Scripts
Now that we have a full understanding of how everything works. Let’s be lazy and [semi-]forget everything by putting it into a script that we can run and handle all the steps for us.
$ git clone https://github.com/jonbcampos/kubernetes-series.git
$ cd ~/kubernetes-series/helm/scripts
$ sh startup.sh
$ sh add_secure_helm.sh
$ sh add_secure_redis.shIf you run these commands in your Google Shell console then you’ll see everything setup for you and ready for active development. Exciting and easy!
Watch These Pitfalls With Certificates
Most people (me included) seem to run into the following problems.
- Mixing up keys (CA) and certificates (CERT) and certificate signing requests (CSR). If you don’t name your CAs, CERTs and CSRs properly it is really easy to get confused which CA you are signing into which CSR and finally into whatever CERT. Make sure the naming is set purposefully so you don’t mess things up.
- I found that you HAVE to set the signing subject and specifically the Common Name (CN) to have this work. I originally tried using the
-batchcommand but thehelm initcommand kept failing due to this. Once I set the CN in the subject everything started working. - When we move the certs into the
helm home, do notice that there is some renaming going on for Helm. You’ll want to make sure you move over the correct CERTs and KEY.
More Best Practices With A Secure Helm Install
In case you are extra paranoid and want to go further, here are some ways to do that.
- You can increase your security by segmenting Tiller even more by creating a namespace specifically for Helm/Tiller.
- In my example I include a Kubernetes Service Account specifically for Helm/Tiller. This helps again provide more security for your Helm install.
- You can continue securing your Helm/Tiller installation using Roll Based Access Controls (RBAC). I would highly recommend going to the docs and see how you can add RBAC to your Helm/Tiller installation.
Conclusion
This is more of your “Best practices” with Helm post that you might have been hoping for over my last post. I would say this is the recommended route by the experts. Now you should have in your Google Kubernetes Cluster Helm/Tiller installed and an easy/secure way to add in Helm Charts. Making your development cycles shorter and much more effective.
Teardown
Before you leave make sure to cleanup your project so you aren’t charged for the VMs that you’re using to run your cluster. Return to the Cloud Shell and run the teardown script to cleanup your project. This will delete your cluster and the containers that we’ve built.
$ cd ~/kubernetes-series/helm/scripts
$ sh teardown.shOther Posts In This Series
Questions? Feedback? I’m very interested to hear what issues you might run across or if this helped you understand a bit better. If there is something I missed feel free to share that too. We are all in this together!
Jonathan Campos is an avid developer and fan of learning new things. I believe that we should always keep learning and growing and failing. I am always a supporter of the development community and always willing to help. So if you have questions or comments on this story please ad them below. Connect with me on LinkedIn or Twitter and mention this story.
