Hosting a private helm repository on OCI with ChartMuseum and OCI Object Storage
In a previous post, we deployed a helm chart (Redis) into Oracle Container Engine (OKE) which we pulled from the stable repository. By default, the stable repo is hosted at https://kubernetes-charts.storage.googleapis.com/. We can also manually add the incubator repository if we want to:
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
"incubator" has been added to your repositoriesThe container images in those repositories are also publicly available e.g. the default image for redis is bitnami/redis in bitnami’s container redis repository on Dockerhub.
We have already described how to host the container images privately on OCIR. What if we want to do the same for our helm charts?
Fortunately, there is a solution for that too. Step forward chartmuseum, an open source Helm Chart Repository server with support for various cloud storage backends, including OCI Object Storage.
Assumptions:
You’re using the terraform-oci-oke project to provision your OKE cluster. See this post for more details. Below is an architecture of what we’ll build:
Install nginx-controller
First, follow this post to install ExternalDNS.
Then, let’s install nginx-controller:
helm install --name nginxcontroller stable/nginx-ingress \
--set controller.name=controller \
--set defaultBackend.enabled=true \
--set defaultBackend.name=defaultbackend \
--set rbac.create=true \
--set controller.service.annotations."external-dns\.alpha\.kubernetes\.io/hostname"=chartmuseum.acme.comReplace chartmuseum.acme.com by your preferred hostname. We’ll then use this to access chartmuseum later.
Verify that an OCI Load Balancer has been created and a DNS ‘A’ record has been inserted in the DNS Zone.
Create OCI Bucket
In the OCI Console, navigate to Object Storage and create a bucket called ‘chartmuseum’:
Deploy chartmuseum
We want a similar experience to the stable or incubator repos i.e. anonymous GET but protected POST requests and we will use Basic Auth for authentication purposes.
Let’s first create a secret to hold the Basic Auth username and password:
kubectl create secret generic chartmuseum-auth --from-literal=user=curator --from-literal=pass=passwordYou also need to create a second secret that will allow chartmuseum to communicate with the OCI APIs.
Temporarily, copy your api keys to the bastion. You can also do this locally if you have kubectl and access to the kubeconfig locally.
Create a file ‘config’:
[DEFAULT]
user=<USER_OCID>
fingerprint=<API_KEY_FINGERPRINT>
key_file=/home/chartmuseum/.oci/oci.key
tenancy=<TENANCY_OCID>
region=<REGION>Enter your user and tenancy OCIDs, api key fingerprint and region value. The key_file value has to be “/home/chartmuseum/.oci/oci.key”.
Next, create the secret:
kubectl create secret generic chartmuseum-secret --from-file=config="/path/to/config" --from-file=key_file="/path/to/apikey.pem"Replace with the appropriate absolute or relative path to the config file and private api key.
Next, download the chartmuseum values file:
curl -o values.yaml https://raw.githubusercontent.com/helm/charts/master/stable/chartmuseum/values.yamlEdit the values.yaml file and replace the parameters as follows:
env:
open:
STORAGE: oracle
STORAGE_ORACLE_COMPARTMENTID:<COMPARTMENT_OCID>
STORAGE_ORACLE_BUCKET: chartmuseum
STORAGE_ORACLE_PREFIX: chartmuseum
DISABLE_API: false
AUTH_ANONYMOUS_GET: true
AUTH_REALM: chartmuseumexistingSecret: chartmuseum-auth
existingSecretMappings:
BASIC_AUTH_USER: user
BASIC_AUTH_PASS: passingress:
enabled: true
labels:
dns: "ocidns"annotations:
kubernetes.io/ingress.class: nginxhosts:
- name: chartmuseum.acme.com
path: /
tls: falseoracle:
secret:
enabled: true
name: chartmuseum-secret
config: config
key_file: key_file
We can now install chartmuseum:
helm install --name=chartmuseum -f values.yaml stable/chartmuseumWait for the pod to be running:
kubectl get pods -wNAME READY STATUS RESTARTS AGE
chartmuseum-chartmuseum-748c8dbbd8-7nctc 1/1 Running 0 5m20s
Verify that the ingress has been created:
kubectl get ingNAME HOSTS ADDRESS PORTS AGE
chartmuseum-chartmuseum chartmuseum.acme.com 80 7m9s
And that it maps to the chartmuseum service:
k describe ing chartmuseum-chartmuseum
Name: chartmuseum-chartmuseum
Namespace: default
Address:
Default backend: default-http-backend:80 ()
Rules:
Host Path Backends
---- ---- --------
chartmuseum.acme.com
/ chartmuseum-chartmuseum:8080 ()
Annotations:
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 6m20s nginx-ingress-controller Ingress default/chartmuseum-chartmuseum
Normal UPDATE 5m31s nginx-ingress-controller Ingress default/chartmuseum-chartmuseumFinally, you can verify whether you can reach chartmuseum publicly with your browser: http://chartmuseum.acme.com
Pushing a chart to chartmuseum
Install the helm push plugin:
helm plugin install https://github.com/chartmuseum/helm-pushNext, add the repo:
helm repo add --username curator --password password cm http://chartmuseum.acme.com/Let’s first create a basic chart which we can use to test:
helm create mycharthelm package mychart
Successfully packaged chart and saved it to: /home/opc/chart/mychart-0.1.0.tgz
Now, we can push the chart:
helm push mychart cmPushing mychart-0.1.0.tgz to cm...
Done.
If we search for mychart, we will only find the local copy:
helm search mychart
NAME CHART VERSION APP VERSION DESCRIPTION
local/mychart 0.1.0 1.0 A Helm chart for KubernetesLet’s do a repo update:
helm repo update cm
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "cm" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈And a new search:
helm search mychart
NAME CHART VERSION APP VERSION DESCRIPTION
cm/mychart 0.1.0 1.0 A Helm chart for Kubernetes
local/mychart 0.1.0 1.0 A Helm chart for KubernetesThe new chart appears.
Testing authentication
We could remove the repo and add it again without the username and password to test. However, we will use Postman and chartmuseum’s APIs to test. This is the behaviour we want:
GET: No Auth Required
POST: Auth Required
DELETE: Auth Required
You can see we are able to get a list of charts using GET without authentication.
However a DELETE or POST without authentication fails:
However, when we enter the credentials, the chart deletion succeeds:
If we now check OCI Object Storage, we can see our chart there:
Let’s Encrypt
Now that our chartmuseum is up and running, we want to also secure its usage by encrypting traffic. We will use Let’s Encrypt and cert-manager, “a add-on to automate the management and issuance of TLS certificates from various issuing sources.”
You can follow the guide to install cert-manager or read on.
First, create the CustomResourceDefinitions, a namespace for cert-manager and disable resource validation on the namespace:
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yamlkubectl create namespace cert-managerkubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
Add the Jetpack Helm repo and update:
helm repo add jetstack https://charts.jetstack.io
helm repo update jetstackInstall the cert-manager using its helm chart:
helm install --name cert-manager --namespace cert-manager \
--version v0.8.1 jetstack/cert-managerVerify the installation:
kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-776cd4f499-98vsh 1/1 Running 0 3h14m
cert-manager-cainjector-744b987848-pkk5s 1/1 Running 0 3h14m
cert-manager-webhook-645c7c4f5f-4mbjd 1/1 Running 0 3h14mTest the webhook works by creating a test-resources.yaml
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager-test
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: test-selfsigned
namespace: cert-manager-test
spec:
selfSigned: {}
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: selfsigned-cert
namespace: cert-manager-test
spec:
commonName: example.com
secretName: selfsigned-cert-tls
issuerRef:
name: test-selfsignedand create the test resources:
kubectl create -f test-resources.yamlCheck the status of the newly-created certificate:
kubectl describe certificate -n cert-manager-testName: selfsigned-cert
Namespace: cert-manager-test
Labels:
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-06-25T00:45:25Z
Generation: 1
Resource Version: 116448
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/cert-manager-test/certificates/selfsigned-cert
UID: 8576413b-96e2-11e9-b5fc-0a580aed39b8
Spec:
Common Name: example.com
Issuer Ref:
Name: test-selfsigned
Secret Name: selfsigned-cert-tls
Status:
Conditions:
Last Transition Time: 2019-06-25T00:45:26Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2019-09-23T00:45:26Z
Events:
You can delete the test resources after testing:
kubectl delete -f test-resources.yamlConfiguring Let’s Encrypt Issuers
For reference, you can follow the quick-start guide for using cert-manager with Nginx Ingress.
Let’s start with creating a staging issuer by creating a staging-issuer.yaml:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: cm-staging
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: your_email_address
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: cm-staging
# Enable the HTTP-01 challenge provider
http01: {}Replace your email address above.
Create the staging issuer:
kubectl create -f staging-issuer.yaml
issuer.certmanager.k8s.io/cm-staging createdRepeat the above but with for production by creating a production-issuer.yaml:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: cm-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: your_email_address
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: cm-prod
# Enable the HTTP-01 challenge provider
http01: {}As above, ensure you provide your email address and create the issuer:
kubectl create -f production-issuer.yamlCheck the status of the staging issuer:
kubectl describe issuer cm-stagingName: cm-staging
Namespace: default
Labels:
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Issuer
Metadata:
Creation Timestamp: 2019-06-25T03:28:16Z
Generation: 1
Resource Version: 143856
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/issuers/cm-staging
UID: 452908da-96f9-11e9-b5fc-0a580aed39b8
Spec:
Acme:
Email: your_email_address
Http 01:
Private Key Secret Ref:
Name: cm-staging
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Status:
Acme:
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/9718789
Conditions:
Last Transition Time: 2019-06-25T03:28:17Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events:
Enable TLS on chartmuseum
Edit your values.yaml file for chartmuseum and look for the Ingress section. The additional configuration is highlighted in bold:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
certmanager.k8s.io/issuer: "cm-staging"
certmanager.k8s.io/acme-challenge-type: http01
## Chartmuseum Ingress hostnames
## Must be provided if Ingress is enabled
##
hosts:
- name: chartmuseum.acme.com
path: /
tls: true
tlsSecret: cm-tlsUpgrade your helm chart:
helm upgrade chartmuseum stable/chartmuseum -f values.yamlRelease "chartmuseum" has been upgraded. Happy Helming!
Cert-manager will read the annotations and create a certificate:
kubectl get certificate
NAME READY SECRET AGE
cm-tls True cm-tls 45mTake a quick peak at the certificate:
kubectl describe certificate cm-tlsName: cm-tls
Namespace: default
Labels: app=chartmuseum
chart=chartmuseum-2.3.1
dns=ocidns
heritage=Tiller
release=chartmuseum
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-06-25T03:33:47Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: chartmuseum-chartmuseum
UID: d5a9cee1-9673-11e9-a790-0a580aed1020
Resource Version: 152567
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/cm-tls
UID: 0ac806f6-96fa-11e9-8836-0a580aed4b3e
Spec:
Acme:
Config:
Domains:
chartmuseum.acme.com
Http 01:
Ingress Class: nginx
Dns Names:
chartmuseum.acme.com
Issuer Ref:
Kind: Issuer
Name: cm-staging
Secret Name: cm-tls
Status:
Conditions:
Last Transition Time: 2019-06-25T04:19:27Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2019-09-23T03:19:27Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Cleanup 44m cert-manager Deleting old Order resource "cm-tls-3817114402"
Normal OrderCreated 43m (x2 over 44m) cert-manager Created Order resource "cm-tls-3433596774"
Normal OrderComplete 43m cert-manager Order "cm-tls-3433596774" completed successfully
Normal Cleanup 2m21s cert-manager Deleting old Order resource "cm-tls-3433596774"
Normal Generated 2m14s (x3 over 47m) cert-manager Generated new private key
Normal GenerateSelfSigned 2m14s (x3 over 47m) cert-manager Generated temporary self signed certificate
Normal OrderCreated 2m14s (x3 over 47m) cert-manager Created Order resource "cm-tls-3817114402"
Normal OrderComplete 2m13s (x2 over 47m) cert-manager Order "cm-tls-3817114402" completed successfully
Normal CertIssued 2m13s (x3 over 47m) cert-manager Certificate issued successfully
Once complete, cert-manager will have created a secret with the details of the certificate based on the secret used in the ingress resource. You can use the describe command as well to see some details:
kubectl describe secret cm-tls
Name: cm-tls
Namespace: default
Labels: certmanager.k8s.io/certificate-name=cm-tls
Annotations: certmanager.k8s.io/alt-names: chartmuseum.acme.com
certmanager.k8s.io/common-name: chartmuseum.acme.com
certmanager.k8s.io/ip-sans:
certmanager.k8s.io/issuer-kind: Issuer
certmanager.k8s.io/issuer-name: cm-staging
Type: kubernetes.io/tls
Data
====
ca.crt: 0 bytes
tls.crt: 3578 bytes
tls.key: 1679 bytesIf you access chartmuseum now, you’ll see a warning:
If you ignore the warning and go ahead anyway, you will be able to access chartmuseum except that now you will be accessing it over https. Your browser will also warn you that it’s added an exception to this site.
Edit the values.yaml for your chartmuseum again and this time change the issuer annotation to cm-prod:
certmanager.k8s.io/issuer: "cm-prod"Run a helm upgrade again:
helm upgrade chartmuseum stable/chartmuseum -f values.yamlRelease "chartmuseum" has been upgraded. Happy Helming!
and delete the secret:
kubectl delete secret cm-tlsThis will cause cert-manager to get a new certificate. You can verify this:
kubectl describe certificate cm-tls
.
.
.Normal Generated 33s (x4 over 61m) cert-manager Generated new private key
Normal GenerateSelfSigned 33s (x4 over 61m) cert-manager Generated temporary self signed certificate
Normal OrderCreated 33s (x4 over 58m) cert-manager Created Order resource "cm-tls-3433596774"
Normal CertIssued 31s (x5 over 61m) cert-manager Certificate issued successfully
Normal OrderComplete 31s (x3 over 57m) cert-manager Order "cm-tls-3433596774" completed successfully
If you now access chartmuseum, you will be able to access it with https and will not be prompted with the security warning/exception.
Conclusion
Chartmuseum enhances your CI/CD capabilities in a number of ways:
- host your helm charts privately and securely
- integrate with your CI/CD deployment tool chain and pipelines
- support multiple teams and organizations with multitenant capabilities
- use a variety of storage capabilities including local file system, Oracle Object Storage, OpenStack Object storage and others.
If you want to run it outside your Kubernetes cluster e.g. on a VM, you can follow this guide instead.
I hope you find this useful.
References:
Chartmuseum docs: https://chartmuseum.com/docs/
Chartmuseum helm package docs: https://github.com/helm/charts/tree/master/stable/chartmuseum
Cert-Manager installation guide: https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html
Cert-Manager with Nginx Ingress: https://docs.cert-manager.io/en/latest/tutorials/acme/quick-start/index.html
