Installing Rancher 2 HA Cluster with Let’s Encrypt
All you need to create HA cluster with Rancher 2 on Debian/Ubuntu or anything else (I’m using Debian 9 as an example). You’ll need basic understanding of Kubernetes.
Rancher is Open Source enterprise cluster management software, more information on rancher.com
Installing Rancher now as easy as possible, there’s official manual how to do that. It’s very detailed, except how to actually use Let’s Encrypt certificates.
Getting ready with TLS
Rancher 2 now requires SSL certificate in place in order to operate. I want to use cert-manager to manage Let’s Encrypt certificate for my cluster. This will require temp self-signed SSL certificate.
- Generate core ca and key:
$ openssl genrsa -out ca-key.pem 2048$ openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj “/CN=kube-ca”
2. Generate ingress key
$ openssl genrsa -out ingress-key.pem 2048$ openssl req -new -key ingress-key.pem -out ingress.csr -subj "/CN=rancher.example.com"
3. Sign new ingress key with ca certificate:
$ openssl x509 -req -in ingress.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ingress.pem -days 7200
4. Encode new certificates into Base64
$ cat ingress.pem | base64
$ cat ingress-key.pem | base64
$ cat ca.pem | base64
Now you’ve got everything you need to start building your cluster.
Standing cluster
I’m using Hetzner with Debian 9 instances. I’ve decided to build 3 node cluster.
Upgrading Debian nodes (always use latest packages) and installing curl:
$ apt update && apt upgrade -y && apt install -y curl
We’ll need a docker 17.03. I’ve tried everything up to 18.03. I’ve managed to install kubernetes on them and it was working fine. 18.03 is a bit bugged. Officially it’s fully tested on 17.03.
curl https://releases.rancher.com/install-docker/17.03.sh | sh
Once all 3 nodes patched and have docker running, it’s time to use RKE. It’s like any other tool is good to stand Kubernetes cluster, supports HA options and described in yaml file.
After adding base64 encoded certificates and keys into cluster.yaml file, it’ll look something like this:
nodes:
- address: NODE1_IP
user: docker
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: NODE2_IP
user: docker
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: NODE3_IP
user: docker
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsaaddons: |-
---
kind: Namespace
apiVersion: v1
metadata:
name: cattle-system
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: cattle-admin
namespace: cattle-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cattle-crb
namespace: cattle-system
subjects:
- kind: ServiceAccount
name: cattle-admin
namespace: cattle-system
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
name: cattle-keys-ingress
namespace: cattle-system
type: Opaque
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNyekNDQVpjQ0NRQzRCTkhuMEF1
...
eU90WVBHdlZHT3k1MU1rc0s5awpmOEZuUnEwa3BycXY0a1RHMVVYaXJabmIxQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K # cat ingress.pem | base64
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBMGhL
...
VHJFaGE1RENUTgpMa0ZkUG5qNkV2RkhlTk92VkxkNmc2YklaSjU5eVN4QXJTaHo5cjZERFIEtFWS0tLS0tCg== # cat ingress-key.pem | base64
---
apiVersion: v1
kind: Secret
metadata:
name: cattle-keys-server
namespace: cattle-system
type: Opaque
data:
cacerts.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNvRENDQVlnQ0NRQ04wc25WeHJJ
...
2JWenVsaFZXZm5IQ2J5NmNlaGlVdwpROVNqc1E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== # cat ca.pem | base64
---
apiVersion: v1
kind: Service
metadata:
namespace: cattle-system
name: cattle-service
labels:
app: cattle
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
selector:
app: cattle
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: cattle-system
name: cattle-ingress-http
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
spec:
rules:
- host: rancher.example.com
http:
paths:
- backend:
serviceName: cattle-service
servicePort: 80
tls:
- secretName: cattle-keys-ingress
hosts:
- rancher.example.com
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
namespace: cattle-system
name: cattle
spec:
replicas: 1
template:
metadata:
labels:
app: cattle
spec:
serviceAccountName: cattle-admin
containers:
- image: rancher/rancher:latest
imagePullPolicy: Always
name: cattle-server
ports:
- containerPort: 80
protocol: TCP
- containerPort: 443
protocol: TCP
volumeMounts:
- mountPath: /etc/rancher/ssl
name: cattle-keys-volume
readOnly: true
volumes:
- name: cattle-keys-volume
secret:
defaultMode: 420
secretName: cattle-keys-server
The only things left todo, is to start process (assuming you’ve installed rke tools):
$ rke up --config cluster.yml
Process will take couple minutes, once it’s finished, you’ll have running cluster with rancher 2 installed in it.
I’m not using Load Balancer node (nginx proxy). This will introduce single failure point. I’m using Cloudflare as DNS provider:
Works really well.
Cluster is ready and waiting you to login: https://rancher.example.com
It’s running on self-signed certificate, now it’s the time to fix that problem. We need to enable Helm stable package repository and wait a bit (5–10 mins packages to download)
Syncing packages will take some time, once it’s finished, you should be able to find cert-manager. You can do this without using Rancher Catalog, but I think Catalog is what makes Rancher so attractive.
Once it’s installed. not is the time to remove self-signed certificate.
I’m using DNS flow to validate domain name, feel free to use HTTP. More documentation here. You’ll need CloudFlare API key base64 encoded in Kubernetes secret
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-dns
namespace: cert-manager
type: Opaque
data:
api-key: VGhpcyBpcyB0ZXN0LCBwbGVhc2UgaWdub3JlCg==
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt
namespace: cattle-system
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: email@example.com
privateKeySecretRef:
name: letsencrypt
dns01:
providers:
- name: cf-dns
cloudflare:
email: cloudflare@email.com
apiKeySecretRef:
name: cloudflare-dns
key: api-key
During installation, RKE created kube config file kube_config_cluster.yaml, we need it in order to connect to our cluster
$ export KUBECONFIG=./kube_config_cluster.yaml
Just to confirm that cluster is ready to go and we have access:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
NODE1_IP Ready controlplane,etcd,worker 20m v1.10.3
NODE2_IP Ready controlplane,etcd,worker 20m v1.10.3
NODE3_IP Ready controlplane,etcd,worker 20m v1.10.3$ kubectl create -f issuer.yamlsecret "cloudflare-dns" created
clusterissuer "letsencrypt" created
We should be able to see that it’s READY to go:
$ kubectl describe clusterissuers letsencrypt...
Conditions:
Last Transition Time: 2018-06-20T16:00:05Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Once this is done, we can request certificate from Let’sencrypt
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: cluster-cert
namespace: cattle-system
spec:
secretName: cluster-cert-tls
issuerRef:
kind: ClusterIssuer
name: letsencrypt
commonName: rancher.example.com
dnsNames:
- rancher.example.com
acme:
config:
- dns01:
provider: cf-dns
domains:
- rancher.example.com
It will take some time to get certificate issued, you can monitor process:
$ kubectl describe certificate -n cattle-system cluster-cert
Please make sure you’ve tested process in sandbox, before requesting certificates in production. Let’s Encrypt limit to 5 duplicated requests a week, you can get banned and have to wait a week. More information here
Good result:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateOrder 2m cert-manager Created new ACME order, attempting validation...
Normal DomainVerified 9s cert-manager Domain "rancher.example.com" verified with "dns-01" validation
Normal IssueCert 8s cert-manager Issuing certificate...
Normal CertObtained 6s cert-manager Obtained certificate from ACME server
Normal CertIssued 6s cert-manager Certificate issued successfully
The only things what’s left todo, is to link this certificate to ingress facing outside and update ranchers deployment
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: cattle-system
name: cattle-ingress-http
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
spec:
rules:
- host: rancher.example.com
http:
paths:
- backend:
serviceName: cattle-service
servicePort: 80
tls:
- secretName: cluster-cert-tls
hosts:
- rancher.example.com
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
namespace: cattle-system
name: cattle
spec:
replicas: 1
template:
metadata:
labels:
app: cattle
spec:
serviceAccountName: cattle-admin
containers:
- image: rancher/rancher:latest
imagePullPolicy: Always
name: cattle-server
ports:
- containerPort: 80
protocol: TCP
- containerPort: 443
protocol: TCP
Apply configuration change
$ kubectl apply -f remove-and-relax.yamlingress "cattle-ingress-http" configured
deployment "cattle" configured
Your cluster is now using Let’s Encrypt certificates running on 3 node HA setup
Thanks