Kubernetes recipe: how to setup baremetal cluster (1.8.1) with Kubespray (2.3.0) and deploy Ingress Controller with Letsencrypt support on ScaleWay 2017-10-30

UPDATE 2017–11–18: Read new version of guide (with GlusterFS Native Storage Service)

1. Create new server on ScaleWay

Create new server on ScaleWay

Remember your Public IP (X.X.X.X) and Private IP (Y.Y.Y.Y)

Test connection to the server

ssh root@X.X.X.X
exit

2. Install requirements on local computer

virtualenv venv
source venv/bin/activate

pip install ansible netaddr

3. Install kubernetes cluster

Clone Kubespray

git clone https://github.com/kubernetes-incubator/kubespray.git
cd kubespray
git checkout -b tags/v2.3.0

Create ansible inventory file (replace X.X.X.X and Y.Y.Y.Y)

cat <<EOF > inventory/inventory.cfg
# Change ansible_ssh_host to your Public IP, ip to Private IP
master ansible_ssh_host=X.X.X.X ip=Y.Y.Y.Y
[kube-master]
master
[etcd]
master
[kube-node]
master
[k8s-cluster:children]
kube-node
kube-master
EOF

Change kubernetes definitions

vim ./inventory/group_vars/k8s-cluster.yml
-kube_network_plugin: calico
+kube_network_plugin: flannel

Run kubernetes installation

ansible-playbook -u root -i inventory/inventory.cfg cluster.yml -b -v

Check installation

ssh root@X.X.X.X
kubectl get all --all-namespaces
exit

4. Deploy Ingress Controller (nginx-ingress)

Login to the master

ssh root@X.X.X.X

Run mandatory commands (from https://github.com/kubernetes/ingress-nginx/blob/master/deploy/README.md#mandatory-commands)

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
| kubectl apply -f -

Install with RBAC roles (from https://github.com/kubernetes/ingress-nginx/blob/master/deploy/README.md#install-with-rbac-roles)

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \
| kubectl apply -f -

Deploy Service (change Y.Y.Y.Y to your Private IP)

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
spec:
type:
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app: ingress-nginx
externalIPs:
- Y.Y.Y.Y
EOF

Check that nginx-ingress-controller and default-http-backend is running

kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-66b447d9cf-4b8d9 1/1 Running 0 2m
nginx-ingress-controller-c5b844d96-vr6bk 1/1 Running 0 1m

Check that nginx-ingress-controller was binded to 80, 443 port

netstat -tupln
tcp        0      0 Y.Y.Y.Y:80           0.0.0.0:*               LISTEN      20771/hyperkube
tcp 0 0 Y.Y.Y.Y:443 0.0.0.0:* LISTEN 20771/hyperkube

Check default-http-backend

curl X.X.X.X
default backend - 404

4. Deploy Let’sencrypt Service for Ingress

Deploy kube-lego (change lego.email)

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Namespace
metadata:
name: kube-lego
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: ingress-secret-admin
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs:
- get
- watch
- list
- create
- update
- patch
- apiGroups: [""]
resources: ["services"]
verbs:
- get
- create
- apiGroups: ["extensions"]
resources: ["ingresses"]
verbs:
- get
- watch
- list
- create
- update
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kube-lego
roleRef:
kind: ClusterRole
name: ingress-secret-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: default
namespace: kube-lego
---
apiVersion: v1
metadata:
name: kube-lego
namespace: kube-lego
data:
# modify this to specify your address
lego.email: "your@email.com"
# configure letsencrypt's production api
lego.url: "https://acme-v01.api.letsencrypt.org/directory"
kind: ConfigMap
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kube-lego
namespace: kube-lego
spec:
replicas: 1
template:
metadata:
labels:
app: kube-lego
spec:
containers:
- name: kube-lego
image: jetstack/kube-lego:0.1.5
imagePullPolicy: Always
ports:
- containerPort: 8080
env:
- name: LEGO_EMAIL
valueFrom:
configMapKeyRef:
name: kube-lego
key: lego.email
- name: LEGO_URL
valueFrom:
configMapKeyRef:
name: kube-lego
key: lego.url
- name: LEGO_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LEGO_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 1
EOF

Check kube-lego is running

kubectl get pods -n kube-lego
NAME                         READY     STATUS    RESTARTS   AGE
kube-lego-674567867b-5wmfh 1/1 Running 0 25s

5. Deploy example application

Add an A record in your DNS and wait about three hours for your changes to propagate

Type: A
Name: echo.example.com (your domain name)
Value: X.X.X.X (Public IP)

Deploy echoserver (replace echo.example.com with your domain name)

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Namespace
metadata:
name: echoserver
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserver
namespace: echoserver
spec:
replicas: 1
template:
metadata:
labels:
app: echoserver
spec:
containers:
- image: gcr.io/google_containers/echoserver:1.0
imagePullPolicy: Always
name: echoserver
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: echoserver
namespace: echoserver
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
selector:
app: echoserver
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: echoserver
namespace: echoserver
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
# CHANGE ME
- echo.example.com
secretName: echoserver-tls
rules:
# CHANGE ME
- host: echo.example.com
http:
paths:
- path: /
backend:
serviceName: echoserver
servicePort: 80
EOF

Check for letsencrypt certificate

kubectl get pods -n kube-lego
NAME                         READY     STATUS    RESTARTS   AGE
kube-lego-674567867b-5wmfh 1/1 Running 0 25
kubectl logs -f kube-lego-674567867b-5wmfh -n kube-lego

Check ingress resources

kubectl get ing --all-namespaces
NAMESPACE    NAME              HOSTS           ADDRESS      PORTS     AGE
echoserver echoserver echo.example.com Y.Y.Y.Y 80, 443 5m
kube-lego kube-lego-nginx echo.example.com Y.Y.Y.Y 80 5m

Test deployment

curl https://echo.example.com