Kubernetes recipe: how to setup baremetal cluster (1.8.1) with Kubespray (2.3.0) and deploy Ingress Controller with Letsencrypt support on ScaleWay 2017-10-30

UPDATE 2017–11–18: Read new version of guide (with GlusterFS Native Storage Service)

1. Create new server on ScaleWay

Create new server on ScaleWay

Remember your Public IP (X.X.X.X) and Private IP (Y.Y.Y.Y)

Test connection to the server

ssh root@X.X.X.X
exit

2. Install requirements on local computer

virtualenv venv
source venv/bin/activate

pip install ansible netaddr

3. Install kubernetes cluster

Clone Kubespray

git clone https://github.com/kubernetes-incubator/kubespray.git
cd kubespray
git checkout -b tags/v2.3.0

Create ansible inventory file (replace X.X.X.X and Y.Y.Y.Y)

cat <<EOF > inventory/inventory.cfg
# Change ansible_ssh_host to your Public IP, ip to Private IP
master ansible_ssh_host=X.X.X.X ip=Y.Y.Y.Y
[kube-master]
master
[etcd]
master
[kube-node]
master
[k8s-cluster:children]
kube-node
kube-master
EOF

Change kubernetes definitions

vim ./inventory/group_vars/k8s-cluster.yml
-kube_network_plugin: calico
+kube_network_plugin: flannel

Run kubernetes installation

ansible-playbook -u root -i inventory/inventory.cfg cluster.yml -b -v

Check installation

ssh root@X.X.X.X
kubectl get all --all-namespaces
exit

4. Deploy Ingress Controller (nginx-ingress)

Login to the master

ssh root@X.X.X.X

Run mandatory commands (from https://github.com/kubernetes/ingress-nginx/blob/master/deploy/README.md#mandatory-commands)

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
    | kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
    | kubectl apply -f -

Install with RBAC roles (from https://github.com/kubernetes/ingress-nginx/blob/master/deploy/README.md#install-with-rbac-roles)

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml \
    | kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \
    | kubectl apply -f -

Deploy Service (change Y.Y.Y.Y to your Private IP)

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  type:
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app: ingress-nginx
  externalIPs:
  - Y.Y.Y.Y
EOF

Check that nginx-ingress-controller and default-http-backend is running

kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-66b447d9cf-4b8d9 1/1 Running 0 2m
nginx-ingress-controller-c5b844d96-vr6bk 1/1 Running 0 1m

Check that nginx-ingress-controller was binded to 80, 443 port

netstat -tupln
tcp        0      0 Y.Y.Y.Y:80           0.0.0.0:*               LISTEN      20771/hyperkube
tcp        0      0 Y.Y.Y.Y:443          0.0.0.0:*               LISTEN      20771/hyperkube

Check default-http-backend

curl X.X.X.X
default backend - 404

4. Deploy Let’sencrypt Service for Ingress

Deploy kube-lego (change lego.email)

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Namespace
metadata:
    name: kube-lego
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: ingress-secret-admin
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs:
  - get
  - watch
  - list
  - create
  - update
  - patch
- apiGroups: [""]
  resources: ["services"]
  verbs:
  - get
  - create
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs:
  - get
  - watch
  - list
  - create
  - update
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kube-lego
roleRef:
  kind: ClusterRole
  name: ingress-secret-admin
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-lego
---
apiVersion: v1
metadata:
  name: kube-lego
  namespace: kube-lego
data:
  # modify this to specify your address
  lego.email: "your@email.com"
  # configure letsencrypt's production api
  lego.url: "https://acme-v01.api.letsencrypt.org/directory"
kind: ConfigMap
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-lego
  namespace: kube-lego
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-lego
    spec:
      containers:
      - name: kube-lego
        image: jetstack/kube-lego:0.1.5
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: LEGO_EMAIL
          valueFrom:
            configMapKeyRef:
              name: kube-lego
              key: lego.email
        - name: LEGO_URL
          valueFrom:
            configMapKeyRef:
              name: kube-lego
              key: lego.url
        - name: LEGO_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: LEGO_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 1
EOF

Check kube-lego is running

kubectl get pods -n kube-lego
NAME                         READY     STATUS    RESTARTS   AGE
kube-lego-674567867b-5wmfh   1/1       Running   0          25s

5. Deploy example application

Add an A record in your DNS and wait about three hours for your changes to propagate

Type: A
Name: echo.example.com (your domain name)
Value: X.X.X.X (Public IP)

Deploy echoserver (replace echo.example.com with your domain name)

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Namespace
metadata:
  name: echoserver
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: echoserver
  namespace: echoserver
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.0
        imagePullPolicy: Always
        name: echoserver
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
  namespace: echoserver
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: echoserver
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echoserver
  namespace: echoserver
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    # CHANGE ME
    - echo.example.com
    secretName: echoserver-tls
  rules:
  # CHANGE ME
  - host: echo.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: echoserver
          servicePort: 80
EOF

Check for letsencrypt certificate

kubectl get pods -n kube-lego
NAME                         READY     STATUS    RESTARTS   AGE
kube-lego-674567867b-5wmfh   1/1       Running   0          25
kubectl logs -f kube-lego-674567867b-5wmfh -n kube-lego

Check ingress resources

kubectl get ing --all-namespaces
NAMESPACE    NAME              HOSTS           ADDRESS      PORTS     AGE
echoserver   echoserver        echo.example.com   Y.Y.Y.Y   80, 443   5m
kube-lego    kube-lego-nginx   echo.example.com   Y.Y.Y.Y   80        5m

Test deployment

curl https://echo.example.com