CODEX

Reliable Kubernetes on a Raspberry Pi Cluster: Security

Scott Jones
Jan 18 · 6 min read
Photo by Miłosz Klinowski on Unsplash

With a fully working k3s cluster at our disposal, and access to dashboards of its contents we need to consider locking access down, especially when we want to deploy other (potentially insecure) apps to our cluster.

Part 1: Introduction
Part 2: The Foundations
Part 3: Storage
Part 4: Monitoring
Part 5: Security

The Problem

Everything we have deployed up to date is readily accessible to anyone with the URL. This isn’t so bad if it’s only on your local network (let’s ignore the fact that people can gain access to your local network for now), but as soon as you expose it to the internet you’re opening yourself up to a whole new can of worms. Let’s say you run a system such as Node-Red like I do. By default, it provides no authentication. Let’s then say you hook that up to your Home Assistant instance. That’s secure right? It forces a username and password on you to access it, or long-lived API tokens for things like Node-Red to access it. Hold on a minute though — what about this?

A possible flow in Node-Red

An anonymous user could set up a flow connecting to my preconfigured Home Assistant service (which coincidently supports auto-complete on entities) and upon seeing what entities I have, create themselves a one button click way of unlocking my front door. With some more work, you could also find out where I live, and also when the house is empty. I’m sure you can see why this needs to be locked down!

The (Catch-All) Solution

A very simple way to secure this is by putting everything behind some oAuth system. Since I have everything tied to my Google account anyway, Google OAuth is the perfect solution for this. This way, only I (and my family) will be able to access any of these systems meaning I’m no longer making the keys to my house publically available. Dashboard metrics are considerably less risky to expose, but let’s go back to the Traefik dashboard we exposed back in Part 2. Looking back at traefik-dashboard.yaml we have this:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: dashboardsecure
namespace: traefik
spec:
entryPoints:
- websecure
routes:
- match: Host(`traefik.yourdomainhere.com`)
kind: Rule
services:
- name: api@internal
kind: TraefikService
tls:
certResolver: cloudflare

The Caveat

Using Google’s OAuth system, it requires that we use a valid top-level domain for our SSO provider. This is good practice anyway since you have control over the DNS records and don’t have to edit host files everywhere. It is also a prerequisite to getting valid LetsEncrypt certificates for our systems. The rest of the article is working on the assumption you have this setup.

SSO Provider

First things first, we need to deploy an SSO provider to our cluster, to handle the authentication. We will put that in traefik-forward-auth.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik-sso
namespace: traefik
labels:
app: traefik-sso
spec:
selector:
matchLabels:
app: traefik-sso
template:
metadata:
labels:
name: traefik-sso
app: traefik-sso
spec:
containers:
- name: traefik-sso
image: thomseddon/traefik-forward-auth:2-arm
imagePullPolicy: Always
env:
- name: PROVIDERS_GOOGLE_CLIENT_ID
valueFrom:
secretKeyRef:
name: traefik-sso
key: clientid
- name: PROVIDERS_GOOGLE_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: traefik-sso
key: clientsecret
- name: SECRET
valueFrom:
secretKeyRef:
name: traefik-sso
key: secret
- name: COOKIE_DOMAIN
value: yourdomainhere.com
- name: AUTH_HOST
value: oauth.yourdomainhere.com
- name: INSECURE_COOKIE
value: "false"
- name: WHITELIST
value: <<your.email@here.com>>
- name: LOG_LEVEL
value: debug
ports:
- containerPort: 4181
---
kind: Service
apiVersion: v1
metadata:
name: traefik-sso
namespace: traefik
spec:
selector:
app: traefik-sso
ports:
- protocol: TCP
port: 4181
targetPort: 4181
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: sso
namespace: traefik
spec:
forwardAuth:
address: http://traefik-sso.traefik.svc.cluster.local:4181
authResponseHeaders:
- "X-Forwarded-User"
trustForwardHeader: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-sso
namespace: traefik
spec:
entryPoints:
- websecure
routes:
- match: Host(`oauth.yourdomainhere.com`)
kind: Rule
services:
- name: traefik-sso
port: 4181
middlewares:
- name: traefik-sso@kubernetescrd
tls:
certResolver: cloudflare

There is a lot going on in here, but its not as horrible as it looks. The image we use for our SSO takes all its configuration from environment variables so we need to set that up. There is an ingress route to our new service, which is something we have come across before. The new thing here is the middleware. This middleware leverages the forwardAuth feature of Traefik. At a high level, all we are doing is telling Traefik that when it runs this middleware, it needs to go to our service to check for authentication.

We cannot apply this manifest yet, because it relies on three secrets which we have yet to create — traefik sso secret, clientsecret, and clientid. To get that, we need to head over to the Google Developers Console. When you’re there, hit credentials on the side, and create credentials up top (You may need to create a project if you don’t already have one). The one we want is OAuth client ID

Creating new credentials

When you’re in there, you need to set a type (Web Application) a name (Not important, but let’s say Traefik OAuth) and a valid redirect URI (https://oauth.yourdomainhere.com/_oauth). The rest is not necessary.

Valid example setup

Upon creation, you will be given your client id and secret which you need for our provider (Note: The credentials in the image are throwaway and were never used)

Example OAuth client

Once you have these, we can now create our secrets in k3s

$ kubectl create secret generic traefik-sso \
--from-literal=clientid=XXX.apps.googleusercontent.com \
--from-literal=clientsecret=XXX \
--from-literal=secret="$(openssl rand -base64 128)" \
-n traefik

Finally, with everything in place, we can apply the manifest we created above

$ sudo kubectl apply -f traefik-forward-auth.yaml

Now when you go to your Traefik dashboard and take a look at the middlewares you will see a new one in there.

SSO middleware

So that should all be in place now, but nothing is using it. We need to finally apply it to one of our routes, such as our dashboard example. Let’s update traefik-dashboard.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: dashboardsecure
namespace: traefik
spec:
entryPoints:
- websecure
routes:
- match: Host(`traefik.yourdomainhere.com`)
kind: Rule
services:
- name: api@internal
kind: TraefikService
middlewares:
- name: traefik-sso@kubernetescrd

tls:
certResolver: cloudflare

Notice, the new middlewares section that has been added. This tells Traefik to use our new middleware.

Finally, apply it the usual way

$ sudo kubectl apply -f traefik-dashboard.yaml

The Moment of Truth

Now that’s all hooked up and running, go ahead and hit your dashboard at https://traefik.yourdomainhere.com. What you should see is the google account sign-in page.

Google sign in for Traefik

Drop your credentials in and you will find yourself back at your (now secured!) dashboard.

Congratulations! You can now secure any/all endpoints using the same OAuth and keep your web systems secure. As mentioned earlier in the article, the importance of this really cannot be understated. Even the simplest of dashboards could leak enough information that in the wrong hands serious damage could be done. Nevermind if you ever get to the point of exposing a mechanism of unlocking your front door to the world!

CodeX

Everything connected with Tech & Code

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store