How to mount SMB share behind OpenVPN to Pod on GKE

A client asked me to prepare an SMB mount behind a VPN to access the integration partner's data. An easy way is to deploy VM, install OpenVPN and deploy the application there, but almost everything is running on GKE then I wanted to explore what options I do have there.

VPN Setup

The easy part was the VPN, the client provided me with OpenVPN config and I just did 2 simple steps:

  • create a VM with IP Forwarding ON and static internal IP (terraformed)
  • run Ansible to that machine and install and setup OpenVPN
  • create a route in GCP to single static IP where the SMB lives (terraformed)

In the config of OpenVPN I used an UP and DOWN script to make sure the routing works as expected

UP
#!/bin/bash
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o ${1} -j MASQUERADE
DOWN
#!/bin/bash
iptables -t nat -D POSTROUTING -o ${1} -j MASQUERADE

The last part was in GCP to add a new route and I did it via Terraform where I just used the previously created IP but you can use for example gcloud:

gcloud compute routes create VPN --destination-range = <your-range> --next-ho-address = <my-static-ip>

I do recommend you add description and tags for better resource management and reporting.

Now I had a working VPN tunnel from any host in my network to that SMB share. To secure this solution I want to re-work it to using network tags and allow traffic only from tagged nodes in my cluster (network).

Kubernetes here we go!

The first and main problem: How to mount it?

  • The first idea was to install https://github.com/kubernetes-csi/csi-driver-smb but it failed on custom client settings
  • Another one was to mount to the second container in POD and share access via {emty_dir} but that's impossible as you will basically re-mount the path in the SMB container and APP container will not see the data
  • Mount directly to the container where the application lives

I decided to go with the third option as the easiest one but it was not like that. I will show you the whole YAML file and explain the most critical parts:

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
container.apparmor.security.beta.kubernetes.io/example-app: unconfined
name: example-app
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
nodeSelector:
pool_name: g1_s_ubuntu
tolerations:
- key: "dedicated"
operator: "Equal"
value: "ubuntu"
effect: "NoSchedule"
containers:
- image: <image>
name: example-app
securityContext:
privileged: true
capabilities:
add:
- CAP_SYS_ADMIN
- DAC_READ_SEARCH
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "mount -t cifs -o username=Space,password=$SMB_PASS //<IP>/data /mnt/external"]
preStop:
exec:
command: ["/bin/sh","-c","umount /mnt/external"]
  • Make sure that no profile is loaded in AppArmor via annotation
  • you can't use default ContainerOS images! You will need to create an Ubuntu node-pool and in my case, I used taints and tolerations to make sure only the designed workload will run on Ubuntu nodes everything else will stay on the primary node pool with Container OS
  • You will need to add two capabilities to your container CAP_SYS_ADMIN and DAC_READ_SEARCH and privileged mode to allow your container to load kernel modules and manipulate with the network. If you're not familiar with capabilities you can find more here.
  • The final step was to mount the share and for that, I used the lifecycle hooks where I just added all parameters needed you can see a simple example. In the container, you need to have installed cifs-utils via apt.

And everything was cool until I tried to run it! I went into error where google was useless, why it worked from VM and not from GKE cluster? I found the solution — you need to allow ports 445 and 139 and you already know how.

The biggest side-effect of this setup is that the application needs to run in the container as root because no other user have the privilege to mount. There might be a better option but I needed to deliver this asap then I did this tradeoff and I will explore the security options later. As for now, I'm happy that it's at least separated on a dedicated node.

TODO:

  • use network tags to allow VPN <> Ubuntu node pool only
  • secure the Container to do not run it as root

Do you have a better solution? Let me know I will be more than happy to explore and re-deploy this application.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tomáš Papež

Tomáš Papež

I'm passionate about all things Cloud, management, leadership and currently digging into data world.