The Road to K8s/vSphere Integration

Kevin Monroe
Aug 21, 2018 · 7 min read

Background

Recently, Juju began supporting cloud-native features via “integrator” charms (e.g.: aws-integrator, gcp-integrator, openstack-integrator). These allow charms to request things like persistent storage from a cloud provider without having to shuffle your super-secret credentials around to all the applications in your deployment.

The way an integrator charm works is simple: you entrust it (and only it) with your Juju credentials for a particular cloud and then relate it to charms that want to make cloud-native requests. The integrator will ensure appropriate roles are created, submit a request to the cloud, and then notify the requesting charm that enough data is available to start using the new resource.

Lately I’ve been testing Canonical Kubernetes (CDK) on the VMware vSphere platform — to Juju, vSphere is supported like any other cloud. I really needed persistent storage for my pods and thought, “it sure would be nice if there was a vsphere-integrator that I could use for this.” So I wrote one.

In this post, I’ll share how the vsphere-integrator came to be, and more importantly, how others that want native vSphere integration with charms can use it.


Last Things First

If I’ve learned anything from watching videos like “how do I quickly turn off the water to my toilet?”, it’s that people want results first.

Here are the source repos that enable vSphere integration with charms:

And here’s the deployable integrator charm:

Using the integrator is straightforward. First, bootstrap a vSphere cloud if you haven’t already done so:

Then, deploy vsphere-integrator and trust it with your vSphere credentials:

Note: juju trust works in Juju 2.4 and above. If you’re using an earlier version, or prefer to see other ways to set credentials on the integrator, have a look at the integrator’s readme.

At this point, the integrator is waiting to handle requests from related charms. Since this is so new, the only workloads that support the vsphere-integration interface are kubernetes-master and kubernetes-worker. If you’d like to enable other workloads, check out the interface docs to see what charm modifications are needed.

Let’s see the integrator in action with a Kubernetes workload.


K8s + vSphere, better together

  • vSphere cloud bootstrapped with Juju 2.4 or greater. Here’s the doc link mentioned above for help with this.
  • This work is based on the recent K8s 1.12 release; without a doubt, it will not work on earlier K8s releases. Welcome to the edge.
  • In vSphere, we need to set disk.enableUUID = true as a VM Option on our VMs so that the K8s vSphere provider knows how to map persistent volumes to machines. You can do this for all VMs by using a VM template, or do it per machine. See the vSphere docs for details.

We need to deploy and relate the vsphere-integrator charm with k8s. Here’s a hunk of yaml that we can overlay to do just that:

Save the above to overlay.yaml and deploy like this (Note: the same overlay will work for both canonical-kubernetes and kubernetes-core):

Note: for those that just skimmed the prereqs, you need to set disk.enableUUID = true as a VM Option on our VMs so that the K8s vSphere provider knows how to map persistent volumes to machines. You can do it for the kubernetes-master and -worker machines that you just deployed, but you’ll have to turn them off and on again for the VM Option to take effect. See the vSphere docs for details.

Once the deployment settles, juju status should look similar to this:

Status after deployment

With deployment complete, let’s define a Storage Class and Persistent Volume Claim so our pods can dynamically request Persistent Volumes from vSphere:

Let’s make sure those were created successfully:

Next, create a pod (busybox in this case) that wants a pv based on the above claim:

And finally, get a microphone, drop it, and tell your friends how your busybox pod has all the persistent megabytes:

Note: if the busybox hostname cannot be resolved, you may get something like Error from server: error dialing backend: no such host. Add an entry with <ip> <hostname> to /etc/hosts and try the command again.

Read on for details on what just happened!


Behind the curtain

The interface and charm parts of vsphere-integrator were super simple to implement. In fact, I just forked the openstack-integrator and interface-openstack-integration repos and did a big ‘ol insensitive sed /openstack/vsphere/g to get the PoC going. The most significant changes were getting the language right (an OpenStack project roughly maps to a vSphere datacenter, etc).

At the end of the day, all an integrator charm does is ensure roles are present and a requestor can make a request. The roles part is a no-op since a vSphere credential is an admin for your cluster; trusting vsphere-integrator means it can do anything an admin can do. Once trusted, the vsphere-integrator simply tells your K8s charms that it can provide enough details for them to start making requests.

The harder part was on the K8s side. If you’ve ever searched for K8s/vSphere integration guides, you know the frustration of figuring out whether a cloud config option is valid for your particular version of K8s. As mentioned earlier, all these bits assume you’re on K8s 1.12 or greater. Why? Because the vSphere provider changed how it deals with UUIDs between 1.10 and 1.12(ish). Valid UUIDs are required for the provider to initialize itself and give consistent disk mapping to pods that need persistent volumes.

So, implementation step 1 was to piece together docs from VMware, as well as a few diffs from the good folks at Rancher to get the kube-apiserver and kube-controller-manager to use the vSphere provider.

When the vsphere-integrator is ready (that is, it has been trusted with your credentials), the kubernetes-master charm will generate a cloud-config that looks like this:

K8s-master will use that config when starting kube-apiserver and kube-controller-manager, resulting in processes that look like this:

Step 2 was to get the kubelets on board. This was shockingly simple, once I realized you could set a--provider-id as a kubelet arg. This alters your K8s nodes with a providerID that matches the UUID that the vSphere provider needs. The end result is a node with a ProviderID that looks like this:

K8s-worker will then set appropriate args so kubelet uses the vSphere cloud provider:

Step 3 was to write all this down and make a PR for the k8s charms. This has been merged, and new K8s charms with this functionality are available in the charm store.


Takeaway

This post is very specific to K8s/vSphere; if this doesn’t match your use case, I hope you at least have a better understanding of “integrator” charms and how they can help enable cloud-native features in your deployments.

If you did make your way here for K8s/vSphere specifics, rejoice! Deploy vsphere-integrator alongside CDK, and then figure out what to do for the rest of the week since all the heavy lifting is done.

Find me (kwmonroe) on Freenode IRC in #juju if you want to chat more about integrating vSphere with your Juju workloads. Thanks for reading!

Thanks to Tim Van Steenburgh

Kevin Monroe

Written by

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade