Cloud Foundry and Kubernetes: A brief history of CF/K8s

Over the last year or so, one of the most discussed topics in the cloud-related open source communities is the relation between Cloud Foundry (CF) and Kubernetes (K8s). Questions being asked are along the lines of “Are they killing each other”, “Are they mutually exclusive” or “Do they blend”?

When you look at commercially available offerings today, it almost seems like there are no touchpoints — the stacks are separated, they both run on various IaaSes, integration is little to none and even if there is, it’s more on the “CF and K8s have a shared service catalog” level than deeper in the guts of the systems.

Now — superficially looking at it this doesn’t make a lot of sense. Granted, the technologies have some degree of overlap, but also are really complimentary in a bunch of key areas — the following table tries to summarise:

   +------+--------------------+-----------+----------------+
   |      | Container Runtime  | IaaS/PaaS | Dev Experience | 
   +------+--------------------+-----------+----------------+
   | CF   |       Garden       |   PaaS    |     great      |   
   +------+--------------------+-----------+----------------+
   | K8s  |  containerd/crio/  |   IaaS+   |     poor       | 
   |      |       docker       |           |                |    
   +------+--------------------+-----------+----------------+

The “Container Runtime” is definitely a huge overlap, and it’s based on different implementations (although both are based on runc/containerd eventually). That has some effects like K8s introducing a new feature (like sidecars, which require container groups), and CF starting to run after that to implement a similar functionality, or potentially vice versa.
However, on the other hand, the developer experience is pretty poor in K8s and generally K8s is considered more an “IaaS+” than a “PaaS”.

But if K8s is more an “IaaS+” rather than a PaaS, wouldn’t it be possible to run CF on top of K8s? I mean one of CF’s value propositions is to be “IaaS” agnostic, so it should be possible, right? So we tried.

As you might or might not know, CF usually comes with a deployment and lifecycle management technology called BOSH. BOSH usually takes care of setting up a CF environment, and it abstracts the underlying infrastructure away through a mechanism called “Cloud Provider Interfaces” (CPI). Think of it as you saying “I want a VM”, and the CPI translating this into the appropriate call on your infrastructure provider, e.g. AWS, IBM, Google Cloud or VMware.

So the first attempt was to stay entirely within that world, and that means writing such a CPI for Kubernetes. It turned out that at least our initial attempts did fail.

Listen to Sandy and Monika’s talk to hear the specifics, but to summarise it in two sentences: The biggest issue is that a CPI assumes the IaaS layer to be “dumb” and the CPI is in control, and that is not necessarily true for K8s. K8s is an intelligent “IaaS+”, and in scenarios like a CF component dies and needs to be resurrected — who’s in charge? BOSH or K8s? The answer isn’t easy to figure out, and we called it the “split brain orchestrator” issue. It became the death sentence for this exercise.

Luckily, there are two other projects provided by our friends in SUSE: Fissile and SCF. They take a more radical approach and basically remove the BOSH runtime aspects entirely. This approach definitely has some consequences around how you build and operate your CF, but the nice thing about it is that it gives a really native way to deploy to K8s. So long story short: we tried it, tweaked it a bit, deployed it and are happy so far! If you want to read more on the details going forward, i.e. how ideally we want to bring this back together with BOSH, read here.

But even with all of the above, there is still a little flaw: As you can see from the table above, CF comes with a container runtime called Garden and Fissile converts that into a Docker image today, so in the end you have a CF App running in a Garden container running in a Docker Container running in a Kubernetes POD. Sounds ugly, eh?

Everybody with a little experience in nested virtualisation probably wants to stop reading here, but really you shouldn’t. Yes, conceptually this is “nested containers”. But, in reality, that’s not so bad since “nested containers” are not really nested, they are really peers, and, as you learned in your containers-101, a container is just a “cage” for an operating system process. You cannot “nest” operating system processes. Therefore, you cannot “nest” containers either. From a call hierarchy, this is a parent-child relation, but in reality the containers run as peers.

Although I just said it’s not so bad, it isn’t ideal either. Not so much in terms of performance, but actually in terms of user / operator experience. If I do a ‘cf push myApp’ and then I kubectl to my K8s cluster, I’d expect to see a POD called “myApp” there. With the above, I’d see a POD called “garden”, and if I dig into that I eventually find “myApp”. So there is at least one layer too many — can’t we just combine CF and put a K8s “behind it”, essentially swapping the container runtime? Wouldn’t that solve all of our problems and give us the best of both worlds? It’s definitely harder to do, but worth a try.

So we tried. In essence, we would like to be able to deploy Cloud Foundry applications using other container schedulers. This enables us to use the benefits of these schedulers and more closely integrate with them while retaining the Cloud Foundry user experience (‘cf push’) and abstractions. Our prototype code currently resides here if you want to take a look — and right below you find a little screen recording of it in action!

Our “Cube” prototype consists of four main parts:

• Sink provides a convergence loop that pulls desired apps from the CF Cloud Controller and creates corresponding Kubernetes resources. It relies on the registry to serve OCI images for droplets, and OPI to abstract the communication with K8s.
• Registry is an OCI registry vending image based on CF droplets. Eventually this would be nice to move in to Bits-service.
• St8ge implements staging by running Kubernetes/OPI one-off tasks
• OPI or the “orchestrator provider interface” provides a declarative abstraction over multiple schedulers, inspired by Diego’s LRP/Task model and Bosh’s CPI concept.

Needless to say, this work represents the beginning of a journey, and a journey that will be full of rocks for sure, however, we fully believe that we can make this successful and really bridge the gap between CF and K8s!

Hope you enjoyed reading, stay tuned!