An Interrogation of Metaparticle and Abstraction Layers

Ryan Clair
Dec 15, 2017 · 7 min read

While at KubeCon + Cloud Native Conference (2017), I watched Brendan Burns go on the main stage and present Metaparticle, a truly novel approach to getting application code running on Kubernetes.

Before I tell you what it does, you have to understand the reason why it was created. And to do that, you need to understand a bit about the workflow that someone typically has to go through to get their application into a Kubernetes cluster.

Application Code => Dockerfile (YAML) => Kubernetes (YAML)

The above workflow is fairly common, and the challenge with it is two fold:

  • You have to learn Docker and Kubernetes’s specific syntax and tooling (in addition to your Application’s language).
  • This can be prone to human errors (example: exposing a network port, which has to be specified in all three files)

The Big Idea that Metaparticle brings to the table is: “Why not import a standard library into your Application that handles these steps for you, using syntax that you’re already familiar with as a developer?”

I’ve got to say, the demo was slick. Not only can Metaparticle handle the container creation, deploy it onto Kubernetes, tell the number of replica’s it needs spun up, and automatically add it to a loadbalancer… but it can also offload some of the synchronization challenges like locking and master election between distributed replicas WITHOUT needing to use docker build or kubectl commands.

Did I say slick? I meant badass 🤘

But as I was walking back with the ~3,999 other fellow attendees towards the slew of sessions, it got me thinking: Was this the right abstraction layer? That’s effectively what Metaparticle is, a new abstraction. Nearly a week later, as I was writing my Reflection on KubeCon post, this question popped up in my head again and I decided to explore it.

Abstraction Layer

So first things first, what’s an abstraction layer?

I’ll use wiki’s explanation as it’s generally right 75% of the time (I just made that statistic up):

In computing, an abstraction layer or abstraction level is a way of hiding the implementation details of a particular set of functionality, allowing the separation of concerns to facilitate interoperability and platform independence. (…) A layer is on top of another, because it depends on it. Every layer can exist without the layers above it, and requires the layers below it to function.(source)

Abstraction layers are literally everywhere.

Here’s a fun hiring exercise (or test for your friends in IT): Ask what all went into serving up a webpage to their device. The amount of steps/layers between you hitting enter and getting the webpage is truly mind boggling. The sheer amount of engineering hours that went into streaming the below video to your eyeballs makes me really appreciate and respect all those [engineers/scientist/developers/mathematicians] who put in blood, sweat, and tears for our own entertainment.

But I’m digressing.

As a self proclaimed Buzzword Tinker, there’s a few principles I believe in, one of which is “Every decision you ever make has a tradeoff”. I like finding the tradeoffs, because it prepares you for what could happen (and ideally you’d want to mitigate your risk as much as possible when it inevitably does happen). An informed decision is the best decision you can make. I say all of this because 1) I don’t want to disrespect the work that Brendan Burns (and all those who are contributing) put into Metaparticle, and 2) this is the method to my madness, and I encourage a healthy (logical, God please logical) debate on my points below.

With that said, let’s start with the obvious tradeoff.

Leaky Abstractions

Joel Spolksy wrote a great article on leaky abstractions and he came up with the Law of Leaky Abstractions:

All non-trivial abstractions, to some degree, are leaky.

But what is a leaky abstraction you say? Well, remember what an abstraction layer does? It attempts to hide the finer details. But sometimes those details surface back up in unexpected, strange, downright despicable ways — especially if you have no knowledge of the underlying complexity you’ve abstracted.

The most basic example I can give is your car. You put a key in and turn it (or push a button). You expect the car to turn on. The key-turn is an abstraction layer between you and a pretty fascinating series of events that lead to a series of mini explosions. And it works 99% of the time. But what about that 1%, when it laughs at you? That’s a leak.

For Metaparticle, I suspect a lot of engineering effort will need to go into preventing leaks. What happens when you hit “run” or “compile” on your application and it-no-workie? Or what if a specific implementation of how Kubernetes does things causes unpredictable things to happen when you run it? Yeah, that’s vague, but it’s difficult for me to come up with an example.

“But Ryan, that’s a cop-out. All abstraction layers have to deal with this.”

Fair point. Moving on!

Making Changes

Due to the abstraction layer being implemented in the application code, that means making any changes to the supporting infrastructure (and as of this writing, that’s limited to replica’s and load balancers) has to be done through the application code. Otherwise you’ll have a conflict of state (read: if you’d gone outside of the application and used kubectl, there’d be differences and you wouldn’t want that [trust me]). This means an infrastructure change will trigger a CI/CD pipeline event, arguably, needlessly.

This very fact was brought up in the GitOps session at KubeCon when an audience member had asked why they were recommending having two different repos (one for application code and one for Ops stuff).

Now you could argue that one could come up with some method of ignoring the CI pipeline being kicked off (a flag perhaps?) or that you’d set it such that these changes would be outliers (ala use autoscaling instead of named replica counts). However, I’m not sure of metaparticle’s roadmap, and whether or not other parts of the infrastructure will be exposed to the library.

But it’s not just changes to the application’s infrastructure you’d have to worry about, you’ll also inevitably have changes to Kubernetes itself… which will inevitably break certain versions of the library, or expose new features which require new versions of the library to be leverage across potentially a slew of languages (not to mention a recompile/redeploy of all your applications). Either way, that’s a whole new level of dependency hell you’ll need to manage. And as a great friend of mine once put it:

Would you rather be holding a hundred pennies or four quarters?

(different context but applicable here)

Separation of Duties

But surprisingly my biggest hangup with Metaparticle is the, IMHO, flawed argument that the developer should be interfacing with Kubernetes so… closely?

There’s a good chance I’m alone on this one — which I’d blame my lack of sample size of real world users — but I’ve got this idea stuck in my head that there should be a separation of duty. Developers and Operators/SRE’s/DevOps’ers (I really hate that word). The idea that these are two different roles that would attract two different kinds of people: Those who like creating applications based on business demands, and those who like to keep it running, securely (including being the detective in the murder mystery that which is microservices — Observability FTW). To some extent, it’s two different lenses that require different mindsets (which is insanely hard for one person to juggle both at any kind of depth).

Maybe this is a flawed assumption on my part, but putting the provisioning logic into the code, as oppose to something external, is like giving enough rope to hang oneself. I don’t have a single shred of empirical data to back this gut feel, or even a hypothetical scenario to exemplify my hesitation.

But you want to know the weird thing? I’d be OK with this strictly being limited to the packaging (read: “turn this code into a docker container and send it to this registry).

Am I crazy/alone on this? I’m open for a discussion in the comments below. 😄

Don’t get me wrong, I think the panacea is something close to Metaparticle. I can imagine a world where there is only developers. Where there’s a system that just runs the code for you, that scales it for you, does capacity management for you, tells you exactly where the issues are, and does everything in it’s power to hide the flaws in your logic in a magically frictionless way.

Isn’t that right HAL?

The Perfect Abstraction Layer

But this level of specific AI (well, more likely a few specific AI’s in concert with each other) is still a ways off. At least for the time being.

Assuming that’s the case (hey, Google and OpenAI have proven us wrong before), then what would be the right abstraction layer?

I suppose that’s the million dollar question that we’re all trying to figure out, but Git does seem to be a natural abstraction layer between the developer and the operator. I suspect the perfect adversary to Metaparticle would be a system that handles all the glue (that Metaparticle is replacing) for you from a Git commit, automagically. A system that would look into the application code itself, realize it needs port 80 exposed, and create that container with port 80 opened, and deploy it onto the a cluster for you based on some Git logic (branch name, tag, etc). Regardless of the type of application.

So quarters, not pennies.

Which I have to say, I keep going back to GitOps. I think William Denniss and alexis richardson are really on to something there.

In Summation

Is Metaparticle the right abstraction layer? At least for now, this hack doesn’t think so. But it’s WAY too early to call this one. And, you know, I’m one troll with an internet connection.

But I’ll be closely watching this project, as I’m excited to see where Brendan and the community will take this. I’ve certainly been wrong before and, as always, I’m open to someone changing my mind.

Ryan Clair

Written by

A Technology Buzzword Tinker. Works at VMware. Opinions are my own… Basically another guy with an internet connection.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade