--
I Want to Scale Stateful Containers Too
Using Amazon for everything feels wrong to me. A bit like the Rebels in Star Wars renting a Star Destroyer from Darth Vader rather than recruiting the Millennium falcon to fight the Empire.
“Raaaw, don’t run stateful apps in a container, Raaaw”. That is what currently stands for sound advice in the development/operations world. I hear it parroted over and over. I get it, running stateful applications in a container is hard. However, nobody should be paying you to spin up 1000 stateless containers to print hello world then click a button in Amazon for the hard stateful parts. You give yourself an out by saying you are following the guidelines of the “12 Factor App”. That tells you writing applications born in the cloud should attach stateful services separately. All you really need to own is dynamically scaling that amazing stateless hello world app you wrote using React and Node.js.
Amazon, MongoLabs, Rackspace, Microsoft; they all whisper sweet nothings in your ear. “Run along child, go play with Angular 2, Node.js, React; give Spring Boot, Rat Pack and Polymer a try. Leave the hard stateful parts to us.” You agree and start putting everything in Amazon’s cloud. They get you hooked on their drug. Then, when it is too late, you learn the real cost of “all your data are belong to us.”
Let me back up a minute and tell you why I am writing this and what I want (short version — I want Google’s Kubernetes, Docker, and Openshift to step up their game). I am working on a rewrite of a large scale application at my place of employment. We are fed up with the complexity that has grown inside a monolith and have decided to pursue a micro-services architecture. The promise of smaller more maintainable apps keeps me working all hours day and night.
This is how things progress. I learn Angular 1, Bootstrap, Grunt, the JBoss stack: REST services running in Wildfly’s JEE7 app server and I secure everything with a central OAuth/OpenID Connect provider to make it easy to share user credentials in a number of separate applications. Yes, I know JEE7 sucks and Angular 2 breaks backward compatibility with Angular 1 (Not the focus of this article). And yes, I was once a native javascript fanboy so I know frameworks like Angular are just awful too. I do not really think they are awful but I do think the slowness on mobile is. However, that is a conspiracy for another time (I think it stems from the unnecessary lack of memory and storage space on mobile devices so Google and Apple can sell cloud storage for your 20 billion baby pictures).
Back to my point. Anyway, I think I have one.
Inevitably, we realize reduced complexity in app development is traded for increased complexity in operations; all of these services and separate clients and databases need to be built, deployed and managed in a sane way.
I dive head first into the “devops” world. I learn straight Docker and use Docker-Compose to orchestrate running containers for app services, clients, and a single mongodb node. Plus, containers for etcd (name value storage for ip/ports), registrator (used to keep etcd in synch), nginx (load balancing services), and confd to reconfigure nginx dynamically as containers start and stop. “Yes, I am amazing!” At least that is what I think until I realize Docker-Compose falls very short of making the management of those containers easy. I do not attempt to scale the setup with Docker-Swarm. There are too many moving parts in the makeshift PAAS to justify selling to management. If the complexity is too much running containers on one node, throwing Docker Swarm into the mix I imagine to be a nightmare. At this point, my notion is that stateful applications, such as mongodb, should run in containers along side stateless clients and services. I did not know that makes me foolish. Like a naive dreamer, I thought the value of containers was to make your entire app easy to scale. I was unaware the community had already given an official name to their surrender (Backing Services).
With some Docker experience under my belt I take another look at the Docker PAAS landscape. I need something a bit more robust than Docker-Compose and Docker-Swarm. How do I choose. There is Kubernetes, Fleet, and Mesos. But wait. I do not use those directly? They are for the use of platform developers? Eventually I understand; they are the technology behind a number of other platforms meant for use by me (the developer). I need to choose my platform after I choose from the lower level technology. Kubernetes, being from Google, has a huge following, is open source and plays well with the Docker knowledge I already have. I choose Kubernetes. Now I need to choose a Kubernetes platform. I already use Redhat Enterprise Linux to run Docker containers and some of the JBoss stack so I choose Openshift Origin. I spend weeks learning Openshift. I spend hours communicating with Openshift developers and finally get everything I had running with Docker and Docker-Compose running on Openshift Origin. Except now I have a build and deployment piece, images automatically get pushed to a registry, I can dynamically scale up and down pods, easily, with a single line command and load balancing is baked into every service I create. Life is good. Except I am still stuck on scaling my stateful mongodb service. Every PAAS has templates for running a mongodb database. They all stop at the easy part and leave the hard part to us. If anyone has used Mongodb, they know the hard part is getting a “production ready” Mongodb instance. That means at least running a Mongodb replica set. For a truly production level configuration, a sharded cluster is required. It gets complicated quickly. There should be a template for doing that in someone’s platform. There is not. Right now, Openshift is blocked because Kubernetes needs to figure out how to attach separate persistent volumes to pods in the same service. They have made that a priority 2. It should be a priority 1. This makes me seriously consider switching. Tutum is bought by Docker. Ok, maybe I go back to the Docker camp and use Tutum. No, they also stop at the easy part and only have an example for a single development Mongodb node. They are still trying to integrate with Flocker which means they are nowhere near solving the problem. Alright, I consider leaving the Docker container world entirely for stateful services because I find an Ansible template to do what I want. However, it is out of date and only works with Redhat 6. That is the problem with Ansible. Ansible playbooks depend on their environment. Docker images are portable and I know they will run. I want a tool that has a repository of templates for making formations of very hard things easy. If that existed I would not need Amazon to do every hard thing for me. I can mostly get what I want on Amazon. But look at the prices:
They get you hooked for free and the next level is $1496 per month… wtf! Mongolabs is little better. I do not understand why everything is becoming a service I have to pay exorbitant prices for. We are so eager to say we do not have time for anything and hand everything to cloud providers. Keep in mind, platform as a service providers should have templates for hard to set up stateful services like Mongodb. Rather than use templates from IAAS providers, I should be able to use someone else’s platform running on my chosen cloud/IAAS provider(call it what you like), like Amazon, to put pressure on them to lower prices. If they know it is easy to leave, their prices will go down. Let’s take back some of our ability to do hard things wherever we choose so that running a database does not cost … wait what, $21,430 per month? Please, tell me I am crazy. Tell me why I am wrong. I sincerely want to know. So many products are inches from being able to compete with Amazon and give choice back to developers whether they want to run on Amazon or retain a bit more freedom (perhaps even run in another cloud or local cloud environment).
Meanwhile, Amazon announces competing components daily. Amazon has API Gateways, Cloud Formations to spin up almost any stack or service, CodePipeline for continuous deliver, Load Balancing; you name it they have it.
Now, you might be wondering, what is the problem? Just go with Amazon, everyone is doing it. “You aren’t cool unless you’re using Amazon.” I do work for a large organization that can afford to run everything on Amazon (maybe — some would disagree). However, I also work intensely on a project outside work that has no VC funding. I have learned to “do more with less”. Using Amazon for everything feels wrong to me. A bit like the Rebels in Star Wars renting a Star Destroyer from Darth Vader rather than recruiting the Millennium falcon to fight the Empire.
I have also found, often, these services have less functionality than expected. Mongodb is limited to only storing files less than 400kb in Amazon. What if I want to use GridFS to store content larger than that? I have to spend time finding a work around.
I am all for “the cloud”, but the cloud does not necessarily mean all Amazon all the time. Amazon is actively blurring the line between IAAS and PAAS. Use Amazon’s infrastructure but give yourself the flexibility to move by using a portable PAAS like Kubernetes.
If not, quite possibly, every systems architect job becomes obsolete. Let me give you a glimpse of the future.
Example conversation between Bob the architect and management
Management: Bob, how do you suggest this application be written?
Bob: Amazon
Management: But what about…
Bob: Amazon
Management: Yeah but…
Bob: Amazon
Management: Genius Bob, you’ve done it again. I have what I need, let me run this up the flagpole.
