Warehouses as code
How much impact on our customers we will have if one of our warehouses gets a network outage?
Daily we handle more than 26 tones of food, we employ, on average, on each warehouse about 500 people and we feed between 2800–3200 families. Daily.
Considering the above numbers, the answer to the previous question scare us.
Today we run all our software in cloud and because of it we invested in communications by having several redundant lines per warehouse, but the business impact of having a connection dependant software running on the warehouses is sufficiently big to invest a bit more on minimising that risk as much as possible.
It was clear that we had to deploy the applications somewhere inside the warehouse. For us that we were born in the cloud, this means that we should look into solutions with bare-metal and to manage everything ourselves, or at least more than with the fancy and “easy to use” that cloud providers offers nowadays.
Did someone say hybrid cloud?
We did a proof of concept on how viable it is for us to run Kubernetes on our own warehouse’s servers. We wrote down those experiences in this article. The conclusion was that the technology was ready to deploy Kubernetes cluster(s) per warehouse.
We realised that the industry was moving fast on this front too; VMWare bought Pivotal, SUSE bought Rancher, Google was moving its cards with GKE on prem via Anthos and RedHat was working also in the Kubernetes Enterprise on its OpenShift platform. So we wanted to check what those players were doing before starting our own development. In the end we sell food and technology is just a tool to help us to do it efficiently and at scale.
We talked with pretty much all the above mentioned players to get more details about their solutions. We chose Tanzu in the end, the solution provided by vmware, since Mercadona has already invested a lot in the vmware software for its own data centres and for the servers located in each warehouse.
In early 2020 we sat down to start defining the road map for the team on the integration and develop what we called internally project mercanetes (a word name joining Mercadona and Kubernetes, we even created a logo!).
One of the ideas we had was to treat the warehouse just like deployment objects in Kubernetes. We wanted to define as much of our infrastructure in code as possible so we could, theoretically, create and destroy fully working warehouses on demand.
It wasn’t our main problem to solve, to have on demand warehouses, but we wanted to accomplish something bigger than just deploy a Kubernetes cluster and plug it into our current Spinnaker deployment orchestrator.
We wanted to be able to deploy all the apps and configs that belongs to a warehouse in a cluster and abstract the location of the Kubernetes cluster as much as possible. It won’t matter anymore if the destination cluster for our deployments is hosted in public cloud or in our bare metal.
Let’s think that we can have some structured way to define what a warehouse is as code. We started with a simple JSON file were we will simply drop few metadata and some applications.
The above json is an example of what a warehouse looks like. It includes:
hive_id: This is the id of our warehouse, which we call hive internally.is_active: Perhaps we are building a new cluster but is not yet into service. We can control it.metadata: values defining cluster-specific data.applications: Here is the key, now we can assign which applications and their variables, including kubernetes manifests and environment vars, we want to be present in the warehouse.
With the above information we faced several challenges:
- How to create clusters on demand.
- How to populate those clusters with all the needed things.
- How to deploy the final apps into those clusters without the need to redo our deployment pipelines.
Creation of the cluster and preparations
Tanzu relies on the project Cluster API, that opens the door to modify it and adapt it to our needs.
What we did was to look under the hood of the tkg (stands for tanzu kubernetes grid cli tool) command line tool to understand how it generates the final yaml manifests that are passed to the management cluster. After a bit of hassling with cluster api resources we were able to define a customised provider, extending it with our modifications, so we can install our “stuff”. We called this piece genesis,which is just a bunch of scripts encapsulated on docker images and cronjobs that manage the prerequisites of a functional cluster, such as:
- Secrets: we do not have yet a proper management for secrets so we handle them manually.
- Service accounts: both Kubernetes services accounts, roles, cluster roles and the associated bindings and also cloud providers service accounts. For example the service accounts to access into Google Cloud resources.
- Creation of namespaces
Once this first part is ready we have as a result a vanilla Kubernetes cluster but provisioned enough that unblocks the next phase: The deployment of the applications.
How to get the applications into the cluster(s)
Up until now we have been happily deploying at a rate of ~500 deploys per week with our pipeline that consists of few steps between Jenkins and Spinnaker.
Jenkins is mainly responsiblefor testing and building artifacts, then Spinnaker is called to deploy those artefacts into the various Kubernetes clusters.
Today we deploy our applications to 3 warehouses (hives), we have 2 more coming on the near future, 1–2 years. In addition to the hive we are exploring the possibility of using the physical stores (~1600 across the country) to prepare as well. Given this scenario we potentially might end up managing few hundred clusters.
Continuing with our push based deployment strategy (Spinnaker push the new changes towards the cluster) means that we need to potentially deploy in parallel to hundreds of clusters. The complexity to keep it all in sync we think is too much to continue down this path.
Instead we looked into a pull based method. We wanted to have something that performs a reconciliation in the same way Kubernetes ensures that if you set a replica set with 2 replicas, there are 2. A system where we could specify which applications we would like to have deployed and the system itself will make it happen when is ready to host them.
Taking the json presented above we built a couple of pieces that performs that job. One of them is what we call Hiver Metadata and the counter part is Hiver Walker.
- Metadata becomes the source of truth when it comes to what each cluster should have installed. It also contains all the variables that each application needs.
- Walker is the piece living in the mercanetes project cluster that performs that reconciliation. We wanted to start lean and so far it only asks Metadata “what’s up for me?” via REST api call and then it auto applies all the kubernetes manifests that are returned. We see as the Sheriff that watches the applied Kubernetes manifests. Like Walker, Texas Ranger!
For Metadata we chose Python and the framework Django with DRF (Django Rest Framework) to build a quick admin panel. In the future we will have visualisation of the variables but for now it does the job.
For the Walker component we chose Go as language. The logic for the Walker is quite simple so we looked into a compact binary to deploy. Go is, in our opinion, excellent for small tools to be placed into a Docker image.
Current status
At the time of writing this article we have already migrated most of the applications of the VLC1 hive to the Mercanetes Kubernetes cluster. We have been serving the applications from within the hive for over a month.
We have been performing tests such as disconnecting the network and unplugging electrical wires randomly — sort of manual chaos monkey — to verify our initial ideas.
So far so good.
In addition to that we are now able to create from the ground up a cluster and populate it until the applications are ready to use in less than 2h without human intervention. For us this is a big step forward and opens a lot of interesting paths as we can start optimising that time for a very complete solution of disaster recovery.
Future work
We have several ideas on how to continue this path that unveiled in front of us. One is the pursue of a better disaster recovery solution. We would like to be able to run all the applications of a hive on premises one day and the next somewhere else without interruption so we can automate that based on some checks. Having an extremely resilient system.
Another path we want to explore is the multiple on demand environments. If we are able to create and recreate full warehouses ready for production, why not creating them for testing purposes? Imagine to create one and run it with the previous day’s data as simulation for a realistic system/integration tests.
Final words
I would like to thank the whole team as this would not have been possible without the implication of extremely talented engineers.
If you are curious about what you read do not hesitate to contact us. In addition if you find it interesting and you would like to help us. We are recruiting!
— -
