Kubernetes at Mercado Libre
At Mercado Libre, managing over 30,000 microservices and supporting 16,000 developers requires a robust and scalable solution. To address these challenges, we adopted Kubernetes (K8s) as the core engine of our internal platform, Fury. This article explores how Kubernetes streamlined our infrastructure management, accelerated software delivery, and improved cost efficiency, enabling our developers to focus on innovation and product creation.
Fury structure
Since this article focuses on implementing Kubernetes within the Fury platform, it’s important to have a basic understanding of Fury to better grasp the topics covered. We recommend a preliminary reading on the subject.
In brief, Fury is an internal developer platform (IDP) composed of cloud applications and services. It provides a web interface and a command-line interface (CLI), allowing developers to create and manage resources. Additionally, the Cloud and Platform teams use a back office to administer and monitor the platform and manage its operations.
Fury and its abstractions
In a simplified way, Fury is an abstraction layer developed internally at Mercado Libre. It serves as a bridge between third-party services, such as Amazon Web Services (AWS), Google Cloud Platform (GCP), Datadog, and others, and user applications. Regardless of the provider or partner used, developers generally interact with services through Software Development Kits (SDKs), while Fury’s APIs communicate with the providers.
This means that applications do not communicate directly with cloud services. There is an anti-corruption layer that prevents lock-in, facilitates version updates, and enables switching to similar services without requiring code changes. This abstraction is essential to support the company’s multi-cloud strategy, ensuring flexibility and better cost and security management.
The genesis: the compute engine
The compute engine is the Fury component responsible for managing user workloads. When Fury was created in 2015, Kubernetes was still in its early stages and wasn’t mature enough to handle the volume of workloads and technical needs that Mercado Libre required. Thus, adopting instance-based computing services from cloud providers like AWS EC2 and Google Compute Engine was the most natural choice.
The first computing model developed was called Standard. It serves as an interface between the platform and cloud providers. This model receives platform commands, such as creating or scaling instances, and executes these operations on the providers.
While the simplified diagram below represents this model, its complexity is much greater. One example is a component called Little Monster (LM), part of the Standard module, which manages the atomicity of distributed transactions. LM requests instances from the cloud provider and ensures they are ready for use, handling necessary waits and pooling during the creation and initialization process.
The diagram shows that this model already operates across multiple regions and cloud providers. Since its inception, it has been designed for this purpose, so all auto-scaling management was developed internally as part of the compute module without relying on AWS Auto Scaling Groups (ASGs) or equivalent features from other clouds. This decision was motivated by the need to create a multi-vendor solution, thus justifying the development of this abstraction layer.
The Standard module has been a reliable component of Fury, operating successfully for many years. It has enabled Mercado Libre to scale and address significant business challenges. However, over time, it presented two challenges that needed solution:
- Low efficiency: Each replica of an application requires a computing instance, such as an EC2, creating a 1:1 relationship. Even though we employed strategies like reservations, spot instances, savings plans, and ARM instances with Graviton for cost efficiency, there was still room for improvement.
- Limited utilization of native cloud provider resources: Due to the developed abstractions, the module did not optimally leverage native cloud provider resources and functionalities, limiting the full potential of the capacities offered.
Alternative study
We conducted an extensive study to find a more efficient solution, evaluating various technologies and approaches. Initially, we explored AWS Batch for on-demand resource activation, but it didn’t meet all our needs. We also considered AWS Lambda, but its limitations for long-running tasks made it unsuitable. We looked into container-based solutions like ECS and Cloud Run as well, but their complexity and lack of necessary features led us to dismiss them. After testing Kubernetes (EKS and GKE) and Nomad, we ultimately discarded Nomad and decided to move forward with managed Kubernetes solutions, EKS and GKE, as they best fit our needs.
This evaluation and its findings are available in this post.
Current implementation: compute service with EKS and GKE
After months of planning and development, we arrived at the current architectural model, named Serverless. Unlike the common concept of serverless computing, which typically refers to Lambda functions, we adopted this name to illustrate the complete lack of direct server knowledge and management required from developers — they interact exclusively with Fury’s abstraction layer, embracing a server-free perspective. The platform, in turn, manages resources transparently, with Serverless serving as a unique differentiator that reflects this philosophy of full automation, thus facilitating adoption.
This evolution introduced the Serverless Module, replacing the old Standard Module while retaining the Little Monster (LM) component, ensuring operations’ atomicity. The most notable change lies in the communication layer between the platform and the cloud providers: the new compute module interacts with a solution called Cluster API. It’s worth noting that this Cluster API is not the same as the one defined by Kubernetes; in our context, it abstracts the Kubernetes control plane, acting as a translation layer between various Kubernetes versions and the compute module.
Cluster API: an efficient abstraction layer
The Cluster API was developed to address the growing complexity of managing multiple simultaneous Kubernetes versions and to handle APIs that are frequently deprecated over time. This component is designed to minimize the impact of these changes on the compute module, leaving the lifecycle management of Kubernetes versions solely to the Cluster API. This approach also allows us to replace Kubernetes in the future, if necessary, with minimal migration effort to another orchestrator.
Serverless module: control and scalability
Unlike the Cluster API, which acts as a wrapper for the control plane, the Serverless Module abstracts the computational “shadings” introduced by clusters. Clusters fragmented what was once a single regional data plane into multiple ones. The Serverless Module is responsible for adapting this new concept to the platform, handling additional complexities such as inferring the most suitable cluster for a given workload.
Scalability and controlled upgrades
With over 130,000 instances, the current architecture is already segmented by regions, providers, and other divisions based on criticality and workload type. These segmentations are crucial in our approach to controlled upgrades, especially during data and control plane component updates. When an upgrade is required, the process begins with clusters of lower criticality and test environments to ensure stability. Only after careful validation are changes gradually promoted to higher criticality clusters. This strategy minimizes risks and ensures that service continuity is not compromised.
Deployment flow
Now that we’ve explored our engine’s architecture, let’s examine the application deployment flow within the Fury platform. We adopted a GitOps approach to deploy our engine’s internal components; however, this topic deserves a separate discussion.
There are different deployment strategies within Fury, such as All-in, Blue-Green, and Canary, each catering to specific developer scenarios. However, there are two platform-specific strategies: Safe deployment and Migration:
- Safe deployment: This strategy is similar to Blue-Green deployment but takes a more gradual and cautious approach. It continuously monitors metrics and makes automated rollback decisions or adjustment recommendations, ensuring a safe and controlled process.
- Migration: A broader strategy that allows workloads to be moved between vendors, regions, or even between different computational environments (from Standard to Serverless and vice versa), depending on resilience and cost needs.
Deployment flow details
In the deployment architecture, Fury triggers the Serverless Module, which initiates the process by requesting the creation of workloads. This request passes through the Cluster API via an Application Load Balancer (Ingress ALB), which distributes the traffic and creates everything necessary to route the applications properly.
Next, the Cluster API configures the required Kubernetes components, such as ReplicaSets and Horizontal Pod Autoscalers (HPAs). We chose to use ReplicaSets instead of Kubernetes Deployments because our goal is to ensure the proper number of replicas without needing additional resources provided by Deployments, as such strategies were already part of the platform.
Monitoring and asynchronous control
Once the workloads are created, the Serverless Module — specifically its Little Monster (LM) component — performs continuous monitoring and handles the asynchronous requests sent to the cluster. The Serverless API Controller (a Kubernetes controller) extends K8S by observing the behavior of the pods and nodes directly in the cluster, keeping the platform informed about any real-time changes. This controller reconciles the state of the cluster with an external state through the “watchers” of these objects, notifying the Serverless Module so that the platform can respond and dynamically adapt.
This structure allows Fury to perform deployments efficiently and robustly, monitor the state of clusters and workloads, and respond to real-time changes with resilience and security.
Deployment flow — Blue-Green
When applying a Blue-Green deployment in Fury, the process follows the same basic flow described earlier but includes some essential specifics.
In this scenario, the current set of workloads in production (the Blue version) is maintained while a new set of workloads (the Green version) is created in parallel. This new set can be deployed in the same cluster or, depending on shard balancing strategies, in a different cluster.
The creation flow is the same as the standard deployment: the platform activates the Serverless Module, which then requests the Cluster API via the Ingress ALB. This step sets up all the necessary Kubernetes components, such as ReplicaSets and HPAs, replicating the current infrastructure for the new set of workloads.
Once the Blue-Green deployment is completed, the platform automatically updates its internal state, recording all changes. This way, we maintain continuous and detailed control over both versions of the workloads (Blue and Green), allowing for a safe and organized transition to the new version once the necessary checks are completed.
Network and traffic swap
A critical point during the Blue-Green deployment is the progressive traffic redirection from the current version to the candidate. To achieve this, we employ a specific approach that ensures comprehensive network visibility and rapid synchronization between components.
Every pod inside our clusters have unique and routable IPs from the VPC instead of VIPs (Virtual IP). This allows each pod to have direct network visibility, facilitating communication with other services and reducing the need for entry points per cluster.
Since we do not use third-party networking products for routing, we developed an internal component called the Mesh Controller. This controller plays a crucial role in the traffic swap process, responding to changes generated by the control plane. As soon as an event is detected, the Mesh Controller synchronizes the state and notifies the control plane API of the platform, that is, our Service Mesh.
This notification automatically updates all Envoy sidecars deployed with the applications. As a result, the routes are adjusted so that the Envoys redirect traffic to the new candidate, completing the traffic transition to the latest version and ensuring that the platform and applications are always aware of the updated route.
Conclusion
The journey of Fury at Mercado Libre exemplifies how technological evolution and constant innovation determine the ability to overcome challenges and drive efficiency at scale. Built on a robust Kubernetes foundation, Fury has transcended operational complexities, offering developers an intuitive, secure, and scalable environment. By abstracting infrastructure management and optimizing the deployment flow, Fury allows our engineers to focus on creating real value through code while also enabling the company to adopt strategies like multi-cloud and make strategic decisions with rapid financial impact. This approach positions us to meet current demands and anticipate and overcome future challenges, reaffirming our commitment to an exceptional development experience and continuous evolution.
Acknowledgments
This article was written with the collaboration of Marcos Antonio Souza Pinheiro and Marcelo Cordeiro De Quadros.
