Safe but flexible: the DevEx-based code management culture at Mercado Libre

As the largest e-commerce platform in Latin America, our engineering teams span multiple and diverse cultures, backgrounds and goals. However, as a company, we want to empower them to share a common DNA. Therefore, achieving a Unified Development Experience (DevEx) across our organization is essential. Join us as we explore our platform-based code management culture, which fosters a highly secure and productive source code, simplifying convergence of different workloads in a consistent and intuitive manner.

In the last 13 years, our technology team has experienced substantial growth, expanding from 200 to 15,000 people (from 2011 to 2024). Also, our complexity, architecture, and source code have grown substantially.

Our big numbers are impressive: more than 26,000 microservices, 25,000 pull requests opened per week, and 19,000 daily deployments in our internal developer platform: Fury. You can read our previous article where we deep dived into our development story and technology evolution.

Safe but flexible: efficient, compliant and focused ownership

Fury started in 2015 and it’s an ever-growing platform that we continue working on. There are almost 1,000 developers (7% of our whole engineering team) working to make Fury even better. It is our backbone, allowing us to have a unified way of working and a world-class Developer Experience.

To start working on Fury, all we need is a microservice, which we refer to as an application: a core entity to address our business needs. It is composed of a GitHub repository, a maintainer team, compute resources (if applicable), and data and/or communication services. This simplifies a lot just to keep our main focus on the code management culture.

To create an application, we need an idea, an architecture design and, most importantly, flexibility (a key priority of our Developer Experience) to choose the right combination of application type and programming language, always prioritizing to solve our needs in the best and cost-effective way.

Flexibility brings complexity

Today, Fury supports 10 different types of applications (including web applications backend or frontend, mobile, libraries, ML, etc.) and 11 different programming languages. So, as you see, there are a lot of possible combinations. You might wonder how we keep our minds sane with all this flexibility?

Application type and programming languages are not the only variables that bring complexity to the equation. To solve an engineering problem at Mercado Libre, we have the ability to choose from almost anything we need to make it real. From the team’s methodology, to architecture design, and even the best practices in the industry, including selecting the right branching model for that application.

Just imagine 15,000 people working in teams developing and maintaining 26,000 applications every day. Each application with different branching models, application type and programming languages. All that combination creates a really complex scenario to deal with. So, our mission is to reduce all that complexity by thinking about abstractions, capabilities. The key point here is how to build a real DevEx to solve that challenge.

Unified DevEx: different teams, different goals, one way of working

What is DevEx? For us, it includes all aspects of a developer interaction with tools, libraries, frameworks, documentation, and community support throughout the software development lifecycle. DevEx aims to empower developers, enabling them to build high-quality software with ease and efficiency.

At Fury, our aim is to offer a Unified DevEx for developers across all projects and teams. A seamless and cohesive experience is ensured by providing daily access to the required codebase. The ability to efficiently code, build and deliver software is recognized as equally important.

Despite the simultaneous work on diverse projects involving various languages, such as mobile applications, libraries, and web applications, absolute mastery of every combination is not expected. Our primary goal, however, remains the same — the efficient delivery of top-quality software solutions. By focusing on this goal and reducing the technical complexities of each language and application type, we maintain a shared DNA and a cohesive culture, achieving a Unified DevEx across all teams.

Let’s go deep on those three pillars that build our Unified DevEx.

Source code AuthN/AuthZ

In our large-scale GitHub organization, managing access to our codebase involves intricate dynamics. Therefore, authentication and authorization processes are crucial as they address two essential queries: who has the permission to access the code and what is their purpose for accessing.

Addressing the challenges of managing codebase access calls for a robust, scalable, and adaptable strategy, especially in the face of business growth. Embracing GitHub’s versatile authentication features effectively addresses these challenges. For enhanced codebase security, we’ve implemented the GitHub IP allowlist restrictions allowing authorized access. Additionally, we use the Enterprise Managed Users (EMU) feature, ensuring comprehensive management of our GitHub users’ lifecycle, from creation and invitation to eventual removal from the organization.

AuthZ based on teams

We’re revolutionizing authorization practices. By having 26,000 GitHub repositories and 15,000 developers, we need to think carefully about how all collaborators access the code.

Assuming that each user requires access to all repositories poses a potential security risk. After all, who can juggle our entire organization simultaneously? To address this challenge efficiently, we’ve leveraged GitHub teams to reflect the organizational structure of Mercado Libre.

We employ a strategy where a team of, for example, 100 developers has access only to their specific 200 repositories. Although this enhances security, focusing solely on team-specific access could hinder broader collaborative efforts. Therefore, we’ve crafted a strategy to facilitate both access and collaboration across all company projects. Our overarching goal is to promote seamless teamwork across the entire organization.

AuthZ based on visibility and pushing inner sourcing

Promoting a Unified DevEx in our extensive GitHub organization relies on standardizing solutions across all repositories. To overcome challenges and implement enhancements, we have numerous specialists well-versed in diverse technologies. By creating specific GitHub teams for each technology, we enable these experts to provide valuable guidance to our entire engineering team. Their authority extends to making recommendations such as the need to update SDKs or discontinue the use of certain libraries, thereby enforcing best practices company-wide.

Promoting widespread collaboration is crucial for us, and to facilitate this, we have implemented varying levels of access and visibility:

• Public repositories serve as our platform to contribute to the open source community.
• Internal repositories encourage intra-company collaboration and are accessible to everyone within the company.
• Private repositories are exclusive for owner teams to work on.
• Locked repositories, being highly sensitive, can only be accessed by certain authorized developers.

This approach has proven successful in our operations.

While we’ve established a unified code access method, the question remains — how do we streamline the development experience? What actions do we need to take with the accessed code?

Common tooling

Our key focus is simplifying the developer’s daily tasks, reducing cognitive load by limiting necessary tools. Our development cycle is carefully structured around four central steps: commit, build, test, and deploy — all starting with downloading the code and interacting with the repository.

To streamline these integral steps, we’ve introduced the Fury Command Line Interface (Fury CLI). This tool fosters smooth interaction with the code and helps configure our laptops for efficient performance of these actions when required.

During the coding and merging phases, strict adherence to quality and security best practices is imperative. To uphold these standards, we’ve implemented a feature called Release Process, our proprietary CI/CD system. In addition, the Release Process aids in version creation, simplifying preparation for Blue-Green deployment (one of our available deployment strategies) via a straightforward two-click action.

Our development workflow is further supported by a comprehensive suite of tools. They serve to enhance efficiency, shift left and maintain code health & quality, and ensure security throughout the development lifecycle. From safeguarding against the accidental push of sensitive data, adopting pre-commit hooks, to orchestrating deployment strategies, every tool plays a crucial role in optimizing our workflows. Here is a brief overview of the pivotal role each tool plays:

• Fury CLI: Facilitates easy configuration and daily task execution.
• Artifact Repositories: Enables pushing and pulling of application dependencies.
• Official SDKs: Incorporates official Fury Services SDKs.
• Release Process: Manages the entire CI/CD pipeline.
• Building System: Constructs and stores application packages.
• Compute Service: Optimizes cost efficiency and performance.
• Deployment Service: Orchestrates diverse deployment strategies.
• Logging Service: Centralizes application logs.

At the core of our strategy lies the seamless integration of these capabilities with Fury. Through a unified interface, developers gain simplified access to our CLI, API, and website. This cohesive and streamlined approach greatly enhances the productivity of our engineering team, allowing them to efficiently and easily carry out tasks.

In our effort to streamline the complexities of diverse application types and programming languages, dedicated teams focus on optimizing our code to achieve higher performance and quality, while also complying with stringent security standards. From a developer’s perspective, our primary focus is on understanding the value of a standardized workflow pipeline.

Standard pipelines

As described above, the complexity among varying combinations of tools, standards, technologies, programming languages, and team dynamics can be vast. To create a significant impact, it is crucial to abstract these variables. A case in point is the selection of different branching models. Despite the variations in these models, one fundamental remains — we need a stable branch to represent the production-ready, stable code.

The stability of these branches is guaranteed by working on and iterating feature branches, where new creations are developed and refined. These act as creative arenas for innovation and growth.

However, for our stable branches, we adhere to specific criteria:

• Attain a high level of code coverage.
• Ensure all tests perform optimally.
• Use only the most secure and stable libraries.
• Avoid the use of hard-coded secrets.
• Minimize the occurrence of code smells and discourage bad practices.

These requirements keep evolving and augmenting. Regardless of the chosen programming language — Java or Golang — or the preference for gitflow or trunk-based development as a branching model, the unified development experience amplifies value for all developers. This enhancement is especially pronounced when we consolidate our quality capabilities into a virtual supply chain to drive high impact.

We’ve devised specific building blocks to guide our engineering team to meet high standards in all quality checks throughout the Software Development Life Cycle. These building blocks comprehensively manage aspects such as code coverage, code quality, dependencies, and vulnerability assessment.

We not only maintain standard checks, but also value flexibility in configuration, optimal DevEx and continuous feedback from developers for continual improvements.

Conclusions

Looking towards the future, we are focusing on gathering more comprehensive data and smarter metrics about each of our capabilities. The aim is to enhance decision-making processes and identify the next set of capabilities to integrate into our software culture and lifecycle. We look forward to adding new capabilities that will accelerate and streamline the output of our engineering teams (and yes, this might include forays into AI).

However, it’s important to note that this is our approach to tackling complexity and may not fit all situations. Careful assessment of your team’s needs, your organization’s requirements and unique blueprint, as well as a detailed understanding of your current context, is vital in determining the right solutions for you.