INCREASING DEVELOPER VELOCITY AND SATISFACTION THROUGH AUTOMATION
Abstract
By 2019, Grupo Boticário started it`s technological transformation, and internalized the development of many solutions. Due to this business change, the technology team grew substantially, from less than 500 in November 2019, to 1,600 people in December, 2022.
As we know, during this period, the COVID pandemic changed the technology market, and good professionals, mainly with cloud and infrastructure skills, became hard to hire and development tasks were commonly blocked by pending provision of infrastructure and cloud services. Going forward, the increase of the maturity of architecture and security of our solutions included extra patterns to be achieved manually, bringing even more complexity and time spent on provisioning.
The solution was to automate the process of provisioning common services, using some well known and some disrupting tools. The success of this solution is so successful that the time spent on the process dropped from 5 days to a maximum 20 minutes.
Background: multiteam and manual way
With more than 1,000 developers working on our solutions, it was a strategic decision to have a specific team to take care of cloud and infra topics, considering costs optimization and the difficult to hire infrastructure workforce.
The process used by that infra team is almost a market pattern and broadly adopted by large tech companies. It already adopts good practices and modern tooling, such as Infrastructure as Code, Terraform and HCL, Github Repositories, Pull Requests and Atlantis
Using the Gitops concept, the source of truth was Github Repositories with HCL defining the infrastructure. The repository contained the infra of a specific product and was maintained mainly by the infrastructure team, making changes and creating new resources on demand, fulfilling requests opened by the development team.
We can resume all process using a activities diagram:
Despite this being a very good and elegant solution, it was common to have a conflicting prioritization between Development and Devops backlogs, and unexceptional delays occurred caused by blocked teams waiting for each other. This all, adding the fact of our restricted workforce on Devops team, made the problem escalate to a very painful grade, so, a solution was urgent for this situation.
Looking for better process
More than just automating the infrastructure creation, the upcoming changes gave us the opportunity to enforce some other patterns that were not always implemented by the manual creation process, like security enforcements and roles and policies restructure. And, of course, this solution must increase the user experience for all teams involved in the creation.
First off, we separate our infrastructure into two categories. Each category has its own stack, target team and a defined desired experience for each one.
The first category is structuring infrastructure that comprehends the base infrastructure to run our solution. It contains the definition of clusters, security groups, global roles and policies, for example. This category is maintained by the devops team and, to not change its experience, the stack remained unchanged, using the flow detailed above, with Terraform, HCL and Atlantis.
The second category is product infrastructure that contains all resources related directly to a product of a development team. For this category we change the stack substantially. The most disrupting change was the adoption of Crossplane over Terraform and ArgoCD over Atlantis.
This stack change enabled us to have the same experience for both application and infrastructure deployment: both are defined into application repository, deployed by Github Actions and ArgoCD, that provide a unified status too.
Going forward, we abstract all complex parameters of cloud services, using helm charts. So the code of infrastructure into the repository is simple and readable, and all patterns of security and infrastructure are enforced by devops teams into the helm chart. We also automated the generation of infrastructure code, application base code and pipeline, using a template machine built by our platform team, called Scafflater. It all uses Backstage as interface and the final experience for developers is a form with few fields to be filled, resulting in a repository ready to be deployed minutes after requested.
With this new process we transform the way to work, mainly of devops teams. Now its work impacts all developers by template implementations instead of manual provision, focusing on how maintaining templates ensuring that all company patterns are implemented:
Conclusion
All global scenarios that we passed the last 3 years showed that tech is not more a coadjuvant on every business. The old IT was transformed, and the way businesses work demands more agile and solution builder teams that need to focus on constructing solutions for their clients, with the velocity that they need. It does not matter the size of your company, if it does not care about automating manual processes, its destiny is to have a time consuming process and a misused and unhappy workforce.
