How we onboard newcomers with tailor-made training
Each quarter, our Foundations engineers provide one full week of tailor-made training to newcomers and volunteers, fostering confidence and autonomy to properly run their applications in BlaBlaCar’s “Software Factory”.
In 2019, BlaBlaCar’s Engineering teams embraced “you build it, you run it” as a strong default of our operating model.
This resulted in having more autonomous development teams, operating their services 24/7 in a distributed way, on a shared platform called the “Software Factory”. Managed as a product, this platform includes infrastructure, tools, practices, and service delivery, all centrally maintained by the approximately 40 engineers of the ”Foundations” department.
The Foundations department is composed of seven teams providing consistent infrastructure, services, and expertise.
Each team owns a different part of the stack:
- Core Infrastructure
- Database Reliability Engineering
- Developer Experience (Common Libraries)
- Engineering Productivity (VCS, CI/CD)
- QA
- Security
- SRE (Observability and Incident Management)
This setup enables our Software Engineers to deploy in production hundreds of times daily, empowering BlaBlaCar to iterate continuously while respecting our reliability, scalability, and compliance requirements.
Why do we need training?
Today, thanks to the DevOps and Site Reliability Engineering (SRE) culture, Software Engineers are very mature in topics surrounding software development. However, the cognitive load generated by the technical environment and the teams’ accountability regarding the “you build it you run it” approach requires solid support in different areas to ensure production traffic is served in a secure and scalable way.
Most of our runtime platform is composed of a galaxy of off-the-shelf technologies such as GitHub, Docker, Jenkins, Kubernetes, MariaDB, Datadog, and many more… Even if these products are well-documented industry standards, their combination, interactions, and the interface contracts we defined are specific to the BlaBlaCar ecosystem.
We aimed to increase our engineers’ confidence and autonomy in operating their services by consolidating their knowledge of the Software Factory stack. Dedicating quality time to a mix of theoretical and practical training seemed to be the most effective approach.
From trials to maturity
Our first tries consisted of dedicating a full week of training on different topics provided by each Foundation’s teams, the initiative was rapidly warmly welcomed and crowned with super feedback.
Despite the good momentum, our Foundation’s teams soon found themselves somewhat overwhelmed by this new responsibility. This occurred several times a year, requiring preparation, alignment, good eloquence, and consistency.
The ambitious goal of having all engineers “more confident and autonomous, having a global understanding of all the important bricks” required structuration to reduce improvisations, stress on speakers, and last-minute calls to support stuck trainees in workshops. And all this, in a full remote context!
We could have chosen to record sessions one time and just share links as an online course but we aimed to maintain live training sessions with direct interactions between Foundations teams and trainees, fostering trust, adaptability, and simply making these weeks a good moment for everyone.
Industrialize to scale
After several successful craft sessions, it was time to make those training a standard aspect of our activity. At that time, as a one-week training is an investment, the key point was to reduce the preparation toil for the training team while maximizing the benefit for the attendees.
As the training requires hands-on participation, we quickly identified that one of the biggest pain points was the environment’s setup. To ensure trainee readiness, we introduced a self-guided training manual providing all the information to prepare the necessary tools and set up laptops. This training manual is sent at least one month before the start providing enough time for everyone to implement it … and the needed time for our trainers to make sure everyone is ready.
To ease communication, we create for each session a specific Slack channel, that fosters interactions by providing a safe place for trainees and trainers to interact. We also designate a Training Lead in charge of organizing the agenda, communication, and point of contact for any organizational issue.
At the beginning of each training week, we organize a keynote session to introduce the different teams and the objective of the training. It’s also the perfect time to go around the table for both trainees and trainers so we don’t need to re-introduce each team during the training sessions, maximizing the focus on technical aspects.
Having learned from our first training weeks, we now ensure that at least two trainers are available for each session. This setup is key to make sure everyone is able to catch up with the training pace and provide thorough follow-up support.
“Feedback is a gift”
To validate that both content and experience are and remain impactful and positive, we send a short survey at the end of each training session to gather feedback and ideas to improve our setup even more. We also make sure to do regular retrospectives and review the content, to stick with up-to-date processes and best practices.
Finally, each year, we send our alumni two short questions to confirm the training impact over time
Learning by doing
We want to keep our training hands-on, however, it feels impractical to have our trainees (who are mostly newcomers) learning on real production applications. As a consequence, we have decided to develop a demo-app for the specific purpose of training.
The demo-app is built using the same patterns as our actual production services and can therefore exhibit the same failure modes given the same causes as our production workloads all the while being simpler to deploy and less resource-intensive, which makes it more practical (and cheaper) to stand up and tear down.
The demo-app is complete with a simple Single Page Application (SPA) frontend (which stands in for the actual web frontend), supply services, aggregations services, database links and feature toggles.
Both the training and the Chaos Engineering groups contribute to the demo-app to make sure they can be used to replicate failure modes or to demonstrate (or test) software factory principles.
Finally, does this training succeed in increasing global understanding and autonomy using the main components of the Software factory for our Engineers?
Yes! Our global yearly survey shows that ~84% of attendees feel more confident after the training, and 97% understand the Software Factory’s purpose and components. Our most recent survey reported an average rating of 3.7/4 for all sessions included. These scores prove that our training is a key component of our toolset, ensuring global knowledge and promoting best practices. We are proud of it and encourage you to adopt it!
In addition to raising the bar, this training adds proximity with our end users which is very important for a support team.
As of today, 115 engineers have been trained, 60 in 2023, and many more to come 🚀
This story has been co-written with Remy Gronencheld and Denis Wernert
Special Thanks to our reviewers Mathieu Turpin, Ceydric Jean-Bolo, Usman Akeju and Nicolas Salvy
