New is legacy
Data journey at Back Market
Written collaboratively with Thomas Clavet, Roxanne Ricci, Lucien Fregosi, Nolwenn Olivier, Marc Alonso, and Aknin Karine.
Please have a seat, we are at Back Market, a refurbished marketplace, and it is moving fast… really fast. We will share with you how data is a key asset valued and safely guarded by our team.
The Data Engineering team at Back Market is rather young, a bit more than a year old. But in a context where business needs and feature requests keep increasing and our data pipeline seriously begins to reach its limits, our tasks to gather data and provide reliable, resilient, and high-standard pipelines are not that easy… But infinitely interesting and challenging!
From the creation of our team to the vision of our data product and actually building it, we take you with us on our data journey. We will share with you our choices, the bumps we suffered along the road as well as how we overcame them. Fasten your seatbelts!
The beginning of it all…
Back in July 2019, the absence of a Data Engineering team brought a global reflection inside the BoT (Bureau of Technology).
As data had been identified as a key asset, it was necessary to invest time and energy in that matter. Furthermore, the data pipelines were maintained by a handful of external consultants, and handling emergencies and production issues were consuming their time at the expense of refactoring, reorganizing, and building a new and more solid platform. The documentation was rare, and the turnover was quite high.
The existing pipeline was facing scalability issues due to Back Market traffic growth, the load on the database kept on increasing… The need to create a team became critical.
And when Black Friday arrived, it was a rough ride…
And the data engineering team was born!
It was agreed that it would be the team’s job to build, maintain and provide a solid data pipeline to guarantee its availability, reliability, and ability to scale and handle Back Market’s growth. All the while slowly but surely migrating the old pipeline to a new, more scalable one.
We have adopted the “You build it, you run it” philosophy while keeping in mind that New is Legacy. Continuous improvement is key for us.
So our mission combines multiple aspects:
- Running the production platform (which includes maintenance and support).
- Designing a new, more scalable, and resilient platform.
- Migrating the old platform to the new one, one use-case at a time.
- Taking business needs into account to deliver new features.
- Continuously improving our architecture and practices to minimize technical debt.
We decided to set our course by identifying what needed to be done to improve the data pipeline for Black Friday 2020. It gave us a good starting point to think about scalability, resilience, and reliability.
As to decommissioning the old pipeline, we settled for a parallel run. Once the foundations of the new pipeline were built, the idea was to identify key features, used by the business teams, and migrate them, one at a time to the new platform.
This way, we could prioritize the features to migrate hand in hand with our end-users and leave altogether what wasn’t used. But to concretely achieve all these great ideas and principles, it was essential to put aside feature delivery until the foundations of the new pipelines were built.
Without these foundations, it would have been impossible for us to deliver the new features and migrate.
So we decided to measure and monitor the pipeline health to understand what’s going on and be able to provide help or fix for guarantee data available for our consumers while delivering business functionalities. Then we negotiated two non-consecutive sprints of 2 weeks with no business delivery with the stakeholders. The first sprint was used to dedicate time to technical development to ensure solid grounds for the future, and we used the second for writing extensive documentation.
Black Friday 2019 left its mark and gave us an opportunity to think about how we wanted to work. It seemed essential to us to share the knowledge about the whole stack amongst the team and the other BoT teams. No one was to be a SPOF (a single point of failure), and our goal was that everyone would be autonomous.
The construction of a brand new data product is in sight
Autonomy should not lead to a scattered architecture with uncorrelated features. A common long-term vision was what we needed within the team to achieve this goal. This was when data arose as a product. We switched from a paradigm of a component team to a full-blown feature team. With this new focus, we needed to borrow all the scrum and agile practices put in place in traditional feature teams. This meant planning sprints, rituals, and so on, but this was useless without the definition of our vision.
It was very clear to us data is a core asset, and as such everyone should be able to interact with it in a clear and easy way. We wanted to build a product that made sense to the business and not a technical product that we would be the only ones to understand. Going for a self-service product as much as possible was the way to go.
To achieve this complex goal, it became clear we needed to position our product at the crossroads of business needs and technical expertise. To bring real added value, we need to work with other feature teams to build features useful to most.
Having everyone aligned on this vision within the team is key! This is where we can achieve real autonomy. Gathering around this same data vision as a product enables us to make collegial decisions. This also means adopting the build & run philosophy to be responsible for our product. Everyone in the team has the technical knowledge of it, thus building it the way we want to achieve our long-term data vision as self-service.
The data platform was born with two main missions:
- Meet the needs provided by the pipeline in place.
- Provide a sustainable architecture for facing the high-growth of Back Market.
Before going into technical architecture details, a manifesto was created to keep in mind our target during the road.
Data Platform manifesto:
- Support multi-cloud provider architecture.
- Provide scalable, reliable, resilient, and high-available data pipelines.
- Provide insights for the data producers, data consumers, and their usages.
- Collect, prepare and serve trusted data.
- Deploy on multi-region.
- Collect, prepare and serve data with one single source of truth.
- Secure and compliant with the GDPR legislation.
It must continuously integrate new features brought by all the back makers (people at Back Market), but we started to work on mandatory features:
- Collect internal data from databases and external web APIs
- Prepare, normalize, historize, and govern the data flows
- Simplify data flow creation and evolution
- Monitor and access data flow from end-to-end
- Automatically scale our data pipelines
- And many more cool technical features that we will share with you!
… With a collaborative teamwork
Collaboration is at the very core of our data team, as it is more generally the case at Back Market. From the way we make our architecture decisions to the complementarity of the team members, we thrive to put teamwork at the core of it.
As the team grew, the onboarding of team members was modified to deal with COVID-19. Human Resources did a great job by providing all information and preparing material for working remotely at the last minute… The team's challenges were adopting a new management style by sharing the same level of information with everyone. There was also more dedicated one-on-one time with the manager to share thoughts and ensure overall well-being during the integration.
We also set up a virtual room opened for daily-to-daily talks and having fun times on the agenda… and we did more peer meetings to share our technical vision. Our architecture is collaborative: everyone comes from a different background or has a different skill set. We use that diversity to our advantage.
No architecture choice is made single-handedly.
In most cases, one person takes the lead and acts as a scout on a specific subject we want to tackle. He or she researches and thinks about that subject, how it will fit into our current architecture, how we will break it down into small iterative steps, and so on. And then, that person shares what he or she learned and his or her thoughts, and we discuss it within the team until we reach a consensus. This way, everybody involved continuously learns and grows from these rich discussions while leaving space for everyone to explore and learn about new tech subjects on his or her own.
The team’s strength relies on skills and background complementarity.
We share the same values of respect, benevolence, conscientiousness, and technical enthusiasm. We are driven by the Software Craftsmanship manifesto through code reviews, peer programming, and switching roles. Our code is the team’s code, there is no room for ego in there! We do not hesitate to regularly take a step back and analyze our code to spot what can be improved, mob programming style, and reduce technical debt. As we said before, “continuous improvement” is our mantra!
Our team is quite eclectic yet complementary. Some of us come from traditional engineering schools, others from a BI team and cross over to the data-engineering field, some from a professional reorientation, and so on. As we speak, we count 3 women out of 8 team members.
All these little ingredients added to the fact that everyone in the team adheres completely to Back Market’s purpose, making for a multidisciplinary, united, and benevolent team.
Prepare yourself, the journey is only beginning!
We hope you had a pleasant read, sharing our thoughts and vision.
These are the first steps we made as a team on our path to reach our technical north star. It is a long way to go, but we will make technical choices to enable change in the architecture over time, better to serve data at the same pace of its usage.
You are probably thinking of how we capture these data flows… one of the first challenges was to determine and track data that has changed from the internal website’s database…
Don’t worry, it will be the next chapter of our story!
If you want to join our Bureau of Technology or any other Back Market department, take a look here, we’re hiring! 🦄
