Organizing Data Teams: Build an Army and Run an Internal Job Fair
From bits of data engineers scattered all over the place to one team
Written by Filip Beslic, Area Manager Data
DPG Media, the largest media company in Belgium and the Netherlands, near the end of 2019 embarked on a mission to become truly data-driven. We’ve surpassed our initial expectations in less than two years, and we now have a fantastic data engineering department in place. Not by spending tons of money bringing in new blood, but by scraping together bits of data engineers we had scattered all over the place. Want to know how we did it?
Rally Up All Data Experts
We wanted to bring data engineers working for DPG Media together under one roof. Sort of speak, as roughly half of them worked in Amsterdam, the other half in Belgium. And they were spread across four different locations and five teams in total at the very beginning.
These data engineering teams did not come alone as they all brought their own set of demanding stakeholders, backpacks, and technologies. Some wrote Python, others Scala and a significant number of them neither. They were responsible all-in-all for a staggering total of 103 production systems! Most were running on on-premise Linux or Windows servers, others in the AWS cloud, and one funny duck in Azure. Right. We estimated at the beginning that 60–70% of the overall time went into firefighting and maintaining business-as-usual. We soon realized that this was not to go away anytime soon.
This really puzzled us at the start, as we had big plans but nearly no bandwidth to put any of them into action.
An Army of Data Engineers
No one could take on a bigger project as firefighting ate large portions of the capacity. We would need to glue together those little bits of free bandwidth spread across these teams to make a true impact. So, we went for the big army approach. Instead of having us all fighting smaller battles, let’s unite and wage war where it really counts.
But we would need to rethink our little data organization completely for that purpose. We needed to tear down the informal walls between our teams as they divided us. Any structure, really, either implicit or explicit, had to go. What we wanted was some form of radical collaboration. That was the idea and the vision, at least.
Inspired by LeSS
But how? We turned our eyes on that one radical concept few tried — and some abandoned quite quickly for being too invasive. Most (if not all) companies go for the easy and SAFe approach; we went for LeSS or large-scaled-scrum and applauded it for its radical thinking and simplicity. But as with all things, putting theory into practice is something else altogether. When companies want to change things, they usually mean they want to take what they already have and change it ever slightly.
Our stakeholders, our raison d’être sort to speak, our financiers and thus implicitly our bosses, we’re no different. Why change things dramatically? People usually want nothing to do with the out-of-the-ordinary, and definitely nothing radical! Even the engineers within our new data area initially were not too fond of change either. So, the journey to our destination is a long one, one of many steps, tries, and failures. The process of getting there would lead us off-topic too much, so I will spare the details for another time, perhaps. Instead, let’s focus on the last steps, just before we established a modestly well-oiled machine that has brought us where we are today.
Sadly, we can’t change everything to our liking. The environment and the context, in reality, don’t allow it. So LeSS became a source of inspiration rather than a framework or methodology for us. This might have some purists triggered as we refer to the dismissive term ‘ScrumBut’ within the Agile community for when teams do not apply Scrum 100% as intended. Some might see such tailoring as a failure, but in all honesty, it has yielded tremendous results for us and a dramatic change in our way of working — for the best.
One of the First Steps We Took
With data running throughout the entire organization, the data landscape is vastly complex, with production systems and a heavy operational burden tied to specific business domains. To make things worse, considerable differences and alternative solutions existed between the two countries for dealing with similar problems. As a result, teams were doing similar things but working with different tools and technologies on the other side of the border.
The first step was to organize ourselves in four strategic domains of B2B, B2C, Tracking, and Profiling. With a Product Owner each and an overall strategy for both Belgium and the Netherlands. The Platform Manager was responsible for the overall data area backlog. Despite the country borders, we then wanted to be able to rotate data engineers between these domains depending on the priorities.
An Internal Data Job Fair
Everything we wanted to build had to be put on that one large backlog, it sounds fancy, but it’s just a simple sheet. The items there would then be prioritized and estimated. Then, depending on the overall capacity, the cut-off would be made, which implicitly also determined the number of seats needed for a given domain at a given time.
But by rotating data engineers, many feared that continuity to maintain firefighting on production systems would suffer as it required specific domain knowledge one could not just acquire quickly. So instead of doing it every month, our initial idea was to rotate every quarter in what we now call the internal job fair. Still risky by some.
At the start of each quarter, Product Owners give a sales pitch to the entire data engineering department (43 engineers) to convince them to join their domain the next quarter. Choosing a domain also means becoming responsible for the firefighting within that domain.
The results of our first job fair:
- 47% opted to stay in their domain
- 84% ended up having their first choice
- 10% was open for anything, so we let them stay in their current domain
- 1 person had his second choice
The second time, everyone got their first choice.
The one data area backlog is also the only instrument to monitor progress. Despite having Product Owners usually having their separate backlog, they and their engineers break down those larger backlog items into smaller, more graspable pieces.
Some domains story-point their work; others don’t. In fact, these short-lived domain teams have absolute freedom to organize themselves. They are barely monitored and controlled but rather trusted, empowered, and enabled.
Many work Scrum, others Kanban? For all we care, they use Prince2; the only thing that matters is the data area backlog they are supposed to work on. They do need to participate in the two-weekly demo to the business, even if they prefer not to work in sprints. For everything else: a lot of freedom. A bit radical, maybe, but the results are staggering! Noteworthy that by rotating, we end up with cross-pollution and eventually an ever-evolving way of working.
So, What Did it Get Us?
By giving people trust, less time goes into monitoring and micromanaging teams. As a result, more time becomes available to do the things to get the entire area to move forward — a shift of focus, sort of say.
We hoped by giving people freedom and a choice in the projects they would work on, that this, in return, would spark their motivation, ownership-taking and provide a greater sense of purpose.
But we were very well aware of the risks and potential problems with the ever-diminishing concept of a team within our area. We hoped people would feel part of the larger data area team instead. So, based on the book “Nine lies about work,” which states the importance of the team, we decided that somewhere in each quarter, we would have to do a small survey to monitor precisely that. Some results:
It was a shocker that people who hadn’t worked together before and often lived on the other side of the border feel part of a team, stating their teammates have their back! The initial skepticism people had quickly demised. Even those not fond of rotating admitted it felt each time like a new wind was blowing in their domain with new faces coming in and other bright ideas and perspectives on how to do the things they were doing. The job fair soon also became a driver for more harmonization and simplification of the landscape. Yet, we do have to admit that, by enabling us to go faster than before, this also comes with a price of growth pains, sometimes in the form of an increase in legacy.
But back on the positive note, we later discovered that the job fair also provided people with a way out. Out of a domain they were unhappy or frustrated about. So occasionally, people left a domain, but luckily not the company! The job fair enables us to identify problematic domains with much legacy, large amounts of firefighting, or other underlying problems. And more importantly, it now also provides the necessary arguments to question the status quo, and with it, the opportunity to change things structurally.
Freedom to Move
Again, the road is one of many tries and failures, and extreme pragmatism led us to success. There is also such a thing as too much freedom, where a more directive approach sometimes is just for the better. But then again, it is surprising how much responsibility people will take on and completely surprise you when given the opportunity. Giving people freedom also taught us that not everything has to be done perfectly. It is a balancing act, but it is essential to let go first to find that right balance.
At the very start, before the job fair ever took place, people would remain in their team — no matter what — as they preferred their familiar environment. Also, it seemed undesirable to move people who were very knowledgeable about a domain, its tools, technologies, and production systems. It was thus nearly impossible to scale up or down, depending on the ever-shifting priorities. The job fair resulted in more motivated and happy people, cross-pollution between domains, and gave us enormous flexibility to deal with the ever-changing environment. With it, we can scale up when the time requires to make an impact.
Would you be motivated to work in such a place where you can choose which projects to work on? Would you be challenged in an environment where one quarter you might be writing Python and the other Scala? How about learning the ins and outs of Redshift first and then later Snowflake?