Platform Engineering
Developer Experience (DX) at DAZN
DAZN DX team focus, structure, operational model, values and more
For the past two years, I’ve always wanted to write about the DX team and DX at DAZN, but I felt we needed more “stories from the trenches”.
Maybe there is never a “right time” but for sure we now have something to share. There is A LOT to say about DX and how the DX team at DAZN evolved, this is the first post about the DX topic.
DAZN ENGINEERING
When talking to engineers from other companies, at some point, we’re inevitably asked: “What even is DX?”. But, before diving into Developer Experience (DX) at DAZN and how the DX team operates, we should share some context about DAZN.
DAZN engineering is distributed in several locations, mainly UK, Poland and the Netherlands but not only. At the moment there are over 500 people in DAZN Engineering across ~60 teams and we are expanding.
At DAZN one of our mottos is “You build it, you own it”, which means that every engineering team is responsible for their services and products, from the architecture and design to development, testing, production deployment, maintenance, and support.
However, even though we embrace autonomous teams, we still see the value in having a centralised “platform” team (Cloud, SRE, Core and DX) that operates to support and empower our engineers. Our architecture also reflects our philosophy, where we extensively use microservices and microfrontends approaches in what we build.
Let’s start with the all important question: “What exactly is Developer Experience?”, at least for us at DAZN.
What’s Developer Experience (DX)?
Whilst talking about DX, we find it can get confusing because there are many interpretations of DX.
We’ve heard DX described as “making it easy for developers to do the right thing”, or as “the tooling that supports developers interacting with our platform”, or just simply reducing the amount of time spent on “undifferentiated heavy lifting”.
So what exactly is DX? We think there are a few different kinds:
Developer Experience (Generally)
There’s a general DX, that is always at the front of their minds of engineers as they do their daily work, always seeking to improve their own tooling and workflows, whether that is by adding a plugin to our IDE, creating a tailored browser extension, optimising our Continuous Integration (CI) pipelines or improving the discoverability of engineering services, data or documentation.
External-facing DX Team
The type of DX which works for external-facing developer products, like Netlify, who have their own take on DX. These teams can be seen more like UX but applied to developer products as their users are developers,(they might disagree!).
Internal-facing DX Team
DX is a new label floating around in recent years, so in this blogpost we want to address our own “flavour” of DX. This is our flavour of DX, which is devoted to improving the efficiency of our own engineers. Our engineers are our customers.
What’s Developer Experience At DAZN?
“Every company has a developer experience, but not every company has a Developer Experience team”.
Probably the best way to present what’s DX at DAZN is to start from the DX team.
The Developer Experience team is part of the Platform department, working alongside the Cloud, SRE and Core Service teams. At the moment of writing the DX team is 12 engineers distributed across UK, The Netherlands, Spain and Italy.
The team is a mix of specialties, from front-end focussed, to more backend and the cloud. Within the team we deliberately chose to not have dedicated product people, instead choosing to share product responsibilities amongst the team (but, more on this later).
So, what is our own flavour of DX? Our personal definition, which also happens to be our team’s mission is: “Empower every developer to build the best products and reach their goals by providing the best tools in a supportive and responsive way”.
What do we mean by being “responsive”? It means that we actively listen and support our engineers, as we’ve seen that our development experience can change and evolve rapidly while we scale as an engineering department and business.
In short: DAZN Engineers are our customers.
How Does The DX Team Work at DAZN?
The engine of the Developer Experience team is our full-time team of engineers with our own backlog, but that’s not the only way that we work, we also have other systems and methods to bring other engineers in to help support our projects.
How DX Manages Our Backlog (Bet Model)
One question we get asked a lot is given how many things we could possibly work on, how do we choose our priorities? So far, we’ve found that our priorities are a mix of:
- Supporting strategic business goals
- Addressing support requests raised by engineers
- Improvements raised by DX engineers themselves
We typically have 2–3 projects in progress in parallel, with all the DX team members working on all or most of the projects.
We considered using a 2-week sprint model, a common approach for engineering teams these days, but given our aspirations for a supportive, responsive DX team, and the way we run multiple projects at a time, we felt the format didn’t suit us.
We also tried a weekly kanban approach, to quickly adapt to changes, and things improved. Even though we were effective in quickly delivering value, unfortunately with a weekly approach we were losing sight of our short-term focuses and goals.
In order to address those challenges, we have taken heavy inspiration from the Shape Up process. So, we work in 6-week cycles (“sprint”), centered around high-level specifications (deliverables) called “bets”.
Why 6 weeks? We think 6 weeks is a good compromise as it’s neither too short nor too long to maximize delivering value and have a strong team direction.
The “betting” process is deliberately open so that anyone in engineering can also propose bets for discussion with the DX team (but more detail about the pro’s and con’s of this later).
We will provide more insights on how we prioritize our backlog/bets in another blogpost.
Solving Knowledge Gaps via DX Rota
There are situations where the DX team doesn’t have enough domain knowledge or in an area to build certain tools, or the capacity (time, people, etc). To address this knowledge gap we introduced the concept of the “DX rota”, which allows engineers outside the DX team to take a secondment into the DX team, to build or improve a tool within their area of speciality.
An example of the rota in practice was when we looked to build the V2 of our front-end deployment dashboard tooling, we thought it would make sense to have at least one front-end engineer from a Product engineering team work alongside us, to help provide feedback and input real-time as we made decisions and trade-offs.
Collaboration Through Working Groups
Another way Developer Experience gets close to our engineers to collaborate is through our “working groups”.
The DAZN concept of a working group is an arbitrary group of engineers, passionate about a specific topic coming together to collaborate and fix/resolve that one specific challenge. Our working groups have formed to implement: a progression framework, chaos engineering, improve service level objectives (SLO) adoption, and to migrate our CI/CD infrastructure to GitHub Actions.
Working groups are a very useful collaboration opportunity for Developer Experience, as they’re usually born to solve an immediate and pressing engineering issue. For instance, our recent move to GitHub Actions was raised first as a GitHub issue, which grew to become a working group, and then our migration was supported by DX.
The “Inner source” Model
If the DX team were to take ownership of all projects, DX would become locked in a cycle of maintenance and support, never being able to work on new features. One way that we are trying to overcome this is through: Inner source, to facilitate all DAZN engineers to contribute to all our internal tools.
Inner source is like open source, but for closed-source software, where engineers collaborate on libraries and tools, similar to how the open-source community does, therefore sharing the responsibility of developing our internal tooling.
Over the last 2 years, we’ve taken steps to actively improve the state of our own inner source processes. Previously, to find shared tools or libraries, engineers had to manually search or grep GitHub, and upon discovering libraries, it was always difficult to answer questions like: is this library still recommended/supported? Who should I speak to about this package? etc.
To tackle this issue, we created our own inner source marketplace.
Projects are highlighted in the inner source marketplace when an engineer updates some metadata that lives alongside their project, the metadata points to the root of the project directory, describes and categorises the project, to make searching for tools easier. Metadata is ingested periodically, and exposed via an API, which then drives the marketplace.
A Misconception: DX Doesn’t Do “All” The DX
A common misconception that we face — even internally — is a lack of clarity about what we as DX team are responsible for, and what remains the responsibility of the engineering teams themselves.
In the early days of DX at DAZN, as we were working on various plugins for our CI system, there was an opportunity to fall into a trap of letting teams believe that the DX team were responsible, not just for the plugin development itself, but also for also updating teams CI configurations for them.
This type of thinking would simply not scale and will not pair nicely with our motto “You build it, you own it”.
Not only does the DX have enough time / capacity to maintain hundreds of CI pipelines or all the internal tools, but critically, we also don’t have the sufficient context or domain knowledge about every project or domain, so these concerns, such as the precise configurations of CI pipelines, we believe are best left as the responsibility of the engineers themselves.
What does it mean to be an engineer working in DX at DAZN?
As mentioned before we are a team of only engineers and other than writing code to develop our solutions, we all take care of the backlog and prioritization, internal processes, roadmap, products, communications, and customers (DAZN engineers) relations and support.
When we started the DX Team two years ago, we were just four engineers iterating on the DX team concept itself and taking care of every aspect of an engineering team. We enjoyed that approach and it proved to be effective especially when you have multiple projects in parallel and you are a responsive team. That’s why we decided to keep that model and structure without a dedicated full-time Product Owner, Project Manager, or Agile coach.
Having a product-oriented mindset and being able to listen, support and get in touch with your customers, the other engineers, are crucial characteristics for an engineer working in the DX team.
For example, we are introducing Backstage at DAZN, a great opensource tool by Spotify for building developer portals and service discoverability.
The main challenge with Backstage is not simply setting up the tool but also making Backstage successful and ensuring that our engineers are adopting it.
Ensuring adoption requires actively getting in touch with the engineering teams, understanding how they can integrate their services to help make the adoption as smooth as possible.
DX TEAM VALUES
Just a quick mention about our values that massively help us to make decision and drive the team forward:
WE:
❤️ support each other
📈 prefer data-driven decision
🚢 ship, ship, ship
🤖 are always evolving
Open questions…
Before writing this post about DX, we asked around for what questions other engineers would want to know about DX. We were inundated with questions like “how do you prioritise?”, and “how do you juggle project ownership?” etc. We’re going to cover off a lot of these questions in a subsequent article that will be released in the next few days, so stay tuned.
UPDATE: “Building A DX Team: Lessons Learned” by Lou Bichard https://medium.com/@loujaybee/4a66446088bc
If you have questions, or want to continue the conversation about Developer Experience, we’d love to! And one of the best ways you can do that is by reaching out to us through our @danzeng twitter account or leave a comment to this blogpost and we will get on touch.
