How to Build a Modern Data Team? Seven tips for success
The modern data world is full of noise and cutting through it to work on what’s important for you to deliver on data initiatives is hard. Last year started as the year of Data Mesh but ended as the year of Gen AI, or maybe this year is Gen AI and last year was Data Contracts. And where did Data Products and Data Catalogs go? I get confused and you may be too. If so, then this post is for you.
I spend my time with clients of Everest Engineering to help them build data platforms, data products and data teams. Learning from them let me share the 7 things that I think are important.
- It starts with the team
- Use Team Topologies as a pattern language
- Think centralised and de-centralised at the same time
- It’s a Full Stack Data Team not a team of full stack data engineers
- Mindset and First Principles always beats certification overload
- It’s important to have fun
- And remember Conway’s law — your data architecture will follow your organisation (including the fun)
It starts with the team
or, the Modern Data Team is your Legacy Data Team
With the recent publication of Tristan Handy’s post titled, “Is the “Modern Data Stack” Still a Useful Idea?”, there has been a lot of discussion that the word “Modern” is not very helpful as technology evolves so quickly. Well, the same is true with people. They learn … quickly.
The reason I start with this thought is that I have seen organisations make assumptions about their data teams ability to build new stuff, based on existing skills and existing tech. The problem is often not the team and their ability to learn new skills, but the system of work that limits innovation. Bringing in a new team feels like a smart move. One reason a new team may succeed is that they can alter the “system of work”.
There’s a thousand reasons that managers give for building a new tech stack over here with a new team and leaving the legacy team over there to keep the lights on.
Aside from the fact that it may not be the best way to motivate your existing employees, the data business is a lot about data and your existing team may know quite a bit about where the skeletons are buried in the current data stack.
If you’re worried about your existing team needing to learn new skills and the time/cost of it, well the cost of losing them to a competitor who just happens to want to invest in their development might persuade you to re-train them.
Technologies change and so do organisations, which means that at some point your Modern Data Stack team will turn into a Legacy Data Team. Treat them as people, not legacy. Remember the incredible ability of motivated individuals to learn new skills (probably faster than a skilled data engineer will learn your business).
Use Team Topologies as a Pattern Language
Team Topologies is a concept (originating in DevOps) that has received a lot of attention recently in the area of designing organisations for fast flow. The authors spotted patterns in DevOps teams (like the dedicated DevOps team living in the halfway world between Dev and Ops) and extended these patterns so they became more generally applicable.
So, yes, Team Topologies is a good model to use for data organisations.
The reason I included this point is because I read a lot of posts suggesting that you apply Team Topologies to design your data organisation. I’d like to suggest this is not the best way to use Team Topologies. And orient the discussion towards the use of Team Topologies as a diagnostic and pattern language to think about how we evolve our teams.
This is worthy of a much longer post so I will try to summarise my views here.
Traditional data warehouses are like monoliths where we put all the data, all the business rules and all our historical mistakes. This creates humongous cognitive load, so it is likely that your current data team is completely overloaded.
Hiring new people doesn’t solve the cognitive load challenge which is one reason that you’re finding that hiring new data engineers does not speed up data work as much as you expected.
Team Topologies educates us on 4 typical team types (Stream-Aligned, Complex Sub-System, Platform and Enabling) and 3 interaction modes (Collaboration, X-as-a-service and Facilitation) and introduces the concept of Team Cognitive Load. I recommend that you start by thinking of Team Cognitive Load as an indicator dial (in Australia we have a fire warning indicator dial that goes from Low to Catastrophic). You should constantly be monitoring Team Cognitive Load and taking action as it gets too high.
Then, think of the interaction types as another sort of indicator to watch off. This is where you start to use Team Topologies as a pattern language. It helps you organise interactions between teams for faster flow.
Lastly the Team Types are how you classify what existing teams do and spot you have the wrong team type or setup in any particular team. And this inhibits their flow of value.
Think Centralised and De-Centralised Together
This one came to me when a client put it in a proposal. My first reaction was “are they stupid? How can they be talking about both centralised and de-centralised data at the same time?”
It took me a while to work it out.
Many years (decades even) of enterprise data warehouse work and Big Data work has developed the Centralised Data paradigm to the point that it is a strong muscle that most companies can flex today. At the same time the application stack has been gradually becoming more distributed. Anyone for microservices?
So, a typical organisation Everest works with tends to have distributed application development teams and (mostly) a centralised data team. This is not a good fit. If you are a fan of Eli Godratt’s Theory of Constraints then there is a candidate right here. Hmmm, I’m just thinking this could be another interesting article to talk about TOC and data teams?
It’s clear to most CTOs I talk to that having the data analytics use cases developed in a distributed fashion (i.e. de-centralised) by the different departments or capability teams in the organisation makes sense. But we have already seen the challenges with different departments building their own data analytics solution where none of the numbers add up across the organisation.
So, a centralised platform to support self-serve analytics makes sense.
But, how much to centralise and how much to distribute? Well that depends on the organisation, the size of the problem and their willingness to collaborate and change together.
My advice is to maintain a healthy tension between the centralised team and the different end user teams (or departments). End users want more access to source data, more control over their work. They will pull de-centralisation if given the right support. But centralised teams have some data governance responsibilities. These should all be developed as rules in code (think Data-Governance-as-code), but how fast you can develop this in the data platform team will keep the push-pull tension.
Oh, and doesn’t this sound like Team Topologies right there — an example of stream-aligned and platform teams collaborating.
The Full Stack Team
My inspiration for this section was a DevOps Enterprise Summit white paper entitled “Full Stack Teams, not Engineers”, which pointed out that the engineering team needs a mix of skills and no single superhero developer can do everything in a complex tech stack.
In data teams we may think we don’t have this problem because we have data engineers with Airflow and Python experience, analytics engineers that know dbt and SQL, data analysts that know Tableau and infrastructure engineers with Terraform skills. So we already have Full Stack Teams, right?
Maybe. One lens we put on this discussion at Everest is that of skills liquidity. This is an approach to reduce key person dependency by spreading skills development and mentoring throughout the team. At Everest we have developed a specific technique for this called Skills Mesh™.
Skills Mesh (no ™ but maybe there should be) is nothing more than a workshop that we use to engage the team in an exercise to map out what skills they have, what they want to learn and what the project needs. It is a fun way for everyone to get involved in the skills development game at the team level and helps Everest and our customers learn together.
More specifically it builds a resilient and highly skilled team, able to accomplish miracles and leap tall buildings in a single bound.
The Value of First Principles
I interviewed 2 Data Engineers for a project that was going to be building a data warehouse using Snowflake. The first interviewee was very experienced and had Snowflake certifications as long as your arm. Every question or problem we asked was answered with some very deep knowledge of Snowflake features, but we detected a pattern. We (the interviewers) decided that this person had a deep knowledge of Snowflake but perhaps did not understand some database and distributed system principles. So, we changed our interviewing approach to take the candidate out of their comfort zone and we found that they had great knowledge of Snowflake but were less able to solve problems if Snowflake was not in the list of tools available. The other candidate had no specific knowledge of Snowflake but was much better equipped to understand cloud platforms, data storage on cloud and the principles of processing and querying data from databases. Effectively the second candidate demonstrated to us that they understood the principles of working with cloud data platforms, despite no specific Snowflake knowledge.
So we picked the second candidate and have never looked back since. That second candidate has impressed a senior group of stakeholders with clear thinking, understanding development cycles and how to integrate and test a complex data pipeline.
Recently an aspiring data engineer asked (in a Slack workspace) what skills they needed to learn to progress in their career. They were angling for advice around AWS or GCP, perhaps Snowflake, Redshift or BigQuery, learning Airflow or Dagster. Instead we talked about learning first principles of databases to start with. What does a DAG represent and how to interpret it? Why model data and why no data model is still a model? What’s the difference between dbt as a framework to build transformation logic and Spark as a distributed processing framework.
I’m a believer in diving below the tool and certification layer to understand the principles of what we are trying to learn. I think it gives you a better understanding of what is actually happening and allows you to better weigh trade-offs in your data engineering work.
Having Fun
People who know me at work will know me as a serious bloke that spouts lots of technical stuff but would not associate me with fun! Perhaps boring, geeky, long-winded come to mind?
Luckily the people I work with are able to get me to relax and have a bit of fun. It makes work worth participating in.
Get stuff done and celebrate …. a bit.
Which leads me to Conway’s Law. To quote Melvin Conway, “Organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations.” So, what if we have design a fun organisation? Does it produce fun systems? I sincerely hope so ….
