Biggest Challenge in Data & Analytics faced by Generalists: Scale.
Tech firms and Fortune 500 companies have various ways to refer to what I call generalists. Some refer to them as Full-Stack Developers, while some call them Analytics Engineers. These people are often solving complex problems with data, tools, and analytics single-handedly, start-to-finish. They usually are very good at delivering results and impact within the business units they support.
That said, there’s a monster that inevitably rears its head for each of these problems they’re asked to help solve. Scale. And, without understanding what’s at play here, these generalists quickly lose capacity to continue to be effective.
The Law of the Instrument in layman’s terms states that to a hammer, everything looks like a nail. Applied into a data and analytics space: to software engineers, everything can be solved through software. To analytics developers, anything is possible in Tableau or Power BI. To systems engineers, everything can be automated. And, to data engineers & scientists, SQL can do everything.
Any of those outcomes will produce a solution — but relevant questions remain:
- Is that particular solution the one that will be most effective for the business and users? In other words, will it become quickly irrelevant?
- Is that particular solution something the team can support in perpetuity? Or is this something only the developer (or generalist) have knowledge to operate & support moving forward?
- Is that particular solution enabling future extensibility throughout the rest of the technology stack you’re responsible for?
If either of those is no, we create tech debt and reduce capacity where we may have been able to add more business impact otherwise. This is where the Product Mindset for my team comes in full-force. Asking ‘Why’ and finding balance between possible solutions, and the users’ & business needs is critical for long-term scale.
In the figure above, the left side (in blue) represents a simplified version of our legacy operating model for solution engineering. The ‘How’ & ‘When’ typically defined ‘Who’ solved the problem because free capacity drove how the work was assigned. But, because no one was asking ‘Why’ — we never built the optimal solutions for our users and for the business. Half of what we built became antiquated and seldom used, and our hard work rarely drove the level of impact we were capable of otherwise.
To make matters worse, the hourglass effect continued to constrain new development, because keeping those sparsely used solutions up and running competed with new development being asked by the business. There are ways to reduce the hourglass effect in this scenario, such as proactive product decommissioning or robust prioritization at planning stages, but we as a D&A team lost the ability to influence the type of solution because the development was typically driven by a person’s availability and/or the business’s assertions.
The right side of the figure (in magenta) shows an operating model similar to the one we employ today — which falls in line with the Product Mindset. This operating model has created space for ideation, and it keeps us out of the one-track mindset of solving problems which are easy for the developer — but difficult and/or useless for the user. We also created space for Total Cost of Ownership evaluations (TCO) because long term operations & development support for products is not cheap.
Because the product teams & leadership teams now prioritize the work, they understand and plan for development dependencies which align directly to the needs from the business & users. We now have a backlog, and a roadmap for each product. This creates a pipeline which has begun to expose where we could scale up (data engineering, software, data science, analytics) to go faster. It also leverages teams’ skills by empowering them to build depth in their technical competencies, rather than trying to have a team comprised of Jacks-of-all-Trades (and master of none).
