Structuring & Scaling a Client-Facing Engineering Team: Part I
At Flatiron Health, one of our fundamental goals is to develop high-quality, de-identified datasets for cancer researchers. The team I work on, Spotlight, is responsible for projects that deliver custom datasets for clients who are interested in specific research questions. To create these datasets, we assemble new data pipelines from a mix of common modules, custom queries and scripts.
Spotlight Engineering currently has the challenge of balancing time invested in common infrastructure and code with time spent on one-off, project-specific requirements. Infrastructure improvements include scripts to automate manual engineering tasks, common libraries to share code between pipelines, and web applications to help engineers monitor the status of projects. Project-specific work includes the implementation of new data variables, QA testing for edge cases in the data and custom code to do more complicated patient selection criteria.
Both are needed to scale up the business by increasing the number of delivered datasets, but doing them at the same time is difficult in practice. First, the two types of work require making tradeoffs around engineering time in order to make sure client projects are delivered on time, even while common tooling is being developed. Secondly, the best team structure for one type of work isn’t necessarily the best for the other, and so coming up with good structures requires careful balancing of the strengths and weaknesses of each.
We’ve tried out and retrospected on a few different team structure models over the past two years; below, we’ll outline some lessons learned and give ways to maximize the effectiveness of each one. The project-specific model maximizes engineering responsiveness at the cost of heavy engineering involvement, while the platform-based approach saves engineering time but makes it a bit more difficult to deliver timely value to stakeholders. In the middle, the split time model handles both issues, but has trouble enabling long-term progress in either direction.
Model 1: Project-specific teams
When Flatiron first began delivering datasets to our research clients, the engineering team was organized around these client projects. Spotlight engineers would work as solo engineers full-time on client projects as part of a cross-functional team, and use any downtime to work on tooling improvements that they thought would save time on the next deliverable, with lightweight coordination via the Spotlight tech lead.
Advantages
- High responsiveness — Complicated requirements, shifting timelines, and unusual requests from clients could all be handled fairly quickly, as everything went directly to the engineer to be triaged and solved. This was particularly important because Spotlight project requests are unique by design and so these situations came up frequently.
- Minimal friction — When an engineer saw something to be improved, they could iterate on it with minimal overhead. Engineers could quickly move ideas from design to prototyping to production, and as Spotlight itself was still searching for product/market fit, these short iteration cycles let improvements keep up with the pace of change on the business side.
- Maximizing domain knowledge — With such a large amount of time spent directly on client-facing data projects, Spotlight engineers quickly became experts in their domain: constructing and managing data pipelines for research datasets. This domain knowledge could then be used to inform the planning of future tooling improvements. As a result, many extremely useful scripts, tests and common modules that are still in use today were developed during this period to solve specific problems encountered on client projects.
Disadvantages
- Long-term investment was difficult — Because of the lack of time and multi-engineer coordination, low-effort tasks that solved immediate problems were much easier to justify and prioritize than high-effort tasks that came with delivery risk and longer time commitments. There was not reserved time and space for thinking through bigger picture improvements and therefore, hard but impactful problems were passed over quarter after quarter with no hope for resolution.
- Client work became repetitive — Although each client project had custom requirements in the domain of cancer research, much of the engineering work evolved to follow the same structure and so there was less opportunity for unique and complex design tasks. After mastering the basic aspects of these projects, the remaining work was frequently not interesting enough to keep engineers engaged in the long term.
- Potential for misalignment with engineering growth path — At Flatiron, engineers typically get promoted on the basis of solving increasingly large and complex problems. However, since client projects are typically of fixed scope and capped complexity, engineers found it hard to take on large enough initiatives to demonstrate skills at the next level.
- Hard to hire new engineers to meet demand — Assigning engineers to solo projects produced a flat structure that had minimal coordination overhead, but that also locked us into the disadvantages above. While business demand for Spotlight engineering was increasing, Spotlight had difficulty attracting new engineers. In multiple instances, candidates ended up either dropping out of the hiring process or choosing other Flatiron teams because they were more interested in larger-scale web application work than the data-focused work on Spotlight projects.
Lessons learned
Teams in this structure are optimized to be effective on time-bounded, cross-functional tasks, and as expected, the advantages noted above made Spotlight engineers very productive in delivering client projects. However, longer-term improvements remained out of reach, like handling Spotlight-wide tech debt and code refactoring or making progress on new workflow optimizations that required more commitment and investment from Spotlight engineering.
In the end, as an increasing number of engineers were needed to keep pace with the growing number of Spotlight client projects, the challenges of this model became too large to mitigate without changing the structure of the team.
In the second part of this post, we’ll take a look at another team structure: time-split teams where engineers spend half their time on client work and the other half on infrastructure work. We’ll also discuss some learnings from a third, in-progress team structure: platform teams in which engineers spend time developing a common platform for stakeholders to do client projects on their own, and therefore don’t spend any time on the client projects themselves.
