Scaling Delivery Using Vertical Engineering Teams — Part 1: Human Challenges

This article was originally published on the Flipp Engineering Blog back in November of last year.

This is part one of a three part article on the various challenges that come with migrating an engineering organization towards vertical (or “mission-based”) teams from system or language-based teams, and how Flipp has attempted to address the challenges as we encountered them. It was written last year during my time as a team lead on one of the first teams to make the jump.

Over the past several years at Flipp, we have begun a shift in the structure of our engineering organization. We have migrated from teams being organized around the systems that they own, to a structure which orients the team around a single business objective. My team at the time, a team focused around coupon content, was the first team to make this leap a couple of years ago.

But why did we make this change?

When I joined Flipp as a co-op student many years ago, Flipp had just exited the “founder’s basement” phase of its journey. Our company grew from a team of six in Toronto’s west-end to over 450 team members over the course of eight years. With an engineering department representing over a third of that number, we started to feel the growing pains that many tech companies face — we felt like we were slowing down rather than speeding up. Features were taking longer to deliver than they had before, and as the size of the feature increased, the time to deliver it would grow exponentially.

We came to realize that the majority of time spent on any given project was spent on inter-team communication and coordination. Synchronizing multiple team schedules, especially when each of those teams were running under a two week sprint, became a challenge. Disciplined project management helped to mitigate this, but we kept looking for a solution that would remove a lot of the communication overhead altogether.

Advantages of Vertical Teams

Our engineering organization at Flipp ended up evolving fairly typically: the back-end developers owned a system, the front-end developers owned a system, and the sysadmins tied the whole thing together. But what happened when one or more teams on the path to delivery was resource-restricted, or unable to deliver their portion of the code when we needed it? We constantly found ourselves up against the Iron Triangle of Project Management.

This is where the vertical team is typically introduced and where it tends to excel: if I have an iOS developer, an Android developer, a front-end web developer, a back-end data processing developer, a SEiT and a SRE all on my team, and my job is to ensure the release of a high-quality feature that touches the whole stack, you can bet that I’m going to have a much easier time than if those individuals belonged to six separate teams.

However, restructuring a company of our size to adopt the vertical team ideology as a standard was no easy task. With the process still underway, we have come across a whole slew of challenges; from people management, to code maintenance, to coordination. Here’s our experience in a (super lengthy, three-part) nutshell.

Part 1: Human Challenges

Technical skill growth of individual team members becomes a challenge.

As our vertical team was forming, the skill growth of our more junior team members was more difficult to manage than in the previous setup. Before, a new iOS developer joining the company would automatically be part of the iOS team, and they would have access to immediate feedback on their code from a whole team of iOS developers. In the new setup, they were on a team of developers with a much more varied skill set, most of whom knew nothing about best practices in coding iOS apps.

Solution: Assign horizontal mentors for each new team member.

Sometimes the simplest solution ends up being the best one: let the experts take on the majority of the technical mentorship. Code reviews often came from outside of our team, especially early on. We would encourage those code reviews to be ultra-pedantic so that we could all rest easy knowing that we had a set of expert eyes on our code.

Team members who had unique skill sets would have frequent 1:1s with their horizontal mentors to discuss design decisions. For example, if we borrowed a team member from another team, their coach from their original team (which was typically still in a system-oriented structure) would provide technical guidance. This also helped organically form groups of experts and architects within our organization.

Technical performance evaluation becomes a very difficult task for the team lead.

It is important that each team member knows that they are being assessed by those who they work with most frequently, in order to maintain a healthy coaching relationship. But what happens when their coach is less experienced than the team member in the team member’s given specialization?

My background is in back-end development, and I found that one of the major challenges for me as the team lead of a vertical team was to assess the technical capabilities of my team members who specialized in a skill set that differed from my own. When performance evaluation time came around, it was tough for me to say with confidence whether or not the code written by any developer was good or not if it did not overlap with my skill set.

Solution: Meet regularly with team members’ horizontal mentors to gather technical feedback and track progression.

Meetings of this nature help solve two problems for the team lead — it helps them track the technical skill growth of their team members, and it helps them slowly acquire knowledge in the domains of all of their team members.

It is never the expectation that the team lead is the expert in every single technology used within the team, but it is very important that they have enough technical insight to understand the reasoning behind a design decision, know when an estimate seems off, or to be able to have back-and-forth debate with team members. Meetings with domain experts help give the team lead the confidence in more technical discussions, as an added benefit.

Vacation planning is stressful for smaller vertical teams.

Our vertical team at Flipp started out small with a full stack developer, a back-end developer, a SEiT, and a SRE assigned to our team part-time, and myself as the team lead. When larger projects came up, other team members specializing in Android and iOS development joined for the duration of the project, and then moved on to other verticals.

All this meant that each individual team member’s importance to the project was amplified by their uniqueness — they were a cross-section of domain knowledge and skill sets that made them critical for delivery.

Solution: Get all of that tasty knowledge out of each person’s head and into one central place — and keep doing it.

At Flipp, we have two established forums for disseminating business domain knowledge: a weekly meeting with all of the engineers where we discuss interesting problems experienced in our respective business areas, and extensive business-level documentation built up around each of our products.

Business domain knowledge is one thing, but we found that technical system knowledge contained within a horizontal team member’s domain (such as iOS development) was a bit trickier to spread to other members of our vertical team. One solution we implemented with a moderate degree of success was to deliberately enforce multiple code reviews, where (in the ideal case) one reviewer should already have both domain and programming language context, and another should be less familiar with either the system or the programming language. The benefits of this process were absolutely worth the time investment for us:

• A less experienced developer will gain knowledge just by reading the code, much to their own benefit and personal growth.
• Each team member slowly gains a diversified skill set that allows them to contribute outside of their area of specialization over time.
• The original developer is still provided with the security of knowing that someone with both domain expertise and programming expertise has reviewed their code.
• A less experienced developer will often ask more questions and force the original developer to reassess why they are doing things in a certain way.

This strategy is not just for the ideal case where you have multiple developers in your vertical from each specialization (so that they can share programming language context). Another member of the horizontal team on another vertical, provided the right context, can fill in the gaps if no one else on the team knows the language being employed. Over time, the team will trend towards a broader understanding of all languages being used.

Check out Part 2 on Process Challenges; or skip ahead to Part 3 on Human Challenges.

Originally published at eng.flipp.com on November 23, 2018.