The Whys and Hows of Engineering Reporting
by Aditya Agarwal, Matt Eccleston, Vivian Guo, and Anantha Kancherla
More than ever, research and development (R&D) has become a key differentiator for technology companies, and as a result, a bigger line item in operating expenses. But how can leaders assess how R&D and engineering developments are tied to overall business outcomes? Unlike finance or go-to-market updates which are consistently reported on each quarter, we have noticed there is not a standardized approach to reporting on engineering. As engineering organizations become more complex, it becomes increasingly challenging to have conversations at the right altitude or backed by the appropriate set of data. In board conversations, we also find that engineering can sometimes be a “black box” for financially-focused or less technical investors.
Recently, ICONIQ Growth’s Technical Advisory Board came together to outline some best practices and frameworks to guide quarterly reporting across engineering teams, which we believe will drive not only a common understanding of key engineering focus areas but also continued conversation and alignment inside and outside the boardroom. While the structure and focus will obviously vary based on the stage of each company, we believe if a company’s R&D budget is over $10 million annually, the management team, board of directors, and engineering teams will all benefit significantly from consistent quarterly reporting on the following four areas:
1. Key Engineering Metrics
Each meeting should start with a summary of the key engineering metrics and objectives and key results (OKRs) tracked on a quarterly and annual basis. Starting in this way forces teams to think through what are the actual key performance indicators (KPIs), focuses attentions on results instead of activities, and level-sets the conversation that follows. Some of the key categories we recommend covering include R&D spend, people, engineering efficiency, and code quality.
2. Engineering Allocation
As an engineering organization grows, different types of questions and challenges start to emerge around the investments in time and people your organization is making and the ROI that comes from the ever increasing engineering team.
It’s critical to have a framework in place that allows the company to talk about and prioritize engineering investments in a way that makes sense for engineering internally, while also being understandable for the rest of the business stakeholders. We recommend a framework that focuses the conversation on the levers and choices available to the business by categorizing engineering allocation into four key buckets: New Capabilities, Quality Improvements, Internal Productivity, and Keeping the Lights On (KTLO). You can read more about this framework here. With such a framework established, you can then examine the key milestones/roadmap and have meaningful conversations about strategic choices and any key tradeoffs that are worthy of discussion.
3. Hiring Update
It’s critical for organizations to hire the right talent at the right time. For engineering teams, it’s especially important that the board and management understand that engineering hiring isn’t about hitting a headcount number. Rather, it’s about identifying the easy hires that can be accomplished at scale and the much harder hires you will really have to devote time to finding.
To ensure the right level of support, we recommend presenting a picture of the 12-month forward engineering org chart and spending some time discussing the open leadership (director and above) positions. Beyond roles, it’s also important to identify the key expertise gaps (e.g., testing, machine learning, etc.) in your organization. As with other function updates, it’s great to make time available to celebrate recent hiring wins. On the other hand, it’s also important to stay aware of how much your recruiting engine works on replacing talent (i.e., regretted/non-regretted attrition).
4. Engineering Efficiency
Just as sales teams measure quotas and ramp time, it’s important for the engineering organization to measure developer productivity and identify bottlenecks. While these metrics will vary across companies, metrics that allow management to understand and track the revenue / FTE cost, release time, and other metrics around developer velocity on a quarterly and trended basis are critical. Presenting a summary of the key development bottlenecks, planned remediation, and technology risks also allow engineering teams to receive leadership and cross-functional support where needed.
In our opinion, there is not a single or set of metrics that can accurately track and report on developer efficiency across companies. Instead, developer efficiency can be viewed similar to a sales funnel, with key metrics that can be tracked at each stage of the funnel (e.g., writing code, code review, testing, deployment, maintenance). For example, you may want to track the time from outlining requirements to code complete (writing code), percentage of code coverage (testing), and percentage of roadmap shipped on time (deployment).
One table stakes metric we recommend all organizations start tracking and reporting on is self-reported developer productivity. This metric should be based on engineering surveys that ask key questions including: Do you have the right tools and processes to do your work? Where are you spending most of your time?
Surveys such as these may feel like a fuzzy and self-reported benchmark, but we believe represent a critical metric that cuts across all aspects of developer productivity. Above all, it’s more important to focus on how you trend over time on this metric, rather than your current score. Measuring individual developer productivity will, for better or worse, start altering behavior. For example, if you measure code review turnaround time and share these scores with engineers, over time they may code faster but probably with lower quality. A good way to mitigate these unintended consequences is by looking at aggregate rather than individual scores.
While these metrics are intended to solicit discussion around key topics like engineering spend, headcount, and efficiency, we also believe that just having quarterly reporting will force engineering teams to be more introspective as they prepare these metrics each quarter. Rather than tracking every single metric, it’s most critical to start building in the practice of regular reporting with the goal of improving on just a select few metrics over time.
These challenges demand different solutions across companies. Nevertheless, the sharing of knowledge will only expand everyone’s toolbox.
To get started developing or optimizing your engineering reporting, you can download the “ICONIQ Reporting Guide — Engineering” PowerPoint template here.
