In search of higher engineering productivity: A data first remote working perspective

Marc Fletcher
Jun 30, 2020 · 7 min read

At Echobox, we constantly strive for maximum productivity. We regularly review our existing processes and are open to exploring new ways of working as a team. It’s part of our DNA at Echobox to be data-driven and take a scientific approach to everything we do. We take inspiration from anywhere but remain cautious of the Halo effect, which is “the tendency to focus on the high financial performance of a successful company and then spread its golden glow to all its attributes”, even the negative ones.

A recent study about technological productivity that spanned six years with over 31,000 responses across a number of industries showed that the choice of technological methodologies or processes could lead to a hundred fold difference in the delivery of new business value confirming such exploration can reap huge rewards.

Remote working was one of those processes we decided to review. At the beginning all Echobox employees worked out of our beautiful London HQ in Notting Hill. As everyone enjoyed working together the demand for remote working was low but we were aware of anecdotes from other leading technology companies about how they increased their engineering productivity through remote working.

There is a plethora of existing research that attempts to provide insights into the impacts of where employees spend their day on productivity. These simultaneously point to the benefits and drawbacks of more remoteness between coworkers, a somewhat unhelpful picture when developing your own corporate policies. In one controlled experiment Stanford Professor Nicholas Bloom reported that getting a randomised half of call centre staff at CTrip, a 16,000 employee NASDAQ-listed Chinese travel agency, to work entirely from home over a 9 month period, resulted in a 13% performance increase (https://ebx.sh/fjiuVV, https://ebx.sh/hlNmTz). Conversely colleagues Ben Waber et al “showed that when a salesperson increased interactions with coworkers on other teams — that is, increased exploration — by 10%, his or her sales also grew by 10%” (https://ebx.sh/VZJWui). In this work, Ben Waber demonstrated that a reduction in distance between team members had the opposite effect as that observed by Nicholas Bloom. These two examples are representative of an extensive body of seemingly contradictory studies. Moreover, neither measure engineering productivity at a technology company.

This ‘paradox’ of contradictory studies may be understood if at a deeper level the results are in fact due to the non trivial interplay of many underlying factors. To clarify, remote working practices can only indirectly impact productivity. Such factors, alluded to by Bloom and many others, which include “distraction vs. focus” are loosely summarised in Appendix 1.

Importantly the contribution and strength of these factors to overall productivity, unique to each set of starting conditions, would reasonably explain why two companies within the same industry, or different departments within the same company, can experience different remote working outcomes. For example, in one of those companies the benefits of additional focus might significantly outweigh a proportional loss of collaboration, as was observed by Nicholas Bloom at CTrip.

Whilst this complexity prevents anyone from accurately forecasting the impact of remote working on productivity, the evidence that an optimal amount can be significant is clearly established in peer reviewed literature. When considering your own corporate policies the biggest risk would therefore be to avoid all experimentation, preventing an evidence based decision from being made. Sandy Pentland observed that the best-performing and most creative teams in their study sought fresh perspectives constantly, from all other groups in, and some outside, the organization — so it’s important that such experimentation is ongoing. For example, a new technology or piece of software might significantly impact any of the factors presented above.

Given the lack of representative studies on remote working, we conducted our own experiment. Although we hadn’t originally planned on sharing our results we felt it would be helpful if we shared the results of our research with the wider community, especially as most technology companies are now forced to let employees work from home due to the coronavirus pandemic.

For our experiment, our software engineers were able to self-select a single remote day per week. After six months, we increased it to two remote days per week. Remote days couldn’t conflict with team meetings or events, for which we’d require everyone in our office.

In their book Accelerate and in further detail in their peer reviewed paper Nicole Forsgren et al established a statistically significant relationship between the frequency of deployments, lead time, mean time to restore and change fail percentage to productivity. In this work, due to it’s easy availability, we have reused the frequency of deployments as our primary metric for reviewing the productivity impact of remote working. We normalise this to ‘per engineer’ to account for our team size varying between 3 and 5 members over the period of the experiment. We also contrast this against productivity of the engineering team measured as average story points completed per day per engineer. Whilst no predictive relationship has been established with story points we include it as it is a more familiar industry measure of productivity, also referred to as “velocity”.

Figure 1 shows that the proportion of remote working amongst engineers doubled between January 2018 and November 2019. Engineers felt this was primarily due to finding better ways to take advantage of the remote working time, for example by improving their work from home environments.

Figure 1. The monthly percentage of remote working in the Echobox engineering department from January 2018 to November 2019.

Figure 2 shows the number of deployments per month per developer against the proportion of remote working during this same period. We observe no statistically significant trend suggesting productivity remained static over the levels of remote working explored in this experiment. Figure 3 compares the remote working proportions, figure 1, against the average story points completed by the engineering team, per engineer, again showing no statistically significant trend with productivity over this 23 month period.

Figure 2. The number of deployments per month, per developer, against the percentage of remote working over the months shown
Figure 3. The average story points completed by the engineering team, per engineer, against the percentage of remote working over the months shown in figure 1.

Based on these results, we concluded that at the levels of remote working observed during this experiment at Echobox there was no statistically significant long term relationship against engineering productivity at Echobox. This outcome was still considered a net positive as team members reported being more motivated and less stressed due to a reduction in commuting time.

And what about Echobox’s corporate remote working policies you might ask? Accepting that remote working didn’t have a negative impact on engineering productivity at Echobox, we decided to cautiously integrate more remote working practices into our processes for non-managerial and non-leadership roles within the engineering department, primarily due to the added flexibility in hiring and retaining amazing talent from outside of our HQ in London. And we definitely plan to continue experimenting whilst forging an environment that promotes mutual trust, experimentation, improvement, knowledge transfer and education into the different factors that influence company productivity, so that everyone can make informed choices — which goes well beyond remote working practices.

If you’re interested in engineering jobs at Echobox we’re always hiring for outstanding candidates, please get in touch at https://careers.echobox.com/

About the author …

Marc Fletcher graduated with a PhD in Physics from the University of Cambridge and has been the CTO at Echobox since 2014. Whilst not jumping out of planes he’s particularly passionate about maximising productivity in high performance cross functional technology teams.

Appendix 1

Underlying factors that impact productivity of remote workers:

Echobox Insights

Echobox Insights showcases what we have learned about AI…

Echobox Insights

Echobox Insights showcases what we have learned about AI, social media and journalism in the digital age by building the first artificial intelligence that understands the meaning of content. Find out more about Echobox at www.echobox.com.

Marc Fletcher

Written by

Graduated with a PhD in Physics from the University of Cambridge and have been the CTO at Echobox since 2014.

Echobox Insights

Echobox Insights showcases what we have learned about AI, social media and journalism in the digital age by building the first artificial intelligence that understands the meaning of content. Find out more about Echobox at www.echobox.com.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store