Reacting to a React Performance Analysis

[ Jul. 6, 2015: Updated with response from Paul Lewis. ]

On Friday, Google employee Paul Lewis (@aerotwist) published a blog post with results from his measurement of React’s DOM performance relative to vanilla JavaScript on a particular benchmark he constructed.

Others have already raised questions about the particular choice of benchmark (append operations to a single large list), observing that this isn’t representative of the kind of complex DOM state management issues that arise in real applications, and that this also probably isn’t how one should implement a large scrolling list view.

But in my opinion there is a more fundamental issue with this post than the choice of what to benchmark. The study did not include source code for the benchmark. Instead there was this note at the end of the post:

I see this as more than just a minor inconvenience and don’t consider this a reasonable approach to making the source code available, for a few reasons:

First and foremost: Do others get the same timing results running the original benchmark code on an identical test setup (hardware, operating system, React library versions, etc.)? It’s impossible to answer that definitively without the original source code (and any other auxiliary artifacts such as build definitions, library dependencies, etc.) needed to build and run it made available immediately at publication time.

But there are other questions of equal or greater importance:

• Is the implementation of the benchmark correct?
• Is the technique used to gather timing measurements reasonable?
• Can the implementation be optimized? If so, is it relatively easy or difficult to make the changes to add those optimizations?
• Is the implementation well organized and modular?
• Does the implementation follow guidelines for idiomatic, performant React code?
• Can an expert spot critical performance issues and respond in Twitter time?

It is simply not possible to answer any of these questions definitively or have that constructive real-time dialog without having the original source code available immediately at publication time.

These questions are primarily about reproducibility, a fundamental principle of the scientific method. Performance studies (even mere blog posts) published without making the benchmark source code available are not systematically reproducible or falsifiable, and hence are more hearsay than science. This seems like a waste of what was clearly a substantial and well intentioned effort by Mr. Lewis.

The original post also raises uncomfortable questions about how individual opinions expressed in a personal blog are connected with one’s employer. Google and Facebook are clearly competitors, and so React is most certainly a competitor’s product for Mr. Lewis, regardless of whether he works directly on competing products and regardless of whether he was thinking of React that way when writing his analysis. While all benchmarking efforts should aspire to fairness and good science, I hold the view that one should make an extra effort to enable a fair response when critiquing a competitor’s product. The only way to enable a fair response from the React team was to make the source code available immediately at publication time. If the code was not yet cleared for publication the responsible thing was to wait until getting that clearance. There is no reason to think that this benchmark code has any IP that is in any way material to Google. If Google’s corporate bureaucracy makes getting such clearance too difficult or slow, the author should work within Google to fix that process.

Detailed performance analysis, technical criticism, profiling and putting claims to the test are all great things and can be valuable contributions to the community. But when you take the time and effort to do such important work, please also make the additional effort to ensure it is good science: make your sources, test data, build configuration, and anything else a competent practitioner needs to reproduce, scrutinize, learn from, respond to and hopefully build on your work available at publication time.

An analysis like this can and should be the start of a dialog with the community, not the end of it. Thankfully, in a fork of the original Twitter thread on the the same day, we got a great example of exactly how this process can and should work from @paul_irish and @floydophone.

Disclaimers / Disclosures: I do not work for Facebook. I use React for some of my personal and consulting projects. Opinions expressed are solely my own.

Update (Jul. 6, 2015): Paul Lewis (@aerotwist) responded magnanimously here and on Twitter, and made the source code for his benchmarks available on github: