Clearly Communicating Performance Improvements

At Adobe, I work on the engineering team responsible for building Adobe Experience Platform Web SDK. This SDK supersedes some legacy libraries that Adobe provided customers to implement on their websites. One of the primary goals of the new SDK was to make it faster than the legacy libraries. Once we gathered and crunched the numbers, we published a blog post communicating these performance improvements to customers.

I wanted to use our benchmark data to declare that the SDK was, for example, 60% faster than the legacy libraries. As I started to calculate the exact percentage, I caught myself wondering if readers would actually understand what that percentage meant in terms of time. For example, if I were to state that the new SDK is 60% faster, and a reader familiar with the legacy libraries knows it takes roughly 1000 milliseconds to complete a given task, would the reader be able to articulate how long it takes the new SDK to perform that same task?

I took a variation of this question to Twitter.

My question wasn’t intended as a trick question. It wasn’t to find out if my followers were good mathematicians. What I really wanted to know is how a sample of readers might perceive these terms and numbers, regardless of whether their perception was correct.

I thought the results might be mixed, but I was still surprised. How could a seemingly simple question provoke such a split result?

The Term “Faster”

I’ve had some time to think about this question and discuss it with others, and it seems the complication typically stems from the term “faster”. The term “faster” refers to speed, or, more accurately, the rate at which a task can be performed. It does not refer to reduction in time (I won’t hold it against you if you disagree). Let’s look at some examples of how we measure speed.

If we’re talking about the speed of a car, we’re talking about how many times the car can travel one mile (or kilometer) in one hour (reported as miles per hour).

If we’re talking about the speed of a printer, we’re talking about how many times the printer can print one page in one minute (reported as pages per minute).

If we’re talking about the speed of an internet connection, we’re talking about how many times the connection can transmit one megabit of data in one second (reported as megabits per second).

Let’s now use this information to solve the question I posed over Twitter. Thing X can perform 1 task in 1 hour. Thing Y can perform 1.33 tasks in 1 hour. If we take our trusty performance equation:

((NEW - OLD)/OLD) * 100

And plug in our numbers:

((1.33 - 1)/1) * 100

Beep. Boop. Beep. Thing Y is 33% faster than Thing X. Also, Thing Y takes 25% less time than Thing X to complete a single task. Notice that the percentage values are not the same because we are talking about two different metrics: speed and time.

Let’s look at it another way. If I told you that Thing X and Thing Y were cars and their task was to drive 60 miles, then Thing X would be traveling at 60 miles per hour while Thing Y would be traveling at 80 miles per hour in order to reach 60 miles in 45 minutes. Again, Thing Y is 33% faster and takes 25% less time than Thing X to complete the task.

Here’s one more way to consider how the term “faster” refers to speed instead of time. If I told you that Battery Y charges 100% faster than Battery X, it doesn’t mean Battery Y charges in 0 time. Instead, it means Battery Y charges at twice the rate of Battery X, or in other words, half the time.

Communication = Correctness + Clarity

I don’t mean to suggest this was all obvious to me when I encountered the problem. I had to think through it. And that’s the point.

If I write an article stating that our new SDK is 60% faster than legacy libraries and our customers know that the legacy libraries take 1000 milliseconds to perform a task, how many customers will clearly understand how long the new SDK would take to perform the same task? Adding “60% faster!” to a slide deck might be mathematically correct and may even convey that it was a meaningful improvement, but it’s not particularly clear communication. Communication is about correctness and clarity.

A Better Form Of Communication

So what’s the alternative? Avoid the possibility of miscommunication by making it easy for the reader. Does the reader want to compare how long it takes two things to perform a given task? Then stay within the metric the reader desires — time — and avoid the “% faster” verbiage entirely. For example:

The new SDK took 1,304 milliseconds to reach First Contentful Paint. Legacy libraries took 1,863 milliseconds to reach First Contentful Paint. This results in the new SDK taking 70% the amount of time as legacy libraries to reach First Contentful Paint.

It might not roll off the tongue, but notice we’re not requiring the reader to translate between speed and time metrics. All numbers and percentages are in terms of time.

One possible exception to this rule comes to mind. If Thing Y takes half the time as Thing X to complete a task, I find it reasonable to say “Thing Y is twice as fast as Thing X”. This statement is quite simple for the reader to translate between speed and time (“twice is fast” equals “half the time”). However, if Thing Y takes anything other than half the the time as Thing X to complete the task, my advice is to avoid “% faster” or “X times as fast” if at all possible. Your readers will appreciate you twice as much.