We keep confusing efficacy for effectiveness
NOTE: This was originally posted on my own corner of the Internet.
I often see the same debates occur, especially around web technology. One pattern I see repeated in these debates is how focus is often on the efficacy of a certain technical approach (or tool or library or…) and the practical effectiveness is ignored.
What the eff are you talking about?
Before we get into web performance and tech, what’s the actual difference between efficacy and effectiveness?
“Efficacy can be defined as the performance of an intervention under ideal and controlled circumstances, whereas effectiveness refers to its performance under ‘real-world’ conditions.”
I remember when Peter Attia spoke about this in regards to his fish oil consumption. Is It better to take fish oil liquid, or fish oil tablets?
“The fish oil liquid is more potent and more concentrated in a higher dose, and if I take it every single day, I get higher levels in my red blood cell membranes (how we measure levels of EPA and DHA), which is the desired effect. But on average I forget to take it at least 2 times per week because the bottle needs to sit in the fridge and I forget to take it. Conversely, the fish oil capsules are easy for me to put in my pill pack next to my sink, which I never forget to take, but they are not as potent. So, while the liquid is more efficacious, the capsules may be more effective.”
— Peter Attia
Yup, we have all been there. What’s key here is both the fact that Peter actually measures the output (red blood cell membranes) and also the practical input. The details matter. How much is the difference between the two? Is the practical approach so weak as to require doing everything possible to make sure to suck it up and take the liquid? Good to go deeper, but our gut can also tell us that taking something is better than nothing.
You quickly see the efficacy vs. effectiveness pattern everywhere. Let’s bring it into our day jobs and how we debate the usage of technology and the outcomes at the other end.
The reason that benchmarks are so hard to do well, is that they measure a pure path that isn’t actually realistic at all. They are like running traffic directions algorithms without taking into account the other vehicles and traffic patterns on the road, or weather, or the time of the day, etc.
We are debating options while missing many many variables, and ones that really do matter. A subset of them are:
How easy is it to stay on a well lit path?
It is one thing for the creator of the technology to build something they have done before using their creation. But how is it for the average engineer? How variable are the outcomes based on the constraints of the system, the tooling, the documentation, the community, and on and on.
How does your app scale?
The performance of hello world is ~insignificant. How fast if your typical information, and what happens as you add features (code)? I would much develop with a system that allows me to add functionality that balances business value with performance. Compare this to something where the initial scaffolding is faster, but everyone slows over time.
What are you actually building?
The Web in particular is a general framework that has a large spectrum of use cases deployed on top of it (the spectrum mentioned here). The characteristics vary wildly as you look at the relative importance of initial load, amortizing costs for longer app sessions, requirements for collaboration, and more.
With Active Recall, which is an offline-first rich application, data is sync’d in a way that means you are basically always seeing optimistic UI operations. You are logged in by default, and the focus is on very fast actions while you use the application. These requirements land you with a very different architecture where the initial bundle isn’t the focus.
DX vs. UX
So, the next time you see another debate popup pitting DX and UX, remember that they aren’t binary variables at odds with each other. Developer productivity matters, can result in better UX, and make sure that you measuring effectiveness as it relates to your precise context.
And If you work on a platform, or on tooling, get real and do everything you can to help developers where they are, and don’t let perfection become the enemy of progress.
