The TypeScript Tax
A Cost vs Benefit Analysis
This article takes a more critical, data-driven approach to analyze the ROI of using TypeScript to build large scale applications.
But if you’re in the position of deciding whether or not to use it, you should have a realistic understanding of both the benefits and the costs. Will it have a positive or negative impact?
In my experience, it has both, but falls short of positive ROI. Many developers love using it, and there are many aspects of the TypeScript developer experience I genuinely love. But all of this comes with a cost.
My understanding of TypeScript, including its benefits, costs, and weaknesses have deepened considerably. I’m saddened to say that it wasn’t as successful as I’d hoped. Unless it improves considerably, I would not pick TypeScript for another large scale project.
What I Love About TypeScript
I’m still long-term optimistic about TypeScript. I want to love TypeScript, and there’s a lot I still do love about it. I hope that the TypeScript developers and proponents will read this as a constructive critique rather than a hostile take-down piece. TypeScript developers can fix some of the issues, and if they do, I may repeat the ROI analysis and come to different results.
Static types can be very useful to help document functions, clarify usage, and reduce cognitive overhead. For example, I usually find Haskell’s types to be helpful, low-cost, pain-free, and unobtrusive, but sometimes even Haskell’s flexible higher-kinded type system gets in the way. Try typing a transducer in Haskell (or TypeScript). It’s not easy, and probably a bit worse than the untyped equivalent.
I love that type annotations can be optional in TypeScript when they get in the way, and I love that TypeScript uses structural typing and has some support for type inference (though there’s a lot of room for improvement with inference).
TypeScript ROI in Numbers
I’m going to rate TypeScript on several dimensions on a scale of
-10–10 to give you a better sense of how well suited TypeScript may or may not be for large scale applications.
Greater than 0 represents a positive impact. Less than 0 represents a negative impact. 3–5 points represent relatively strong impact. 2 points represents a moderate impact. 1 point represents a relatively low impact.
These numbers are hard to measure precisely, and will be somewhat subjective, but I’ve estimated the best I can to reflect the actual costs and rewards we saw on real projects.
With margin-of-error so broad, I gave up on objective quantification, and instead focused on feature delivery pace and observations of where we spent our time. You’ll see more of those details in the ROI point-by-point breakdown.
Because there’s a lot of subjectivity involved, you should allow for a margin of error in interpretation (pictured in the chart), but the over-all ROI balance should give you a good idea of what to expect.
I can already hear the peanut gallery objections to the small benefits scores, and I don’t entirely disagree with the arguments. TypeScript does provide some very useful, powerful capabilities. There’s no question about that.
Let’s look at each point in more detail.
In fairness, if you use default parameters to provide type hints, you don’t need to supply the annotations for TypeScript code, either, which is a great trick to reduce type syntax overhead — one of the overhead costs of using TypeScript.
TypeScript’s tooling for these things is arguably a little better, and more all-in-one — but it’s not enough of an improvement to justify the costs.
Even with the best inline documentation in the world, you still need real documentation, so TypeScript enhances, rather than replaces existing documentation options.
Refactoring. In most cases, if you can gain a significant benefit from TypeScript in your refactoring, that’s often a code smell indicating that your code is too tightly coupled. I have written an entire book on how to write more composable, more loosely coupled code, called “Composing Software”. If TypeScript is saving you a lot of refactoring pain, there’s a good chance tight coupling is still causing you a lot of other avoidable problems. I strongly suggest reading the book, particularly the chapter “Mocking is a Code Smell”, which provides a lot of information on the causes of tight coupling and some best practices that can help you avoid them.
On the other hand, some companies run very large ecosystems of connected projects sharing the same code repository (e.g., Google’s famous monorepo). Using TypeScript enables them to upgrade API design choices to account for better designs and new use-cases. The developers responsible for those upgrades are also responsible for ensuring that their library changes don’t break any of the software in the monorepo that depends on those libraries. TypeScript may offer significant time savings for this very limited subset of TypeScript users.
I say very limited subset, because giant, closed monorepo ecosystems are the exception, rather than the rule. The process might scale across Google, but can’t scale to repositories that the library authors are not aware of. Making breaking changes to library APIs used by a broader ecosystem can break code you don’t even know exists.
In traditional, more decentralized library ecosystems, people avoid breaking changes to APIs, and instead create new features following the open/closed principle (APIs are open for extension, and closed to breaking changes). This is how the web platform itself has mostly evolved, with a few exceptions. This is why React still supports features that have been replaced by better options since React 0.14. React evolves and adds great new features, radically improving the developer experience without breaking old functionality. For instance,
class components will still be supported by React, even after the much improved React Hooks API matures.
That makes changes across the whole ecosystem optional, rather than required. Teams can upgrade their software gradually, on an as-needed basis rather than heaping a whole-ecosystem code change project on the library team.
Even in cases where whole ecosystem code changes are required, type inference and automated codemods can help — no TypeScript required.
I initially mentally scored refactoring a zero and left it off the list because I strongly favor the open/closed approach, inference, and codemods. However, some teams are getting real benefits from it under limited circumstances.
Type safety doesn’t seem to make a big difference. TypeScript proponents frequently talk about the benefits of type safety, but there is little evidence that type safety makes a big difference in production bug density. This is important because code review and TDD make a very big difference (40% — 80% for TDD alone). Pair TDD with design review, spec review, and code review, and you’re looking at 90%+ reductions in bug density. Many of those processes (particularly TDD) are capable of catching all of the same class of bugs that TypeScript catches, as well as many bugs that TypeScript will never be able to catch.
TypeScript is only capable of addressing a theoretical maximum of 20% of “public bugs”, where public means that the bugs survived past the implementation phase and got committed to the public repository, according to Zheng Gao and Earl T. Barr from University College London, and Christian Bird from Microsoft Research.
The authors of this study think they’ve underestimated the impact of TypeScript because they assume that all the other quality measures have already been applied, but they made no effort to judge the quality of the other bug prevention measures. They acknowledge the variable, but leave it entirely out of the calculations.
In my experience, the vast majority of teams have partially applied some measures, but rarely applied all important bug prevention measures well. On my teams, we use design review, spec review, TDD, code review, lint, schema validation, and company-sponsored mentorship, which all have dramatic impacts on bug density, reducing type errors to very near zero.
In my experience, all but linting have a larger impact on code quality than static types. In other words, I’m starting from a much stricter definition of zero than the authors of the paper.
If you have not properly implemented those other bug prevention measures, I have no doubt you’ll see 15% — 18% reduction in bug density using TypeScript alone, but you’ll also completely miss 80% of the bugs until they get to production and start causing real problems.
Some will argue that TypeScript provides realtime bug feedback, so you can catch the bugs earlier, but so do type inference, lint, and TDD (I set up a watch script to run my unit tests on file save, so I get very near immediate, rich feedback). You may argue that these other measures have a cost, but because TypeScript will always miss 80% of bugs, you can’t safely skip them either way, so their cost applies to both sides of the ROI math, and is already factored in.
The study looked at bugs that were known in advance, including the exact lines that were changed to fix the bugs in question, where the problem and potential solutions were known prior to introduction of typings. What this means is that even knowing that the bugs existed in advance, TypeScript was unable to detect 85% of public bugs — catching only 15%.
Update: We’re going to give TypeScript the absolute theoretical maximum benefit of the doubt and use 20% in our calculations, to drive home the point about exponentially diminishing returns.
Why are so many bugs undetectable by TypeScript, and why did I call that 20% reduction a “theoretical maximum” effect? For starters, specification errors caused about 78% of the publicly classified bugs studied on GitHub. The failure to correctly specify behaviors or correctly implement a specification is the most common type of bug by a huge margin, and that fact automatically renders an overwhelming majority of bugs impossible for TypeScript to detect or prevent. In “To Type or Not to Type”, the study authors identified and classified a range of “ts-undetectable” bugs.
“StringError” above are the classification of errors where the string was the right type, but contained the wrong value (like an incorrect URL). Branch errors and predicate errors are logic errors that led to the wrong code paths being used. As you can see there are a variety of other errors that TypeScript just can’t touch. There’s little potential that TypeScript will ever be capable of detecting more than 20% of bugs.
But a 20% sounds like a lot! Why doesn’t TypeScript get much higher bug prevention points?
Because there are so many bugs that are not detectable by static types, it would be irresponsible to skip other quality control measures like design review, spec review, code review, and TDD. So it’s not fair to assume that TypeScript will be the only thing you’re employing to prevent bugs. In order to really get a sense of ROI, we have to apply the bug reduction math after discounting the bugs caught by other measures which were not adequately factored in by the study authors.
Imagine your project would have contained 1,000 bugs with no bug prevention measures. After applying other quality measures, the potential production bug count is reduced to 100. Now we can look at how many additional bugs TypeScript would have prevented to get a truer sense of the bug catching return on our TypeScript investment. Close to 80% of bugs are not detectable by TypeScript, and all TypeScript-detectable bugs can potentially be caught with other measures like TDD.
- No measures: 1000 bugs
- After other measures: 100 bugs remain — 900 bugs caught
- After adding TypeScript to other measures: 80 bugs remain — 20 more bugs caught
Some people argue that if you have static types, you don’t need to worry about writing so many tests. Those people are making a silly argument. There is really no contest. Even if you’re going to employ TypeScript, you still need the other measures.
In this scenario, reviews and TDD catch 900/1,000 bugs without TypeScript. TypeScript catches 200/1,000 bugs if you skip reviews and TDD. You obviously don’t have to pick one or the other, but adding TypeScript after applying other measures leads to a very small improvement due to exponentially diminishing returns.
Airbnb recently reported a 38% reduction in bugs by adding TypeScript to their development process. How could that be? According to this article, that should be impossible, right? That’s not how the math works. We’re dealing with percentages, averages, and diminishing returns, not concrete values.
The study this article relies on represents averages, and the presence or absence of other quality measures impacts the percentage of bugs remaining that TypeScript could address.
The more ts-undetectable bugs the other measures address, the higher the percentage of remaining bugs TypeScript can address, but those other measures also reduce the total number of remaining bugs for TypeScript to address. So the percentage might go up, but the total number of bugs caught might change only a little.
As of this writing, they have not released their methodology or reported what other bug reduction measures they’re employing, but my guess is that they’re employing some form of design/spec review process to reduce the share of specification bugs that make it into their code in the first place.
In other words, when you eliminate a lot of bugs that TypeScript can’t help with, TypeScript can provide a higher percentage of bug reduction to the remaining bugs.
This result doesn’t change the 20% max value if TypeScript is the only quality control measure, and doesn’t invalidate the point about exponentially diminishing returns.
Instead, it implies that Airbnb may have better-than-average design or spec review (or both), coupled with lower-than-average automated code coverage — perhaps missing unit test coverage, functional test coverage, or both. Proper unit test coverage can catch close to 100% of the bugs that static types can catch, along with a lot of bugs TypeScript can’t catch.
Most teams have little or no design/spec review process implemented. Even having an engineer look at mock-ups with a critical eye before handing them off to a developer to implement would be better than average. Many teams don’t have any formal design review process at all.
Here’s what their TypeScript benefit chart might look like:
TypeScript is still catching just 38 out of 1,000 potential bugs, but since most of the potential bugs are caught by previous steps in the pipeline (like people reviewing mockups before they go to a developer to implement), TypeScript can address a larger share of remaining bugs. In this case, 18 more bugs than teams missing Airbnb’s additional code quality measures.
The diminishing returns math could only be completely invalidated if TypeScript could catch a much larger share of all bugs: closer to 75%+, because at that stage, it might be viable to replace other expensive parts of the quality control process, like code review or TDD.
It would be interesting to learn exactly how many bugs Airbnb caught during the conversion to TypeScript, to learn about the classification of the bugs that TypeScript couldn’t prevent, to learn the bug density (and how they calculated it), and to learn what other quality control measures they already employ.
Keep in mind: I’m not arguing against using TypeScript. I’m arguing for people to consider the costs and benefits and make a rational, informed decision that’s right for you and your team. Some products require stricter quality control, and it may be worth the extra cost to eliminate 18 more bugs out of a thousand. For example, if your code powers critical parts of the self-driving system for a Tesla, I hope you’re using static types along with all the other quality measures, because the cost of bugs is much higher. Each team should conduct their own ROI analysis and make the decision that is right for them.
Having implemented quality control systems on large scale, multi-million dollar development projects, I can tell you that my expectations for effectiveness on costly system implementations are in the territory of 30% — 80% reductions. You can get those kinds of numbers from any of the following:
- Design and Spec Review (up to 80% reduction)
- TDD (40% — 80% reduction of remaining bugs)
- Code Review (an hour of code review saves 33 hours maintenance)
It turns out that type errors are just a small subset of the full range of possible bugs, and there are other ways to catch type errors. The data is in, and the result is very clear: TypeScript won’t save you from bugs. At best, you’ll get a very modest reduction, and you still need all your other quality measures.
Type correctness does not guarantee program correctness.
It looks like the benefits are not living up to the TypeScript hype. But those can’t be the only benefits, right?
To figure that out, we need to take a closer look at the costs of TypeScript.
On the other hand, if you only need to hire one or two developers, using TypeScript may make your opening more attractive to almost half the candidate pool. For small projects, it may be a wash, or even slightly positive. For teams of hundreds or thousands, it’s going to swing into the negative side of the ROI error margin.
TypeScript could improve on this cost by providing better documentation and discovery of TypeScript’s current limitations, so developers waste less time trying to get it to behave well on higher order functions, declarative function compositions, transducers, and so on. In many cases, a well-behaved, readable, maintainable TypeScript typing simply isn’t going to happen. Developers need to be able to discover that quickly so that they can spend their time on more productive things.
Ongoing Mentorship: While people get productive with TypeScript pretty quickly, it does take quite a bit longer to get feeling confident. I still feel like there’s a lot more to learn. In TypeScript, there are different ways to type the same things, and figuring out the advantages and disadvantages of each, teasing out best practices, etc. takes quite a bit longer than the initial learning curve.
For example, new TypeScript developers tend to over-use annotations and inline typings, while more experienced TypeScript developers have learned to reuse interfaces and create separate typings to reduce the syntax clutter of inline annotations. More experienced developers will also spot ways to tighten up the typings to produce better errors at compile time.
This extra attention to typings is an ongoing cost you’ll see every time you onboard new developers, but also as your experienced TypeScript developers learn and share new tricks with the rest of the team. This kind of ongoing mentorship is just a normal side-effect of collaboration, and it’s a healthy habit that saves money in the long term when applied to other things, but it comes at a cost, and TypeScript adds significantly to it.
Typing Overhead: In the cost of typing overhead, I’m including all the extra time spent typing, testing, debugging, and maintaining type annotations. Debugging types is a cost that is often overlooked. Type annotations come with their own class of bugs. Typings that are too strict, too relaxed, or just wrong.
This cost center has gone down since I first explored it, because many third party libraries now contain typings, so you don’t have to do so much work trying to track them down or create them yourself. However, many of those typings are still broken and out-of-date in all but the most popular OSS packages, so you’ll still end up backfilling typings for third party libraries that you want type hints for. Often, developers try to get those typings added upstream, with widely varied results.
You may also notice greatly increased syntax noise. In languages like Haskell, typings are generally short one-liners listed above the function being defined. In TypeScript, particularly for generic functions, they’re often intrusive and defined inline by default.
Instead of adding to the readability of a function signature, TypeScript typings can often make them harder to read and understand. This is one reason experienced TypeScript developers tend to use more reusable typings and interfaces, and declare typings separately from function implementations. Large TypeScript projects tend to develop their own libraries of reusable typings that can be imported and used anywhere in the project, and maintenance of those libraries can become an extra — but worthwhile — chore.
Syntax noise is problematic for several reasons. You want to keep your code free of clutter for the same reasons you want to keep your house free of clutter:
- More clutter = more places for bugs to hide = more bugs.
- More clutter makes it harder to find the information you’re looking for.
Clutter is like static on a poorly tuned radio — more noise than signal. When you eliminate the noise, you can hear the signal better. Reducing syntax noise is like tuning the radio to the proper frequency: The meaning comes through more easily.
Syntax noise is one of the heavier costs of TypeScript, and it could be improved on in a couple ways:
- Better support for generics using higher-kinded types, which can eliminate some of the template syntax noise. (See Haskell’s type system for reference).
- Encourage separate, rather than inline typings, by default. If it became a best practice to avoid inline typings, the typing syntax would be isolated from the function implementation, which would make it easier to read both the type signature and the implementation, because they wouldn’t be competing with each other. This could be implemented as a documentation overhaul, along with some evangelism on Stack Overflow.
I still love a lot of things about TypeScript, and I’m still hopeful that it improves. Some of these cost concerns may be adequately addressed in the future by adding new features and improving documentation.
However, we shouldn’t brush these problems under the rug, and it’s irresponsible for developers to overstate the benefits of TypeScript without addressing the costs.
TypeScript can and should get better at type inference, higher order functions, and generics. The TypeScript team also has a huge opportunity to improve documentation, including tutorials, videos, best practices, and an easy-to-find rundown of TypeScript’s limitations, which will help TypeScript developers save a lot of time and significantly reduce the costs of using TypeScript.
I’m hopeful that as TypeScript continues to grow, more of its users will get past the honeymoon phase and realize its costs and current limitations. With more users, more great minds can focus on solutions.
As TypeScript stands, I would definitely use it again in small open-source libraries, primarily to make life easier for other TypeScript users. But I will not use the current version of TypeScript in my next large scale application, because the larger the project is, the more the costs of using TypeScript compound.
Updated: 2019–02–11 — Clarified how Airbnb could report a 38% reduction in bugs after adding TypeScript without invalidating any of the claims made in this article. TL;DR — better than average design/spec review + lower than average test coverage could lead to a higher than average ts-addressable % of remaining bugs, without substantially changing the total percentage of bugs prevented by all measures.
Updated: Jan 26, 2019 — added “Refactoring” benefit.
Updated: Jan 26, 2019 — clarified bug reduction math, increased bug reduction benefit from 8% of remaining bugs to 20% to give TypeScript the maximum benefit of the doubt and demonstrate that it still doesn’t materially impact ROI.
He enjoys a remote lifestyle with the most beautiful woman in the world.