Scala Almost Succeeded
And it still could
Scala is an academic language. It attracted attention from the industry because it promised better scalability for multi-core computers at a time when Java was stalling.
Then, out of nowhere around 2015, Scala experienced hyper-growth because of Spark, the general-purpose cluster computing framework.
By “success,” we mean the number of developers using Scala in the industry. If the Redmonk ratings and frequency of job adverts are anything to go by, Scala has since plateaued. With that said, Scala has managed to find its way into almost all JVM systems — even if only a part of that system is a library or framework written in Scala.
This article offers an opinion on why Scala’s early success may be coming to an end and what I believe the Scala community needs to do to get back into a phase of growth.
Disclaimer: This is not an attack on any individual. If you feel that I have spoken negatively about you as a person, please contact me and I shall try to put your mind at ease.
Scala developers consistently complain about compile times, quality of tooling, and which language features are too complex to be included in their code base.
To be precise, developers complain about slow build times, which is the time it takes to build a project from cold. This also includes startup time and resolving/downloading dependencies.
Complaints about tooling can be anything ranging from the complexity of the build tool to the lack of support for advanced language features in the IDE to the difficulty to publish a project.
The thing that terrifies most Scala developers — and causes the fiercest debate — is the use of the Shapeless generic programming library. I am partly responsible for this because I once gave a Scala eXchange workshop championing its use. I since regret this. Please forgive me!
In 2019, we saw good progress from Lightbend on compilation performance: The Scala 2.13 compiler is the fastest Scala compiler yet and the proprietary GraalVM technology can make it even faster.
Lightbend have, to their credit, doubled down on micro optimisations with the assistance of their commercial customers. That’s a laudable achievement because Lightbend are primarily concerned with their own products and the Scala Team is very small (only four full-time developers) and they are also responsible for the maintenance of sbt.
For anybody needing even faster compiles, triplequote’s Hydra has got you covered.
Twitter are rewriting the compiler with RSC (the Reasonable Scala Compiler) and Kentucky Mule was proposed by a former Scala compiler team member, so there are still further options that could be taken.
It took a long time for the academics at EPFL to accept that compile times and tooling were a problem. The Scala Center was set up in 2016 to “coordinate and develop open source libraries and tools for the benefit of all Scala users.”
I was an early supporter, but I lost faith when they spent their first three years working on pet projects and vapourware such as Scaladex, Spores, Scastie, Scala Native, and Dotty instead of addressing imminent problems.
In 2019, we saw more projects being delivered by Scala Center that are relevant: They are writing faster build tools, refactoring tools, and alternatives to IntelliJ.
Unfortunately, their lack of industrial engineering experience shows in their decision to invent new solutions rather than to contribute to incumbent tooling that is in use in production. For example, we can criticise bloop by noting that unless the benefits are seen in sbt, then nobody will benefit. And we can criticise the creation of Metals by noting that 10% of Scala users (circa 2018) are using ENSIME and they have dismissed the contributions of literally hundreds of volunteers.
This follows a tradition of reinvention that goes back to Scastie (reinventing Scalafiddle) and Scala Native (competing with Graal). If the Scala community wants better tooling, then they must hold the Scala Center to account: They were set up to “coordinate and develop” — not to ignore what exists and develop new things.
I could also say that if the community wants better tooling, then they have to build it, but I don’t think this is fair to the volunteers who dedicate their time to it. It is a full-time job and nobody should have to sacrifice their family time for that.
In 2018, Jon Pretty released the Magnolia macro library. Teams are replacing their most complicated Shapeless “typeclass derivation” code with simple Magnolia alternatives, which can be 100 times faster to compile.
Looking at how everybody is responding to the main problems, Scala is in good shape!
Scala Center and Lightbend are delivering value, and the community is stepping up to fix gaps in the ecosystem. So why is Scala hitting a plateau? Is it all too little, too late?
I believe that the problem is multi-faceted:
- It is really hard to hire experienced Scala developers.
- While Scala has been burning time, Java has caught up.
- Scala 3 and the Osborne Effect.
Hard to Hire Scala Developers
A consequence of a hyper-growth phase is that the vast majority of Scala developers are relative beginners. This, on its own, can be fixed with access to good training. But Scala’s inherent complexity means that a training course is often not sufficient for a developer to be able to navigate all the language features, runtime environment, and the fragmented, often tribal ecosystem.
One of Scala’s selling points was that it is compatible with Java libraries. This is a double-edged sword because although it helps transition Java developers to Scala, it makes it harder for people who don’t know Java to learn Scala. One must learn about the JVM to use much of the runtime tooling (profilers, debuggers, optimisation, etc.).
The many different styles of Scala mean that it is not enough to hire or train somebody in “Scala.” Play, Typelevel, Scalaz are all very different technology stacks, creating niches within a niche. Entrenched politics have caused these stacks to drift further apart. They must come together rather than fight. Unfortunately, in 2019 we saw the opposite: Lightbend and Scala Center actively excluded projects and individuals from the community because they contributed to the “wrong” stack.
The politics are so dirty that several members of the Scala compiler team have recently engaged in “no platforming” of a prominent contributor having keynotes removed from Scala Days (UPDATE: and Scala eXchange), with pressure applied to all conference organisers. UPDATE: And a former Lightbend executive has called for a boycott of recruiters who work with that contributor.
Then there is Typelevel. In my opinion, they are the worst thing to happen to the Scala community. They are a small but very vocal minority who have decided that they are morally superior and represent the functional programming community. But nobody elected them, and a lot of people are unhappy about that.
The drama in itself pushes people away from Scala. It’s also a huge disincentive for contributors, lest they be seen to be picking a side, or worse, finding themselves unemployable.
Java Has Caught Up
Java now has much better support for concurrency and functional programming. There are proposals to add pattern matching to the language.
Android, which accounts for 80% of all smartphones on the planet, runs a stripped-down version of Java. Scala intentionally dropped support for Android, meaning that any developers targeting the largest App Store in the world have had to find alternative languages. Many have returned to Java, while others have moved to Kotlin as a “better Java.”
I am not aware of any plans for Scala to support Android again, but it seems like a mistake to exclude the largest market in the world.
The Osborne Effect
The Osborne Effect is the myth that the Osborne Computer Corporation went out of business in the 1980s because they announced Version 3 of their personal computer while Version 2 was still in the shops. Although a myth, the effect is very real. Scala 3 has been an academic research project for nearly five years under the codename “Dotty” and is supposed to arrive within the next few years. At that point, Scala as you know it will be decommissioned. So why bother investing in Scala 2 when you can wait for Scala 3 and make do with Python until then?
The changes in Scala 3 are so dramatic that all Scala developers will need to be retrained and all code bases will need to undergo a migration. Personal contacts tell me that their migration estimates range anywhere from “a weekend” on small projects to one month on medium-sized projects (100K lines of code) to years of effort for larger projects (millions of lines of code). And that assumes that the free software ecosystem has already been migrated, which is relying on a lot of volunteers mobilising for the common cause.
All the compiler and tooling improvements that have been made in the last few years will be thrown away. They are targeting the Scala 2 compiler. It’s not even clear if the IDEs and text editors will be ready (the effort to support significant whitespace and new grammatical constructs cannot be overstated and is too low-level to be solved by LSP).
The Scala 3 compiler is slower than the Scala 2 compiler and is not capable of building core ecosystem libraries. As a community volunteer contributor, you are expected to rewrite all of your projects and it might not be possible to rewrite some. A lack of free time will leave gaps in the ecosystem.
Scala 3 is a nightmare for anybody who is maintaining a legacy codebase. What if you cannot afford the risk of rewriting your codebase to Scala 3? I’m sorry, but you will have to stay on Scala 2 and pay a premium to maintain the compiler and ecosystem yourself, and your access to the recruitment pool will be significantly reduced, so you’ll need to pay another premium to attract and retain talent.
I know a lot of Scala developers and project managers who are very concerned about where Scala 3 is headed. Unfortunately, the official response has been to double down on moderation: It is against the Code of Conduct to speak negatively about Scala (the Code of Conduct was originally introduced to suppress the sexist community tone but has since been used as a political weapon). So any concern about Scala 3 is met with fierce objection and you will be under threat of a ban for raising your concerns in public.
The most passionate proponents within the Scala community have attempted to shame and dismiss anybody who voices a concern about Scala 3 as “toxic trolls” or “FUD.” But there is genuine concern, and it should not be ignored. Debates should not be silenced and individuals should not be dismissed or banned for questioning the estimates.
What can be done? I believe the Scala community needs to speak up, loudly, and tell the academics and custodians of Scala to change the name of Scala 3. Launch it as “Dotty,” a new language with its own improvement process. Hand over control of the Scala Improvement Process (SIP) to the industry and let’s do things by consensus.
Top ecosystem contributors, who originally supported Scala 3, are already pleading for the scope to be reduced because they fear a repeat of Python 3 or Perl 6. Follow their lead.
“I am seeing aggressive, massively breaking changes being proposed and pushed in against negative consensus.” — Author of fastparse and ammonite