Scala Almost Succeeded

fommil
fommil
Aug 25 · 8 min read

And it Still Could.

Scala is an academic language. It attracted attention from industry because it promised better scalability for multi-core computers at a time when Java was stalling.

Then, out of nowhere around 2015, Scala experienced hyper-growth because of Spark, the general purpose cluster computing framework.

By “success” we mean number of developers using Scala in industry. If the Redmonk ratings, and frequency of job adverts, are anything to go by, Scala has since plateaued. That said, Scala has managed to find its way into almost all JVM systems even if only a part of that system is a library or framework written in Scala.

This post is an opinion on why Scala’s early success may be coming to an end and what I believe the Scala community needs to do to get back into a phase of growth.

This is not an attack on any individual. If you feel that I have spoken negatively about you as a person please contact me and I shall endeavour to put your mind at ease.

Pain Points

Scala developers consistently complain about compile times, quality of tooling, and which language features are too complex to be included in their codebase.

To be precise, developers complain about slow build times, which is the time it takes to build a project from cold. This also includes startup time and resolving / downloading dependencies.

Complaints about tooling can be anything ranging from the complexity of the build tool, to the lack of support for advanced language features in the IDE, to the difficulty to publish a project.

The thing that terrifies most Scala developers, and causes the fiercest debate, is use of the Shapeless generic programming library. I am partly responsible for this, because I once gave a Scala eXchange workshop evangelising its use. I since regret this, please forgive me!

Compile Times

In 2019 we have seen good progress from Lightbend on compilation performance: the Scala 2.13 compiler is the fastest Scala compiler yet and the proprietary GraalVM technology can make it even faster.

Lightbend have, to their credit, doubled down on micro optimisations with the assistance of their commercial customers. That’s a laudable achievement because Lightbend is primarily concerned with their own products and the Scala Team is very small (only four full time developers) and they are also responsible for the maintenance of sbt.

For anybody needing even faster compiles, https://www.triplequote.com/hydra/ has got you covered.

Twitter are rewriting the compiler with RSC (the Reasonable Scala Compiler), and Kentucky Mule has proposed by a former Scala compiler team member, so there are still further options that could be taken.

Tooling

It took a long time for the academics at EPFL to accept that compile times and tooling were a problem. The Scala Center was set up in 2016 to “coordinate and develop open source libraries and tools for the benefit of all Scala users”.

I was an early supporter but I lost faith when they spent their first 3 years working on pet projects and vapourware such as Scaladex, Spores, Scastie, Scala Native, and Dotty, instead of addressing imminent problems.

In 2019 we see more projects being delivered by Scala Center that are relevant: they are writing faster build tools, refactoring tools, and alternatives to IntelliJ.

Unfortunately their lack of industrial engineering experience shows in their decision to invent new solutions rather than to contribute to incumbent tooling that is in use in production. For example, we can criticise bloop by noting that unless the benefits are seen in sbt then nobody will benefit, and we can criticise the creation of Metals by noting that 10% of Scala users (circa 2018) are using ENSIME and that they have dismissed the contributions of literally hundreds of volunteers.

This follows a tradition of reinvention that goes back to Scastie (reinventing Scalafiddle) and Scala Native (competing with Graal). If the Scala community wants better tooling then they must hold the Scala Center to account: they were set up to “coordinate and develop” not to “ignore what exists and develop new things”.

I could also say that if the community wants better tooling then they have to build it, but I don’t think this is fair on the volunteers who dedicate their time to it. It is a fulltime job and nobody should have to sacrifice their family time for that.

Complexity

In 2018, Jon Pretty released the Magnolia macro library. Teams are replacing their most complicated Shapeless “typeclass derivation” code with simple Magnolia alternatives, which can be 100x faster to compile.

Looking Good?

Looking at how everybody is responding to the main problems, Scala is in good shape!

Scala Center and Lightbend are delivering value, and the community is stepping up to fix gaps in the ecosystem. So why is Scala hitting a plateau? Is it all too little too late?

I believe that the problem is multi-faceted:

  1. it is really hard to hire experienced Scala developers.
  2. while Scala has been burning time, Java has caught up.
  3. Scala 3 and the Osborne Effect.

Hard to Hire Scala Developers

A consequence of a hyper-growth phase is that the vast majority of Scala developers are relative beginners. This, on its own, can be fixed with access to good training. But Scala’s inherent complexity means that a training course is often not sufficient for a developer to be able to navigate all the language features, runtime environment, and the fragmented, and often tribal, ecosystem.

One of Scala’s selling points was that it is compatible with Java libraries. This is a double-edged sword because although it helps transition Java developers to Scala, it makes it harder for people who don’t know Java to learn Scala. One must learn about the JVM to use much of the runtime tooling (profilers, debuggers, optimisation, etc).

The many different styles of Scala mean that it is not enough to hire or train somebody in “Scala”. Play, Typelevel, Scalaz are all very different technology stacks creating niches within a niche. Entrenched politics have caused these stacks to drift further apart. They must come together rather than fight. Unfortunately in 2019 we’ve seen the opposite: Lightbend and Scala Center have actively excluded projects and individuals from the community because they contribute to the “wrong” stack.

The politics are so dirty that several members of the Scala compiler team have recently engaged in “no platforming” of a prominent contributor having keynotes removed from Scala Days (UPDATE: and Scala eXchange), with pressure applied to all conference organisers.

Then there is Typelevel. In my opinion they are the worst thing to happen to the Scala Community. They are a small but very vocal minority who have decided that they are morally superior and represent the functional programming community. But nobody elected them, and a lot of people are unhappy about that.

The drama in itself pushes people away from Scala. It’s also a huge disincentive for contributors, lest they be seen to be picking a side, or worse, find themselves unemployable.

Java has Caught Up

Java now has much better support for concurrency and functional programming. There are proposals to add pattern matching to the language.

Android runs a stripped down version of Java, which is 80% of all smartphones on the planet. Scala intentionally dropped support for Android meaning that any developers targetting the largest App Store in the world have had to find alternative languages. Many have returned to Java, others have moved to Kotlin as a “better Java”.

I am not aware of any plans for Scala to support Android again but it seems like a mistake to exclude the largest market in the world.

The Osborne Effect

The Osborne Effect is the myth that the Osborne Computer Corporation went out of business in the 1980s because they announced Version 3 of their personal computer while Version 2 was still in the shops. Although a myth, the effect is very real. Scala 3 has been an academic research project for nearly 5 years under the codename “Dotty” and is supposed to arrive within the next few years. At that point, Scala as you know it will be decomissioned. i.e. why bother investing in Scala 2 when you can wait for Scala 3 and make do with Python until then.

The changes in Scala 3 are so dramatic that all Scala developers will need to be re-trained and all codebases will need to undergo a migration. Personal contacts tell me that their migration estimates range anywhere from “a weekend” on small projects, to 1 month on medium sized projects (100k lines of code) to years of effort for larger project (millions of lines of code). And that assumes that the free software ecosystem has already been migrated, which is relying on a lot of volunteers mobilising for the common cause.

All the compiler and tooling improvements that have been made in the last few years will be thrown away, they are targeting the Scala 2 compiler. It’s not even clear if the IDEs and text editors will be ready (the effort to support significant whitespace and new grammatical constructs cannot be overstated and is too low level to be solved by LSP).

The Scala 3 compiler is slower than the Scala 2 compiler, and is not capable of building core ecosystem libraries. As a community volunteer contributor, you are expected to rewrite all of your projects and it might not be possible to rewrite some. Lack of free time will leave gaps in the ecosystem.

Scala 3 is a nightmare for anybody who is maintaining a legacy codebase. What if you cannot afford the risk of rewriting your codebase to Scala 3? I’m sorry but you will have to stay on Scala 2 and pay a premium to maintain the compiler and ecosystem yourself, and your access to the recruitment pool will be significantly reduced so you’ll need to pay another premium to attract and retain talent.

I know a lot of Scala developers and project managers who are very concerned about where Scala 3 is headed. Unfortunately the official response has been to double down on moderation: it is against the Code of Conduct to speak negatively about Scala (the Code of Conduct was originally introduced to suppress the sexist community tone but has since been used as a political weapon). So any concern about Scala 3 is met with fierce objection and you will be under threat of a ban for raising your concerns in public.

The most passionate proponents within the Scala community have attempted to shame and dismiss anybody who voices a concern about Scala 3 as “toxic trolls” or “FUD”. But there is genuine concern and it should not be ignored. Debates should not be silenced and individuals should not be dismissed or banned for questioning the estimates.

What can be done? I believe the Scala community needs to speak up, loudly, and tell the academics and custodians of Scala to change the name of Scala 3. Launch it as “Dotty”, a new language, with its own improvement process. Hand over control of the Scala Improvement Process (SIP) to industry and let’s do things by consensus.

Top ecosystem contributors, who originally supported Scala 3, are already pleading for the scope to be reduced because they fear a repeat of Python 3 or Perl 6, follow their lead:

“I am seeing aggressive, massively-breaking changes being proposed and pushed in against negative consensus” — Author of fastparse and ammonite

If you would like to discuss this opinion piece, please use https://www.reddit.com/r/hascalator/ noting that the “official” Scala channel on reddit is compromised by authoritarian moderation: any complaints about Scala or its future will result in you being permanently banned.

Chartered Mathematician, Haskell, Free Software. Author of https://leanpub.com/fpmortals

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade