Permutive Community Engineering, November 2019

We’d planned a few big releases for November (Cats 2.1.0, Jawn 1.0.0, and Sangria 2.0.0), and while none of these actually happened, Cats 2.1.0 in particular is shaping up to be a more significant release than we had originally planned, and we’ve made a lot of progress on an experiment that we’re hoping will allow us to cross-build Cats and some related libraries on Dotty in the near future (see the third section below for details). We’re also now running most of the internal services that support Permutive’s API on Scala 2.13, which is something that we’d been working toward for the past couple of months.

Cats 2.1.0-RC1

We dropped support for Scala 2.11 on the master branch of Cats after the 2.0.0 release, unblocking a large number of issues and pull requests that we had put on hold because they would break binary compatibility on 2.11. I spent much of the first couple of weeks of November helping to get these changes merged, and our 2.1.0-RC1 release candidate was the biggest release since 1.0, with 78 pull requests by 35 developers, almost half of whom were first-time contributors to the project.

I also worked on several post-2.11 clean-up projects of my own in 2.1.0-RC1, including the rearrangement of type class operations that could previously only be added as syntax methods, the removal of some unnecessary Either right-biasing syntax, and improved coverage for binary-compatibility checking.

Although we had dropped Scala 2.11 on the master branch in Cats, we had planned to continue publishing releases for 2.11 from a separate branch where contributors could backport new features. We’re still interested in working with community members who want to help with this effort, but right now it seems unlikely that we’ll be able to publish Cats 2.1.0 for Scala 2.11.

Cats 2.1.0-RC2

We began updating some of our services at Permutive to Cats 2.1.0-RC1 a few days after the release, and immediately ran into an instance of broken binary compatibility that hadn’t been caught by any of the tests or API checking in Cats itself. I tracked this breakage down to a bug in Lightbend’s MiMa tool, and put together a fix that we were able to use to find a few other instances of broken compatibility that we had accidentally introduced in RC1.

We released a second release candidate with these fixes (and a few other minor changes) a couple of weeks ago, pushing back the 2.1.0 release, which we’d initially planned for late November. We’ve been using 2.1.0-RC2 in production at Permutive since last week with no issues, and I’m hoping that we can publish 2.1.0 by mid-December.

Cats and Dotty

The other major project I worked on in November was an experiment focused on getting Cats building on Dotty, a new research language and compiler that’s intended to become Scala 3. I’d spent some time this summer experimenting with a Dotty port of Cats and Circe, primarily as a way to explore what some of the new language features will mean for projects like these. Now Dotty is approaching a feature freeze, and I wanted to have a better sense of what work needs to be done in order to get these libraries cross-building on Dotty — not in some hypothetical future version, but in the next few months or even weeks.

One major blocker is Cats’s use of Simulacrum, a tool for generating boilerplate related to the encoding of type classes in Scala. Simulacrum 1 is built on Scala 2’s macro annotations, which up until Scala 2.13 were only supported via a compiler plugin, and which are not supported at all in Dotty. We’ve had this issue on our radar for over a year, and had discussed a couple of possible approaches for addressing it. There seemed to be agreement that the most promising approach would be to rewrite Simulacrum as a set of Scalafix rules, and I began working on a proof-of-concept implementation in mid-November.

This implementation is now available, together with some related tools that make it possible to cross-build cats-core and several other Cats modules on Dotty. As part of this effort I’ve also opened ten pull requests that are designed to make the current Cats codebase more Dotty-friendly in cases where this can be done without also making it less idiomatic or readable. Altogether it’s possible to cross-build most of Cats on Dotty with only a few dozen lines of manual changes—the rest can be done by our Scalafix rules.

I’m currently working on a few follow-up threads: getting the remaining Cats modules building on Dotty, gathering community feedback, and implementing Dotty’s support for kind-projector-compatible syntax for type lambdas.

At Permutive we’re excited about Dotty and Scala 3, and we’re hoping that investing in experiments and efforts like these will help to make the idea of migrating from Scala 2 a little less scary.

Other tasks and projects

• Published circe-yaml 0.12.0 with several number-related improvements.
• Published the first circe-droste release following Droste 0.8.0.
• Published a new Catbird release.
• Continued helping to coordinate work toward Sangria 2.0.0.
• Reported and fixed a Shapeless bug affecting Circe’s generic derivation.
• Reported and fixed a minor issue with imports in Scalafix.
• Reported and fixed a bug in the Tokens API in Scalameta.

December preview

• Scalafix-based code generation for Circe codecs.
• An implementation for -Ykind-projector in Dotty.
• Cats 2.1.0, probably Jawn 1.0.0, maybe Sangria 2.0.0.