Permutive Community Engineering, January 2020

My focus in January was on getting our core dependencies (both internal and external) cross-compiling on Dotty, while reporting all of the issues I ran into along the way. We also finally got a couple of important 1.0.0 releases out the door, and did some work on lining up changes we’d like to see in Cats 2.2.0.

Dotty cross-building

At Permutive we’re excited about many of the new features that are coming in Scala 3, and we’re committed to investing in the migration of the open source Scala libraries we use, starting now with Dotty. In December we kicked off this effort with a new Scalafix-based implementation of Simulacrum and a pull request adding kind-projector compatibility to the Dotty compiler (which was merged a couple of weeks ago and is now available in the nightly releases). We continued in January by adding cross-building support to Jawn and Discipline and opening several pull requests for Cats:

• Added return types, parentheses, etc. required by Dotty.
• Changed kind-projector syntax to the subset supported in Dotty.
• Removed syntax methods in implementations in cats-core.

These have all been merged, and I’m now working on a larger work-in-progress PR that passes tests and is almost complete, but is still blocked by several external dependencies.

It’s worth noting that by far the hardest part of adding Dotty support to Cats was the tests—this is where we had to make the most changes, and where we ran into the most compiler bugs. This isn’t surprising, since the tests are meant to reflect usage, and we’re hoping that by making this effort now we’ll be able to smooth this migration for Cats users, both by reporting (and helping to resolve) the bugs that we’ve run into, and by recording and documenting the changes we’ve had to make.

I did a similar exercise for cats-effect, another Typelevel library we use extensively at Permutive, and made a few sets of changes that have already been merged:

• Adjusted kind-projector syntax, added return types, etc. for Dotty.
• Restricted ArrayStack element type to avoid undefined Scala 2 behavior.
• Added some helper methods, etc. for laws checking.

I have an open PR adding Dotty to the cats-effect build, but it’s currently blocked by the same issues as the one for Cats. I also started working on a cross-building branch for fs2 last week, but decided to wait for a reworking of type aliases that was in progress then (and is now merged).

Dotty bug reports

Last month I opened 14 Dotty bug reports for issues that I ran into while setting up cross-building for these Typelevel projects and some of our internal libraries. One of these issues turned out to be a duplicate, and three were closed as won’t-fix, and of the remaining ten, seven have already been fixed:

• Type checker complains about valid Java code.
• Type alias causes problems for implicit search.
• ClassCastException on by-name parameter in enrichment method.
• Implicit conversion isn’t applied for Id type alias.
• VerifyError for generic class with private constructor and self type.
• Type alias results in confusing type mismatch.
• Derivation crashes for case object.

Of the remaining three issues, only one (the one related to implicit resolution) is inconvenient to work around:

• Spurious implicit argument ambiguity.
• Illegal access error when constructor parameter more specific type.
• Serializing val that refers to another val captures enclosing class.

One of our goals for this effort was to evaluate the risk of runtime errors involved in migrating to Dotty. Our speculation was that while there would likely be costs related to language changes and compile-time issues, there would be relatively little chance of runtime failures because of compiler bugs. This hypothesis wasn’t exactly supported by our experiments: half of the issues above only turned up at runtime. It’s still early days, though, with the first Scala 3 release candidate scheduled for the end of this year, and the fact that we’re still running into runtime issues probably shouldn’t be surprising.

Circe and new Dotty features

Our work on most of the projects above was focused on cross-building the same code for Scala 2 and Dotty, and didn’t really explore many of the new features that Dotty offers. The one project where we tried to go beyond cross-building was Circe, where I’ve added Dotty cross-building but have also put together an initial implementation of generic derivation built on Dotty’s new type class derivation mechanism.

This implementation is still lacking some features that are available in circe-generic and circe-derivation, but as an early experiment it’s quite promising—it lets us avoid both the horrible ad-hoc macros of circe-derivation and a lot of the runtime overhead of circe-generic. In February I’ll be working on extending this initial experiment to add three things:

• Support for the same cases as circe-generic (recursive case classes, etc.).
• Source compatibility with circe-generic (both semi-automatic and auto).
• Experimental configurability (snake-case field names, etc.).

I’m also hoping to be in a position to benchmark compilation times, which has historically been a major pain point for circe-generic users.

Jawn 1.0 and Circe 0.13

Jawn is a JSON parsing library for Scala that’s used by Circe and many other Scala libraries. It was created by Erik Osheim in September 2012, and finally, after almost seven and a half years, we published Jawn 1.0.0 last month. This release included one last major API change that I wanted to get in before committing to long-term binary compatibility, a fix for a potential denial-of-service attack vulnerability, a few new convenience methods, and some small API changes related to Dotty compatibility.

I’ve also published a Circe 0.13.0-RC1 release that’s binary-compatible with Circe 0.12.x for all of the modules that don’t depend on Jawn (e.g. circe-core and circe-generic), and that updates all of the modules that do depend on Jawn to 1.0.0. There are also two new http4s 0.21.0 release candidates that depend on Circe 0.13.0-RC1 and Jawn 1.0.0, and I’m planning to publish the Circe 0.13.0 release in the next few days.

Testing

We also published a 1.0.0 release for discipline-scalatest, a library that’s designed to make type-class law checking as convenient as possible for ScalaTest users. I’d personally been avoiding discipline-scalatest (and reimplementing its functionality) because it depended on a ScalaTest module that was only available in pre-releases. This module (which provides integration with ScalaCheck) now has a general-availability release, so I put some time this month into working through a few changes that I’d need to use discipline-scalatest in my own projects (specifically support for ScalaTest’s FlatSpec and FunSpec testing styles).

Over the course of the month I published three 1.0.0 release candidates with incremental versions of these changes, and tried them out in a range of projects, including Circe, Cats, cats-retry, Paiges, Decline, and many others.

Testing turned out to be a central theme of my work in January, and I worked on a number of other testing-related efforts, including improving test consistency for Circe, updating Dotty support for ScalaTest and ScalaTest’s ScalaCheck compatibility module, and publishing custom artifacts of the entire ScalaTest and Typelevel stack for recent versions of Dotty.

Preparing for Scala.js 1.0

Scala.js 1.0.0 is coming soon, and some Cats and Circe users have been asking for support for the 1.0.0 release candidates. Last month I added cross-building support to Cats, discipline-scalatest, and Circe, and published Scala.js 1.0.0-RC2 releases for all of these projects (excluding a few Circe modules that have dependencies that don’t support Scala.js 1.0).

Other open source tasks and projects

• Fixed a Dotty compatibility issue in sbt-doctest.
• Fixed a macro hygiene issue in log4cats.
• Proposed making the Id type alias in Cats covariant.
• Published a new Catbird release.
• Published two new milestone releases of Permutive’s fs2-google-pubsub.
• Introduced Scalafmt in several projects, including Jawn and Claimant.
• Fixed a publishing issue in the Sangria build and released 2.0.0-M2.
• Cleaned up some minor issues in the Cats build, tests, and licensing.

February preview

• More Dotty cross-building (in particular fs2 and http4s).
• More experiments with generic derivation for Cats and Circe on Dotty.
• Cats 2.2.0 milestone with standard library instances in implicit scope.