Permutive Community Engineering, March 2020

The highlight of March for the Community Engineering team was the release of the first Cats 2.2.0 milestone, which introduces a major change that we’ve been working towards since last September. We also published new releases of some other Typelevel and Circe libraries, moved a couple of the libraries we use a little closer to Dotty cross-building, and began work on a new project that we’re planning to launch this week.

Cats 2.2.0-M1

The Cats library has always provided type class instances for Scala standard library types as “orphan” instances that require imports in order to be used. Scalaz did the same before it (at least back to 2011, when this change was introduced, for reasons nobody seems to remember or understand exactly). This approach works against the grain of the language, which allows authors of type classes to put instances for types from their dependencies (including the standard library) into “implicit scope”, which doesn’t require imports. The new Cats 2.2.0-M1 release changes the library to take advantage of this language feature (without breaking binary compatibility with previous releases).

Putting type class instances into implicit scope has a number of advantages. It simplifies the experience of using the library, since you never have to remember to add imports to bring type class instances into scope. It can also significantly improve compile times for users of the library, because it avoids cluttering up lexical scope. In previous releases of Cats, idiomatic usage involved putting over 650 implicit definitions into lexical scope, each of which then had to be considered for every implicit search. In Cats 2.2.0 this number is under 200 (for extension methods), with the rest of the implicit definitions living in implicit scope, where the compiler can resolve instances more efficiently. Please see this blog post about my initial experiments in this direction for some concrete examples of compile time improvements.

In addition to the implicit scope change, the 2.2.0-M1 release introduces many new convenience methods, optimizations, and improvements, with 85 pull requests being merged since 2.1.0 in December. These changes represent a lot of work by the Permutive Community engineering team over the past couple of months, together with the 28 other human contributors (including 14 first-time contributors), and one robot:

     77 Travis Brown
     19 Scala Steward
      9 Daniel Spiewak
      8 Frank S. Thomas
      3 Lars Hupel
      3 Gagandeep kalra
      2 Vitaly Lavrov
      2 mooi
      2 Mateusz Wójcik
      2 Kai(luo) Wang
      2 Jakub Kozłowski
      2 Diego E. Alonso-Blas
      1 Yannick Heiber
      1 Vasilis Nicolaou
      1 Shan Sikdar
      1 Sanjiv Sahayam
      1 Ross A. Baker
      1 P. Oscar Boykin
      1 Peter Perhac
      1 MaT1g3R
      1 Markus Appel
      1 Luka Jacobowitz
      1 Luis Miguel Mejía Suárez
      1 Luís Campos
      1 Leandro
      1 kenji yoshida
      1 Filippo Mariotti
      1 enzief
      1 dantb
      1 Brian Wignall

Thanks to everyone who contributed to this release, and to everyone who’s helping to test it out in preparation for the upcoming 2.2.0 final release.

Dotty cross-building

Most of my work on Dotty compatibility in March was focused on http4s, a framework for building HTTP services that we use extensively at Permutive. The biggest challenge here was that http4s depends on a fork of parboiled2, a parser combinator library that relies heavily on Scala 2’s macro system. I’d never worked with Dotty’s new macro system before, but this seemed like a good excuse, and I ported enough of parboiled2 to make it possible to identify the other changes that were needed to get http4s building on Dotty, most of which are now merged. It’s still not clear what the future of parboiled2 will look like on Dotty, but it’s likely we’ll wrap up the port in April, if only for use in http4s’s fork.

Other releases

In February I nominated Simulacrum Scalafix for Typelevel membership, and in early March this proposal was approved, so I made a couple of fixes and improvements that had been suggested and published the new first release in the Typelevel organization.

I also worked with a couple of external contributors to add some new features to circe-derivation, including support for transforming constructor names in derived encoder and decoder instances, which brings circe-derivation essentially to feature parity with circe-generic.

I also published two new circe-golden releases, which fix some issues that we’d run into when using the library in tests for internal services.

Dhall for Java

The final project that I worked on in March wasn’t a Scala library, for a change. Like many companies, Permutive uses Kubernetes for container management, and we have many thousands of lines of YAML configuration that make this possible. YAML’s problems as a configuration language are well-known: it’s complicated, unintuitive, insecure, and error-prone. There are many alternatives to YAML, but one that we’re particularly interested in is Dhall, which is a minimal language with a clear specification, an active community, and a programming model that fits in well at Permutive, where we use functional languages like Haskell, Elm, and Scala extensively.

One of the disadvantages of Dhall for us is that while there are implementations in Haskell, Ruby, Rust, Go, etc., the integration with the JVM ecosystem is less clear. There’s a Clojure implementation, but it’s slightly out-of-date (version 5.0.0 vs. the current 15.0.0) and missing some pieces. There’s also a Java wrapper for the Haskell implementation via Eta, but it also hasn’t been updated recently (it’s on version 4.0.0), isn’t published to Maven Central, and has a build process that’s somewhat complex.

Early last year I started working on a Java implementation, but didn’t get much beyond parsing. I chose Java instead of Scala because I wanted to support build tooling use cases where a dependency on Scala would not be convenient (and also because I thought it would be a fun challenge to implement a programming language type checker and interpreter in Java).

Last month we decided to reboot that project, and we currently have a zero-dependency, Java 7-compatible, almost complete implementation of parsing, normalization, CBOR encoding, and type checking, all in a 260K jar. My colleague Tim Spence has also built a reference implementation of import resolution in Scala that makes it possible to load dependencies in your Dhall documents—including remote ones. We also have a Scala wrapper library that works a lot like Circe, allowing you to map between Dhall and Scala types, but which additionally has the ability to decode Dhall expressions into Scala functions.

This is still a work in progress, but we’re now able to evaluate non-trivial programs that use functions from the Dhall standard prelude, and at the moment 1,045 of the 1,047 acceptance tests we’re running are passing. We’re planning to finish up the remaining pieces, get this number to 100%, and publish the project in the next few days.

Other open source tasks and projects

• Created and published an sbt plugin for the JavaCC parser generator.
• Proposed a Cats policy change to help with 2.11 backporting.
• Submitted Catbird for Typelevel membership.
• Minimized a bug in mdoc.
• Fixed a minor issue with an unnecessary constraint in http4s.
• Simplified http4s’s use of macros a little.
• Fixed an issue with a leaking test dependency in Circe.
• Fixed (partially) an issue with an unsafe method in Cats.

April preview

• New Cats 2.2.0 milestones and the 2.2.0 release.
• First release of our Dhall implementation for Java and Scala.