Open in app

Sign in

Write

Sign in

Permutive Community Engineering, February 2020

Travis Brown
Permutive
Published in
8 min readMar 4, 2020

My two primary goals for the Community Engineering team in February were to prepare for a new Cats 2.2.0 release and to get Circe’s new Dotty-powered generic derivation to a state of feature parity with circe-generic. While we didn’t publish the first Cats 2.2.0 milestone, we did merge several significant improvements that will go into it, in addition to publishing a new Circe release that introduces Dotty artifacts, kicking off a new statically-sized collections library, and helping push forward the community effort to upgrade to the new Scala.js 1.0.0 release.

Cats 2.2.0

One issue that we’ve run into repeatedly at Permutive is inconsistency in the APIs of the collection types that Cats provides in cats.data. The Chain type for example is presented as a replacement for List or Vector that’s likely to perform better in cases where you’re doing a lot of concatenation, but it was missing many essential collection operations like zipWithIndex or sortBy. In other cases we’ve started replacing NonEmptyList with NonEmptyVector and immediately run into the fact that NonEmptyVector was missing groupBy and several other methods that were available on NonEmptyList.

One of the things I worked on last month was an overhaul for these types and their tests that eliminates most of this inconsistency (although not all of it, unfortunately, because of the constraints of our binary compatibility guarantees). These changes (and some follow-up clean-up) have been merged and will be available in 2.2.0.

Another issue in Cats that I’d run into a few times was related to a change in the way operations on BigDecimal work between Scala 2.12 and 2.13. This change caused Cats’s CommutativeMonoid instance for BigDecimal to be neither commutative nor associative in some rare cases on Scala 2.13, which was causing our laws-checking tests to fail occasionally. I proposed a solution that went through several rounds of review and is now merged, and opened a follow-up issue to track a related problem that’s still unaddressed.

I also made some significant changes to the way type class instances and other definitions are brought into scope in the Cats tests, with the goal of making the tests reflect standard usage more closely. During this process I ran into some inconsistencies in Cats’s syntax packages, which will be fixed in 2.2.0.

I made a few other changes in Cats in preparation for 2.2 and Dotty:

I also spent a substantial amount of time in February on code review for Cats, and we ended up averaging over one pull request merged per day.

Circe 0.13.0

The Circe 0.13.0 release for Scala 2 is relatively uneventful, with most of the modules (including circe-core and circe-generic) being fully binary-compatible with their 0.12.x versions. I fixed one minor bug in the JsonCodec annotation, and we introduced one new method, but the focus of this release series (at least on Scala 2) is on updating to the new Jawn 1.0.0 release and supporting Scala.js 1.0.0.

The most exciting thing about this release is that it introduces Dotty artifacts, which include generic derivation built on Dotty’s new derives mechanism. Circe’s generic derivation on Scala 2 is currently a minefield of decisions and trade-offs. Do you use automatic (arguably convenient but slow and confusing) or semi-automatic derivation? Do you choose circe-generic (principled and clean but takes forever to compile) or circe-derivation? Do you write your own custom instances or use the configurable derivation in circe-generic-extras? Dotty simplifies all of this, allowing us to provide fast and robust derivation in circe-core via the new derives Decoder syntax.

In my initial experiments derives Codec compiles about twice as fast as circe-generic’s semi-automatic derivation for some common use cases (e.g. a couple dozen case classes with different numbers of members), and only slightly slower than circe-derivation. The Shapeless-powered derivation in circe-generic also has some runtime overhead—about 10% less throughput than manually-written instances in our current benchmarks for decoding, and a little more than that for encoding. The new derivation closes this gap, thanks to Dotty’s simplified generic representation, and in our benchmarks the derived instances actually outperform Codec.forProductN.

Statically-sized collections for Scala 2.13 and Dotty

At Permutive we use Shapeless’s Sized to provide extra type safety in a few places where we have collections with statically-known sizes (e.g. labels associated with a metric). Using statically-sized collections has many advantages, and turns a number of common bugs (out-of-bounds indexing, providing too many or too few values) into compile-time errors. It’s a little like working with tuples, except all of the elements are required to have the same type, and you can abstract over size in useful ways (something Scala 2 has never really supported for tuples).

Shapeless’s Sized works well in many cases, but it’s starting to show its age. It uses Shapeless’s own Nat representation of type-level positive numbers, for example, instead of Scala’s native integer singleton types, which are now supported by literal types in Scala 2.13 (which we use for most of our projects at Permutive). Literal singleton types have many advantages over the Nat representation: they’re cleaner to read and write, they won’t cause stack overflows in the compiler for large values (where “large” is a few hundred), and in Dotty they’re supported by new type-level operations.

In addition to literal singleton types, Scala 2.13 also introduced a new collections API, which will be shared by Scala 3.0. Combining these two new features to create a successor to Sized seems like an obvious win, but to my knowledge it hadn’t been done, so I put together a proof-of-concept implementation that supports Scala 2.13 and Dotty (and published it to Maven Central for both).

This implementation highlights some differences in the facilities that Scala 2 and Dotty provide for encoding and tracking information like collection sizes in types. On both Scala 2 and Dotty, for example, we want our types to track the fact that if we drop three values from a collection that’s known to have five elements, we’ll end up with a collection with two elements. On Scala 2 I’ve done this with a Pair type class that carries around some information about the relationship between two numbers, and that requires a macro to generate its instances. Our size-tracking drop method then looks like this:

def sizedDrop[X <: Int](implicit
  X: ValueOf[X],
  ev: Pair.LtEq[X, N]
): C[A, ev.Diff]

ValueOf here is a type class introduced in the Scala 2.13 standard library that makes the value of a singleton type available at the value level. Pair.LtEq is our own custom type class that witnesses that X is less than or equal to N (so that we don’t drop more elements than we have), and also computes their difference (at the type level).

On Dotty the method signature is completely different:

inline def sizedDrop[X <: Int]: C[A, N - X]

Most notably there are no type-class constraints here, and we no longer have a path-dependent type (ev.Diff) in the result type.

The implementation is also very different. While Scala 2.13’s sizedDrop implementation was a one-liner that was actually much shorter than the method signature, Dotty’s is more interesting:

inline def sizedDrop[X <: Int]: C[A, N - X] =
  inline constValue[X] match {
    case x if x < 0 =>
      error("Can't drop a negative number of elements")
    case x => inline constValue[N] match {
     case n if x <= n => unsafeWrap(unsizedDrop(coll, x))
     case _ =>
       error("Can't drop more elements than the collection size")
  }
}

Instead of the ValueOf type class, we now use Dotty’s new constValue to go from the type level to the value level. While on Scala 2.13 we had to do our own type-level comparisons and arithmetic (in our materialization macro for Pair) and represent the difference with a path-dependent type, Dotty’s new metaprogramming features let us write our comparisons in an ordinary pattern matching guard (if x < 0 => and if x <= n =>), and we can do our type-level arithmetic directly on the types (N — X).

I’m excited about what this comparison suggests about the future of metaprogramming in Scala 3. While Scala 2’s macro system gave a lot of power to library developers, this power came with heavy costs, including user-hostile compiler error messages, incomprehensible and fragile code, and long compile times. The new Dotty features I’ve used in this project—inline methods, type-level singleton operations, etc.—are strictly less powerful than Scala 2’s macros, but in my view they’re a much more promising foundation for sensible library development.

If you’re interested in reading more about this project and my impressions of metaprogramming in Dotty, I’ve written a more detailed answer to a related question on Reddit, and would be happy to follow up on any other questions there.

Scala.js 1.0.0

The core Scala.js artifacts were released in early February, although the official announcement didn’t happen until last week. In between the release and the announcement, Scala library maintainers were encouraged to publish their libraries for 1.0.

While we don’t use Scala.js extensively at Permutive, we know that Scala.js support is important to many of the users of libraries we help to maintain, and we wanted to help make the transition from 0.6 to 1.0 as easy as possible for the community, so we updated and published a number of projects, including Simulacrum, discipline-scalatest, and Cats. My initial attempt to back-publish Cats 2.1.0 for Scala.js 1.0 failed, so I took some extra time to prepare and test a 2.1.1 release, which we published last Tuesday. I also back-published most of the modules of Circe 0.13.0 for Scala.js 1.0, and am planning to update and publish the remaining modules as our last few dependencies get updated.

Other open source tasks and projects

March preview

Scala
Open Source
Dotty

--

--

Travis Brown
Permutive

Written by Travis Brown

38 Followers
Writer for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams