Permutive Community Engineering, August 2019

This article summarizes some of the projects that I worked on during my first month on Permutive’s new Community Engineering team. We’ll be publishing a follow-up post that goes into more detail about how we’re imagining my role, and how we’re balancing internal and external priorities; this report focuses on the open source side of the work I’ve been doing.

We’re planning to publish these reports at the beginning of every month, although the format is likely to change as the team grows and as we refine our goals and metrics. If you have questions or feedback, please get in touch in the comments below.

Cats 2.0.0-RC2

Cats is a widely-used open source Scala library that provides fundamental abstractions for functional programming. Its core module is downloaded from Maven Central over half a million times per month, and Maven Central includes over a thousand modules that depend on it. Permutive uses Cats (and libraries that are built on Cats) extensively in almost all of our web services.

Cats 1.0 was released in December 2017, and this summer we’ve finally been preparing for a 2.0 release. When I started as a community engineer at Permutive, one of the efforts that I wanted to prioritize in my first month was finalizing this preparation and publishing a second 2.0.0 release candidate. This ended up taking the form of three projects:

1. Reworking the Parallel type class. Because of a design mistake in Cats 1.x, this component was essentially unusable without third-party workarounds, and was a frequent source of confusion, especially for new users. Along with the other Cats maintainers, I'd thought that we would have to wait for Cats 3.0 to fix the mistake, because of binary compatibility guarantees we'd made for Cats 2.0. A couple of weeks ago I figured out that it could be fixed without breaking these guarantees, and this fix is now available in 2.0.0-RC2.
2. Resolving some issues in the handling of Scala 2.13 deprecations. The latest release of the Scala language (2.13) includes some major changes to the standard collections library, and in the initial Cats 2.0.0 milestone releases we made some decisions about how to address these changes that turned out not to work very well in practice. Over the course of a couple of weeks I outlined these problems, worked with the other maintainers toward consensus on the best way forward, and then worked through over a dozen iterations of an implementation (which is also available in 2.0.0-RC2).
3. Miscellaneous clean-up tasks. Over the last year and a half of development on Cats 1.x, the project had accumulated a large number of minor inconsistencies and mistakes that generally didn’t affect usability, but did make maintaining and understanding the code more difficult. Because these issues will be harder to fix after 2.0.0, this work was happening under a deadline, and I wasn’t able to get consensus on all of the corrections I wanted to make. The most important ones are now in place, though, including a much more limited public API, more consistent naming, and the addition of some missing type class instances.

I ran the Cats 2.0.0-RC2 release last week, and have been working with maintainers of other open source libraries that depend on Cats to update and publish for the new release (including cats-effect, which we also use extensively at Permutive). I’ve also been working with adopters to verify that these changes don’t break their code (or in the few cases where breakage is expected, that the migration path is clear). The reception so far has been positive, and we’re planning to publish the 2.0.0 final release on 9 September (assuming no major issues turn up before then).

Circe

Circe is a library that I maintain that’s also used pretty widely at Permutive, and at many other organizations, including Stripe, Twilio, Spotify, the Guardian, the BBC, etc. It depends on Cats, and I’m coordinating its 0.12 release with Cats 2.0. I’ve published four release candidates over the course of the month, which have included one fairly significant bug fix, a lot of last-minute small improvements (some guided by what I’ve learned about our use cases at Permutive), and some external contributions.

Internally, I replaced a module called circe-generic in one of our core repositories with an alternative that I’d originally created for Stripe (circe-derivation), which cut clean compile times for the project by 25–30%. I’ve also made several changes to circe-derivation to better support our use cases, and in the last of this month’s four circe-derivation releases I’ve incorporated and extended a large external contribution that adds many new features to the library.

I also did some early investigation toward creating a new JSON Schema validation library for Circe, which would allow us to replace some inefficient and inelegant internal code, but haven’t had time to flesh out the idea yet.

Scala 2.13

The latest release of the Scala language includes many improvements that will be beneficial to us, such as a faster, more robust compiler, features like literal types that simplify some of the patterns we use, and a significantly better collections library.

While we’re not likely to be in a position to use 2.13 in production for at least a couple more weeks, we do have 2.13 update branches prepared and tests passing for most of our Scala repositories. One of the things we’re waiting on is the Cats ecosystem: we can’t update to Scala 2.13 until we’re on Cats 2.x, and while I’m personally fairly confident in the 2.0.0-RC2 milestone, the final release is only about a week away, and I think it makes sense to wait.

Another issue involves an open source GraphQL library we use called Sangria. Sadly the owner and primary maintainer of the project passed away in May of this year, and there have been difficulties in working with GitHub and Sonatype to transfer project ownership. I had never used or contributed to Sangria before this month, but I took up some work-in-progress by a couple of the maintainers and was able to prepare 2.13 branches for the core modules (and all the additional pieces we need). I’m currently publishing these changes locally for testing, and we’re hoping to reach a resolution to the ownership issues soon.

In the meantime I’ve made a number of changes in our internal Scala repos that are compatible with 2.12 (our current version for most projects), but that will make the transition to 2.13 easier.

Other tasks and projects

• Alice published the video of my talk from Scala Central last month, and we’ve gotten some good feedback (see the Reddit thread, or Twitter).
• Published a new release of fs2-google-pubsub, one of Permutive’s own open source projects.
• Fixed a stack-safety issue in the 19.8.0 release of Twitter Util.
• Minimized and reported a regression in the Scala 2.13.0 compiler.
• Wrote up a Circe FAQ as a Stack Overflow Q&A.
• Did some initial experiments with running a Scala Steward instance on our internal Scala repositories. Scala Steward is a maintenance bot that can automatically open pull requests updating dependencies for Scala projects, which can save developer time and help us avoid issues with dependency mismatches across our internal projects and libraries.

Other notes

I’m still working out how to track the distribution of my time effectively, but my rough estimate for this month looks like this:

• 30% purely internal work (onboarding, code review, planning, admin, etc.)
• 20% open source work that’s directly motivated by internal needs
• 30% open source work that will have benefits for internal projects
• 20% open source work with little or no internal motivation

I’m hoping over the next couple of months to have a clearer sense of what these categories should look like and how I can track them most accurately. I’m also hoping to develop a more concrete set of metrics to include in these reports.

September preview

• First Circe 1.0.0 milestone
• Scala 2.13 in production at Permutive
• New circe-json-schema library
• Proposal and proof of concept: Cats standard library instances in implicit scope