Things Fall Apart

Let’s work together to reduce the impact of JavaScript fragmentation

Yesterday, after the announcement of Yarn, some in the JavaScript community, including myself, criticised Facebook and the project. The comments I made (regarding Ava and pnpm) were reactionary and written in haste; ultimately I was just plain wrong.

Normally, I like to think before speaking. When I don’t, it’s typically a case of my buttons getting struck or nerves being pushed. Now that I’ve counted to ten (10), I’d like to share my thoughts.

There’s a tendency for companies and individuals to fork or just re-implement OSS projects instead of working with that project’s community. It happens for a lot of reasons, but it also happens more than it should. It’s understandable to question why Yarn had to happen.

Punk’s Not Dead

There’s nothing inherently bad about DIY in OSS. For developers, it can be a great way to learn or show what you can do. Unless you’ve got a fan club, it’s a drop in the pool.

But a business that can dedicate resources and talent to a project — with all the requisite networking and marketing — is akin to a cannonball. Acting purely in self-interest can and does cause harm to OSS communities.

I want to explain some ways in which a company can arrive at this decision (see also: Not Invented Here), and the consequences.

Paved With Good Intentions

If it appears OSS project X can work for a company’s use-case, they’ll try it.

But maybe they didn’t research it well enough, or maybe the needs changed, or scope creep, or X’s bugs aren’t getting fixed, or a billion other reasons. Instead of collaborating, the company bails and writes their own version.

A company’s developers make the case that it’s going to be cheaper to DIY. I know this, because I’ve done it too. And it made a hell of a lot of sense.

The business agrees and loves the idea; they can “give back” to the community by open-sourcing a new software project! The developers at the company also love this, because they get to write new code with their own standards and practices. They’re no longer at the whim of somebody else’s priorities and timelines.

A more mindful businesses will make a good-faith effort to contribute back to project X. As long as the code is merged at a reasonable clip, everyone is happy. But if a pull request sits for too long, or the project maintainers disagree about the direction and end up refusing to merge, the collaboration can grind to a halt.

Both of the above, which result in new project Z, can cause fragmentation on several levels:

• Community. The split can cause other users to jump ship for Z. Maybe Z fits their use case better, or they believe Z will move at a quicker pace, or simply because it’s sparkly. Contributions to X may decline.
• Effort. The user(s) previously will now contribute to Z instead of — not in addition to — project X. Keeping feature parity or compatibility will duplicate effort.
• Ecosystem. If X is a widely-used tool, it might have plugins and provide interoperability with other tools. These existing projects in X’s ecosystem may or may not be compatible with Z.

Individuals, businesses and project maintainers could be doing a better job of addressing potential fragmentation. Maintainers become too attached, or don’t often understand what their users really need, or simply have different motiviations. Businesses think they face “unique challenges” more often than they really do.

Of the types above, I’d like to hone in on ecosystem fragmentation, because it’s relevant to the recent announcement of Yarn.

The Same, ‘Cept Different

When I’ve tried to use alternatives to the npm CLI, I’ve found packages break. They break because they expect to be installed via the npm CLI. Not too unreasonable — lifecycle scripts are a thing, and the CLI treats symlinked folders special-like.

I have reason to believe there are many such packages, and that you’re probably using at least one.

Yet, alternatives to the npm CLI shouldn’t be expected to provide 100% compatibility with the existing ecosystem around the npm CLI. If they did everything the npm CLI does except better, everyone would stop using npm CLI, including npm itself.

Even if an alternative CLI addresses concerns about security, determinism, and performance, widespread adoption will cause some level of ecosystem fragmentation.

It Ain’t Easy Being Green

Those new to the Node.js & JS communities will be faced with yet another choice of what tool to use. They will wonder why npm doesn’t work with a library that was written by someone using an alternative CLI. Or, conversely, why the alternative CLI doesn’t work with that library written expecting npm.

Even though great care and effort was put into Yarn specifically, it cannot cover all of the edge cases, nor can anyone expect 100% compatibility. A veteran of the JavaScript ecosystem can find workarounds, but it’s a frustrating experience for noobs.

Multiplying Build Matrices

You’re a project maintainer. One day, you get a bug report:

hey this package doesn’t work with quuxbaz, the new npm client, add support plz

Nevermind that you spent a lot of time last week fixing bugs that broke Windows support. Now you need to worry about users of your project running quuxbaz on Windows.

You can see how easily this problem grows. If you already support three (3) versions of Node.js, Linux, Windows, and macOS (9), your build matrix doubled in size to eighteen (18), assuming quuxbaz works on those original nine configurations anyway. It might not, so nevermind those weird bugs you spend time investigating that are actually problems with Node.js 4.x on macOS using quuxbaz.

I’m not really looking forward to this — I don’t want more overhead, nor do I want to outright refuse to support quuxbaz because of it. And I apologize in advance to everybody if I create this issue in your tracker.

For a more concrete example:

When Browserify happened, many Node.js packages could now magically run in the browser. Others started breaking when bundled. Some packages were able to “fix” this “bug”; others weren’t. Time passes, and Browserify works OK, but now it won’t Webpack (verb) right. And then fails in a headless browser. Or Electron, or a phone, or toaster, etc.

How can we mitigate these problems?

Teamwork!

Regardless of the reasons why we’re in this, we’re all in this together.

For Users

As a consumer of Node.js modules, you can help project maintainers by submitting detailed bug reports, and/or code that fixes them. If you’ve reported a bug, keep ’em coming! If you’ve contributed code, please help maintain it. That last part is probably the most important thing.

Maintainers

As a maintainer, keep yourself open to new ideas and perspectives. If you put something on GitHub, you should expect someone else to use it in a way you didn’t think of. You’d be surprised how many disagreements “put it behind a flag” solves.

Biz

For businesses — and especially R&D teams — if a project has an active community, join it. You might find out another user wants the same missing feature. Maybe you can work on it together, like Team Yarn.

Yarn Team

Team, this is an outstanding effort. There may be some lingering misunderstandings around the project’s existence — not its governance.

Please help make sure the community, and especially those new to it, understand:

• What and who is Yarn for?
• What and who isn’t it for?
• What can I expect it to work with?

If I see new issue “doesn’t work w/ Yarn” tomorrow — though I don’t expect I will — I’d love a wiki or documentation with common problems, solutions, known incompatibilities, which are user error, etc.

And if you’re feeling charitable, plz stick 100% npm compatibility behind a flag, thx.