Measuring Success in Open Source Projects

Everyone in open source gets asked this question. “How do you measure success of a project?”. And by corollary, “how do you make a project successful?”.

Neither of these questions has an easy answer. And obviously the answer to the first question has major implications for the second. The easy thing to reach for are simple statistics. Number of lines of code, growth of (or maybe change of?) number of lines of code. Number of stars on GitHub. Number of hits to your website.

I hope no one is seriously considering these. They are all trivially manipulated and really don’t measure anything that actually reflects the value and success of a project.

How about number of contributors? Number of reviewers? Number of people active in the community? Now we are getting into measures that are harder to game, more likely to reflect what is really going on. This still doesn’t create an easy path to a universal “grade” for an open source project, though. The area in which a project is active can have a huge impact on the size of the community — and projects with relatively small communities can still be very successful. Not every project can be a Linux with a release every eight to nine weeks, with thousands of active contributors, billions of users.

Very few projects reach a thousand contributors. Even a hundred active contributors is very very rare. But a few dozen, that seems a reasonable goal. And a diverse group of contributors. In every meaning of the word. From different companies, from different geographies, different cultural background. Look for diversity of gender, diversity of perspective. The more homogenous the team looks, the more worried you should be.

A regular release schedule is impressive, the cadence of Linux over the past decade is just simply stunning. That’s a high bar to strive for, especially in projects where “running that project” isn’t the day job of most of the contributors. But a decent level of consistent activity, a steady flow of commits show that a project is alive and well. Signs of interaction (like bug reports and pull requests) are important. And reasonable responsiveness to such interactions. What’s reasonable? Again, it depends. If there are community interactions that sit for weeks without a response from the maintainer, that’s definitely something to look out for.

Also look for a steady stream of conversation. That can take many forms. It’s the outbound communication of the project through announcements, a Twitter feed, maybe a blog or some other way in which the maintainers keep the community informed. Even more important when looking at projects and trying to gauge their likelihood of long term success, are the conversations that are happening inside the project. What is the tone on the mailing list or IRC channel (or yes, for the Millennials, the Slack channel or whatever the latest hip tool you use might be). Is it inclusive, is it open to newcomers, does the project make it easy to get started, easy to join the community? Are people respectful of each other?

Note, however, that this can take different specific forms. It doesn’t have to look like a love fest. In fact, many extremely successful projects have key developers that respect each other, but don’t necessarily like each other. And while I personally dislike the sometimes excessively harsh tone in some communities, that doesn’t always mean that something is broken. Don’t just read and judge, ask the people involved how they perceive the tone of their community; this isn’t always black or white. Some communities believe in very clear communication to avoid misunderstandings. Again, the Linux kernel is an example. Many people disagree with this approach, many women whom I highly respect as developers, point to this as a great warning sign. Admittedly it doesn’t appear that this has hurt the success of the Linux kernel development, much, but who knows, maybe it would be even more amazing if the community were able to attrackt and engage more women. I’d add it to the warning signs. Maybe the project can be successful despite a negative tone of internal conversation, but the likelihood of success seems much higher in projects that enforce civility. A code of conduct (and a community that takes it seriously and enforces it if necessary) is usually a good sign.

The other aspect that I’m always curious about is the integration with other projects — both up and down the dependency chain. How well do the developers interact with others. Do they listen to projects that depend on them? Do they communicate effectively with the projects that they depend on in return? Is there a healthy amount of cross-polination? Members of this project also contributing to adjacent projects (and vice versa)?

If you read this far through my post (wow — your attention span is way above average) you may ask yourself “that’s great, but how can I quantify all this?”. And sadly, I have to admit that after all these years I still haven’t come up with a good way to turn this into something that fits on a management dashboard or can be condensed to a single number, a single grade. When I get asked to evaluate an open source project, I look at their repository, I check the website, I look for mailing list archives or chat archives. And in the end it often ends up being what Judge Stewart described as “I know it when I see it” in Jacobellis vs. Ohio (in a very, VERY different context). It’s really hard to create crisp, infallible criteria that measure a successful (or even a promising) open source project. But after a while you can take a look, spend a few minutes, and you usually can pick out potential problems quite easily.

So in the end this is less of a predictor of success and more of a warning sign of failure. And the absence of one is not necessarily a sign of the other. But to me this is better than nothing. And certainly better than looking at misleading numbers and statistics.

What criteria do you look at when evaluating a project that you are unfamiliar with?