It takes a village

What companies and open source can learn from each other

We know that “open source” has become a buzzword among software companies, who are:

• Starting their own open source projects
• Using open source to build their own software
• Recruiting talent through open source communities
• Grappling with the legal implications of open source and IP

But it goes beyond that. Open source changed corporate culture too, helping companies embrace transparency and collaboration in their own workflows.

Yet when it comes to management, decision making and sustainability, companies have something to teach open source as well. By understanding what happened to software companies, we can begin to guess what’s ahead for open source.

What did companies learn from open source?

Companies have changed their approach to software development considerably, thanks to the cultural influence of open source.

1970s-1990s: Hierarchy and the Waterfall Method

In the early days of commercial software development, there were no clear theories on how to efficiently build software. Software engineers looked to their analog counterparts — manufacturing and construction — for inspiration.

The 20th century was dominated by a sequential approach to organizing people and processes. Manufacturing used assembly lines to make goods. Companies used hierarchies to manage employees.

It made sense, then, that the dominant software process of this time would be the waterfall method, conceived in 1970 as a way to build and manage large software systems.

First, a product manager writes up requirements. Then, the software is designed and developed. Then it is tested, and finally deployed and maintained.

Design, build, deploy: what worked for civil engineering seemed to work for software engineering. But after several decades in the field, many companies found that the waterfall method was simply unrealistic.

Software changed much more rapidly than physical buildings. Clients would revise their requests upon seeing working versions of software, which led to costly redevelopments. Teams needed to communicate about those changes more frequently.

2000s-today: Collaboration and the Agile Method

Although alternatives to the waterfall approach were discussed for years, nothing really caught on until the early 2000s, following the first dot-com boom. Software as an industry had grown so rapidly during the late 1990s that the slow, hierarchical waterfall approach no longer seemed to fit.

It was also becoming faster and easier for people to communicate with each other, thanks to new technologies like cellular phones and email.

In February 2001, a small group of software developers met in Snowbird, Utah to discuss a new approach. From their meeting emerged the Agile Manifesto, a new approach to software development that prioritized frequent interactions between people rather than siloed processes.

Agile methodologies were quicker, more lightweight and allowed for shorter turnaround times. It was the perfect approach for a collaborative era.

The rise of agile coincided with the rise of startups using open source projects. At first, startups just incorporated those projects into their own software. But like any cultural exchange, open source began to affect how startups built products, too.

Open source projects had to integrate a lot of different changes from a lot of different people, who might come from different countries, companies, and workflows. So a project had processes in place to make sure things wouldn’t break when new code was added. Git, the version control system released in 2005, made those processes even easier.

Over the next 15 years, those collaborative concepts worked their way into companies, including:

• Using tests to ensure that newly written code won’t break the whole system (ex. processes like test-driven development, or TDD)
• Integrating new changes into code more frequently, sometimes several times a day (continuous integration)
• Letting code go live after passing a series of automated tests (continuous deployment)

These concepts allowed companies to work faster, deploy faster, and ultimately become more productive.

Last year, O’Reilly Media went so far as to coin the term InnerSource to characterize this phenomenon:

Inspired by the spread of open source software throughout the areas of operating systems, cloud computing, JavaScript frameworks, and elsewhere, a number of companies are mimicking the practices of the powerful open source movement to create an internal company collaboration under the rubric InnerSource.

When we think of open source’s influence on software, we tend to think about the legal implications. Open source made it acceptable to share, use and modify software freely.

But open source changed how software companies build proprietary products, too. That open source principles are being used to develop closed source software speaks volumes about its cultural legacy.

What can open source learn from companies?

Companies learned how to efficiently write software by observing what worked in open source. What can open source learn from companies?

Let’s go back in time again to look at who built these software companies.

1980s-early 2000s: Engineers build startups

In the early days of commercial software, software companies were heavily weighted towards engineering talent. The ideal founder archetype looked more like Bill Gates than Steve Jobs.

You might bring in an MBA to run the business side of things, but because startups were building all the software themselves, founders had to demonstrate to investors that they had the technical chops to execute.

As the software industry took off, a couple of things started to happen:

• There were more startups, so companies had to compete for their customers’ attention (ex. Apple’s 1997 “Think Different” campaign)
• More startups used open source software, meaning founders needed to build less themselves
• “Software ate the world”, bringing tech to the consumer masses and giving it a friendlier look

Mid 2000s-today: Startups need more than code to succeed

Among the second wave of startups, who dabbled in social, content, and services, founding teams started to look a little different:

• Design became a valued skill, thanks to companies like AirBnB, whose founders were designers rather than engineers
• Marketing became a serious discipline under the guise of “growth hacking”, a term popularized by journalist Aaron Ginn in 2012
• Brand became a differentiator for startups, even among developer tool companies like Heroku or Twilio

In 2012, AKQA’s chief creative officer Rei Inamoto debuted a new wisdom at SXSW: “To run an efficient team, you only need three people: a Hipster, a Hacker, and a Hustler.” The startup trifecta was born.

Today, it is common wisdom that a successful startup requires more than code to survive. Design, marketing, brand, and the ever-elusive “culture” are just as important. Nobody believes that engineers can do it all themselves anymore.

If a crowded market and commoditization are to blame for this shift among startups, open source is arguably going in a similar direction.

Just like startups, there are many more open source projects than ever before. Just like startups, everything now depends on open source. Just like startups, open source is becoming relevant to more than just developers.

A sustainable future for open source

Open source projects still heavily value code over all other skills. This seems obvious at first: without code, there would be no open source project.

But we once thought that about software companies, too.

There is little support for functions that directly help code, like documentation or test automation. There is even less support for functions that indirectly help code, like community or evangelism.

And yet, we know intuitively that these things matter. Linus Torvalds himself, describing Linux’s growing pains in the early 2000s, said:

We had really big fights back in 2002 or so…where I was dropping patches left and right, and things really weren’t working. It was very painful for everybody, and very much for me, too….there was no technical solution. The solution ended up being better tools, and a work flow that allowed much more distributed management.

A developer cannot reasonably handle all aspects of managing a large project: nobody can.

Thanks to GitHub’s easy-to-use interface, less technical skill is now required to contribute to an open source project. Editing a project can be as easy as editing a wiki. In theory, more types of people should be participating in open source. But cultural norms still hold us back.

We would never accept that a software company could be run solely by engineers. Open source projects are no different. What organization of any kind, really, could survive without a multitude of skills?

Today, nearly all of society relies upon open source infrastructure. Not just software companies, but banks, hospitals, aircrafts, government. It is dangerous to pretend that these projects will simply “figure it out” on their own.

Should open source projects look like companies? Of course not. For one thing, a company is a business, and an open source project is a community, with very different goals and needs.

But there are probably more efficient ways to organize open source projects that would ease the burden and headache of maintainers. Yet research is still nascent on this topic.

(One interesting model to study is Node.js, a project that successfully scaled and distributed its workload among many contributors. Mikeal Rogers recently outlined some great theories on what a healthy project should look like, complete with diagrams.)

“Funding infrastructure” means more than funding maintainers. This is partially why I think crowdfunding platforms and donation campaigns, while helpful, can only go so far.

Instead, we need to take a holistic approach to sustainability: fixing systems, not just throwing money at the same old problems and forcing developers to figure it all out themselves.

I’m currently exploring better ways to support open source infrastructure. If you want to stay involved, you can sign up here to get updates when I post something new, or follow me on Twitter.