A Brief Guide to FLOSS
Beginning to FLOSS
In 2015 I began a rather large collaboration with several major American companies to develop a new set of tools and standards for web development. It started with a goal to develop a simple, but smart, web framework that would focus on zero configuration, be modular, and have limitless extensibility.
This article is about how that project unravelled due to its original contributors pulling out, and how to manage that risk in your own projects.
Over the years the project’s goals evolved to add privacy-preservation and decentralisation as core tenets.
The project is not particularly large, but large enough that it required support to develop new approaches to lower-level networking components that could be used by a larger community. These components would need to be supported, and be production-ready. That’s a lot of code to write and maintain, on top of the web framework itself.
It’s common for developers to want to re-write common libraries of code for themselves, rather than re-use what’s already available, but doing so is typically frowned upon. Re-use is an essential precept in software development and is even a fundamental paradigm in object-oriented programming (OOP) and forms several points in Eric Raymond’s 17 Unix rules.
As a result of this, if you’re going to rewrite lower-level components to solve fundamental issues then it’s good to do it with enough industry backing that those components can form part of a greater ecosystem. This creates a natural network of support, momentum, and reduces the surface area of vulnerability as more eyes and more CPU cycles are being put to use implicitly testing the components.
Open sourcing such a component is the best way to improve adoption, and also goes a long way to improving code quality due to the fact there’s a natural open review process going on. That being said, it’s not a perfect process, and code quality follows a similar curve to code complexity: the more developers using the project often correlates to the amount of code being written, which directly impacts the level of complexity unless this is controlled within the change management process.
Defining FLOSS
Open source is a contentious term to use. There’s almost been a monopoly on what can be defined as open source since 1998 when the Open Source Initiative (OSI) was founded after the Netscape source code was released. The founders (including the aforementioned Eric Raymond) saw a future where many projects would be released as open source and wanted to provide a set of rules to denote what qualified as open source, called the Open Source Definition (OSD).
Did Eric Raymond et al invent open source? No. Developers had been releasing their code with their binaries for far longer than the release of Netscape. Despite many of the myths surrounding the origin of open source it can clearly be traced back over decades to a developer ecosystem where freely sharing source code was fairly normal.
We simply have to go back to 1991 when Linux Torvalds, still a student at the University of Helsinki, released the Linux kernel under the GNU GPL license, which was written in 1989 by Richard Stallman.
This is where things get murky as this type of sharing of code preceded the rather hardline stance provided by the OSI and created a Free / Libre / copyleft approach to source code, championed by the Free Software Foundation.
At this point we could simply separate the two philosophies as open source software and free software (or collectively referred to as Free / Libre / Open Source Software: FLOSS), but that would mean we only have three choices when it comes to licensing our projects: freely, under an open source license, or as a proprietary product.
This simply does not work, mostly due to the responsibilities we have as developers, and I’ll explain why later, but first back to the project in question.
Many Styles of FLOSSing
Since 2015 the project had experienced restarts and had been shared in many production projects within startups all the way up to the Silicon Valley behemoths. The supporters of the project included some of the major software companies in the US and so over time the project took on more responsibilities and began to push upstream into core technologies used by these major companies.
This made sense at the time: these fundamental concepts should be used by as many developers as possible and it’s much easier to get that momentum when there are big players backing the ideas and using them in production themselves. This enabled me to focus on the specific problem areas I was originally trying to solve by creating a dependency on a set of projects I contributed to, but was not in control of.
This is a very important realisation to take into account when using a third-party open source project as a component within another project: you’re not in control of that dependency code. Yes, it’s open source, but in reality what does that mean? Primarily it means you can access to the code, but that’s it, and conversely having that access creates several potential problems:
- You have to wait in line for the features and bug fixes you want. You’re not the priority, there’s no support license, and creating pull requests does not guarantee upstream changes will be applied in a timely fashion, or at all;
- If the project is too slow, or abandoned altogether, then you may now need to maintain that code. Exciting right? Actually, this negates one of the main benefits of open source: shared responsibilities; more eyes, more CPU cycles. It also means that time spent on your projects is now at risk.
I’ve hit all of the above issues over the years, so they are very real: even the big projects can fall with the departure of one major backer.
So why do people use FLOSS? From observation, they:
- have a problem to solve, they searched the Web, they found a component, and they added it as a dependency;
- wanted a widely used component that everyone else was using because: a) it’s trendy, or; b) the amount of materials generated by the community makes it easy to implement;
- couldn’t, or didn’t want to, spend money on a proprietary product;
- wanted something that had permeance beyond the whims of another company by having access to the code;
- wanted freedom to modify a component to fit their specific problems;
- wanted to contribute back to the collective community to create a better set of tools.
I ordered that list based on my own strong opinions, through observations. I don’t believe that’s the order that should exist, but open source is not built around the communal good.
This brings me to some of the major issues with 1: the licenses. It is rare for many developers, even in commercial environments, to check the licenses of the code they’re using, and how it applies to their project. Many developers take a snippet of code from the underlying project and use it in their project, or take the entire project and modify a small part, or implement it in a proprietary product.
Taking snippets of code is almost always a violation of an open source license. There are ways to handle this, but I’ve landed in several high-value commercial product-based projects to find the product has been delivered to the market without following specific licensing conditions about distribution.
I’ve also seen many cases of developers taking a large block of code, with the license fragment itself, and pasting it into their project, assuming they could simply do that without consequences.
There’s also an assumption that any project in GitHub, or available on the Web in general, is open source or FLOSS. Actually the opposite is the default: if there’s no license available, then in most cases the code is copyrighted implicitly by the author.
End of FLOSS
Over the past few years my dependency on the core open source components grew as I built product roadmaps on the use of those technologies, and subsequently built companies on those products. I had a good reason for feeling secure as I was a contributor and, at one point, consulted for the big companies on the future roadmap of those components.
In January the biggest contributor to those components pulled out.
This wasn’t entirely a surprise as they had been contributing less and less to the evolution of the projects over time and other alternative projects were gaining more traction. The alternative projects weren’t well suited to the issues at hand however, so I became stuck.
The promise of the open source licensing meant whatever happened to the project, I could continue. In reality I now had a lot of source code that had become suddenly stagnant, with a community that were forking the code or simply jumping ship, and I had very little time to support and maintain the original code.
The company chose to leave the project as it no longer was seen as an essential part of their profit development plans. The developer adoption had not lived up to expectations, and that adoption estimate contributed to projections of paid licensing figures for their major cloud platform. I know this because that made several public statements to this affect at conferences.
They had the financial resources to continue the project, but it wasn’t seen as a profitable expenditure. It’s important to realise this: open source projects are often part of an indirect exercise in profit making, especially for the largest technology companies. If you’re not contributing back to that project in effort, then there must be something else the company is hoping to gain from your use of their product, whether it’s licensing of other products, use of consulting services, subscribing to cloud services, or a general warm and fuzzy feeling about the company to redirect attention from their less beneficial practices.
This lends to the belief by many that you can’t make a profitable business built on open source software: this is simply not the case. Sure there are some horrendous 2-tier business models, or projects that are only useable if you’re paying for a managed service, but there are also coffee shops closing daily: it doesn’t mean you can’t make money selling coffee. In the case of open source, the organisation has to be structured to fit the project’s goals and stability is essential to adoption and growth.
I spent the next 3-months evaluating my options. With major projects in production using these components, and a very ambitious long-term plan for developing the privacy-focused web framework, I decided to rebuild the core components.
The effort is going to be huge, and it will impact every other project I had planned, but it’s essential to getting the projects where they need to be. This doesn’t mean I’m copying anything from the original projects, or forking the codebase, and it doesn’t impact my project in production due to careful abstraction of interfaces, continuous iterations of development branches, and blue / green deployments.
There are a lot of issues with the original approach — it was built to replicate a legacy model in a different language and certain ideas were restrictive and needed effort to work around, but when using an open source library you typically have to take what you get. Now I can start afresh with years of experience working on these projects to create an improved take on the low-level fundamentals. Unfortunately it’s still going to be a lot of lines of code, and not a lot of support going forward.
You may think I could have continued to write everything from scratch in the beginning, but using dependencies temporarily to get to a milestone quicker is not bad practice. It just needs to be done with careful consideration.
Maintaining FLOSS
A lot of people implement FLOSS within a project without evaluating the maintenance required in both their project and the FLOSS project. Any dependency you add to your project brings increased risk and effort. No software is without issues and if you take on a dependency you’re adding their existing issues to your own.
With proprietary licensing you’re typically supported with a service level agreement (SLA) or some other support licensing which means you have other resources at your disposal enabling you to focus on your own project. It’s not perfect, but it’s something.
With FLOSS you may often find the issues you want fixing are going to have to be fixed by you, and those fixes may not be welcome, which puts you in a difficult position: you need those fixes, but they’re just collecting dust in some pull request (PR).
Day-0 delivery of a project is the easiest target for any developer, but a developer’s job is to look beyond day-0, to day-30, day-90, day-365, even day-1095. No code is stagnant: platforms change, dependencies change, the surface-area for risk changes. Continuous integration and continuous delivery (CI/CD) are useful tools, but they need to be backed by testing and need to be taken seriously. Many projects will have failed builds that go unattended, if they have builds at all, and others will lack adequate testing, if any.
To mitigate this risk it’s good practice to find your use case, and write your own tests, submitting them to the upstream FLOSS project, so you can at least be assured that your needs are being accounted for.
It’s also useful to be involved in the conversations, including in issue management. If critical issues are dormant, then feedback. You don’t need to just contribute code to give a project momentum, and most of the contributions I’ve made to FLOSS projects has been in discussions.
No project owner is responsible for satisfying your needs. There’s no obligation to deliver a better product. They could change all of the APIs and that’s their prerogative. Sure, it’s bad practice, but that’s not going to protect your project.
There’s also a common myth that the owner of a project is using it themselves in production. Often that’s either not the case, or the company is using an internal fork that omits some essential parts they don’t wish to share. Feeling a sense of stability and assuredness from a big company backing a project, and even giving talks on how they use it, is a dangerous comfort to take.
Take a step back and take a look at the dependencies you added to your last project and quickly evaluate what would happen if one of them suddenly lost their main contributor. What’s your risk management plan for your project?
Ethos of FLOSS
I briefly touched on the free versus open source ideologies, but it’s important to understand the complexities we face as a society focused on problem solving.
Our personal ethics often do not form part of the licensing of a project, and that is a problem. FLOSS is built on the idea that there are no limits as to who can use a project, or what a project can be used for.
Recently Joe Redmon, core contributor to YOLO and significant contributor to computer vision as a whole, quit his research. No only YOLO, but all of the work he was doing to improve computer vision. He did so because of the ethical issues he was facing with how his computer vision technologies were being used to harm people.
Few developers face Joe’s ethical concerns and make the decision to walk away, and I believe that is a problem inherently in open source projects, and software as a whole.
Both FSF and OSI do not permit an ethics-based license due to the tenets of copyleft and the OSD respectively. This means new licenses are required if we’re to apply ethics to the rights of developers to use our contributions. And this means writing licenses that no longer comply with OSI. Does that mean they’re not open source? Not at all. It means they’re not OSI-approved, and that’s a distinction we need to recognise.
The impact of such ethics being wrapped into licenses includes adoption and support. The companies I often work with take on some rather questionable contracts and this would mean my own projects would see much less support and adoption by companies that have made a significant contribution to my own work.
But if Joe Redmon can make an exit from career-defining research, then it hardly seems fair that the rest of us would not put our ethics in writing and commit to them in our own work.
