The Evolution of Open Source — Standing on the Shoulders of Giants
This blog post was originally published at https://www.polidea.com/blog/
Over the last 10 years, our team has created a good number of open source libraries. Android BLE, React Native BLE, iOS and OSX BLE for RxSwift, just to name the most popular ones. But open source is about so much more than just good old “humblebrag” — it’s about sharing knowledge. At Polidea the belief that what we do at work can benefit other developers and businesses has been with us from the beginning. This is why we try to drive #CodeLeadership through meetups, conferences, workshops, and sharing code. Today, follow me into the story of open source evolution and how it affected the work and lives of us — humble engineers.
Past
The past of open source
I’ve started my engineering career when the open-source movement was still in its very early days. The Internet was not a thing yet, 8-bit computers were all that you could (barely) get your hands on. There were BBSs (Bulletin Board Systems for those who don’t remember) — which you could connect to via expensive and slow modems and download a code someone published. We’ve been behind the Iron Curtain in Poland, and even if there were signs of freedom, you could not (even if you had money) buy the software you wanted officially. Most of the software you could use was a pirated copy that you had either downloaded over the slow BBS modem or exchanged at one of the shady booths at “Computer market” — the fair that was taking place every weekend in the center of Warsaw. All software was “open” but in a very twisted way.
Gee, how much has changed since then!
Pay-for-software
After 1989 things started to get more civilized but back then you had to pay for the software if you were going to make any money out of it. It took many years for people to understand that it makes sense to pay — even if it’s super easy to copy — because we should reward the authors for the work they’ve done.
The business model of developing software was fairly straightforward. You built software, you owned it, and you asked everyone using it to pay for it — in the form of the one-time payment or the license fee. Sometimes people went to great lengths to protect their software — they would even include hardware keys to run it. However, those efforts were often counter-productive. There is a story (which might be taken with a grain of salt) that Lotus 123 spreadsheet had too sophisticated security and lost with Excel precisely because the latter was easier to pirate. People learned to use it at home thanks to their pirated copies and then when it came to paying for it — they were more willing to do so.
Try-before-you-buy
Pay-for-software was starting to be not good enough. It was one of many signs that the traditional business model of paying for software was probably not the best one, and if taken literally, it might even stifle innovation. There were several different models — shareware was a thing for a while (limited version of the software you could use for some time before buying it), but it never really succeeded. Viruses, malware, and adware started to spread around with the freeware software, and it was clear that if you start using some software for free, you need to be able to confirm that you are not getting some extra “passengers” along with it.
Using others’ software was crucial for us — software engineers — “standing on the shoulders of giants” was the fastest way to accelerate the computer revolution. I remember the early days of the development when we spent many hours developing schedulers, loggers, object-relational mappings and many other pieces of software that were needed but they were never at the core of our business. Paying for all those components was possible, but the whole process with commercial software was not engineering-friendly. It was difficult to compare different commercial stacks — you had to deal with salespeople and marketing promises rather than data and facts. Try-before-you-buy was fairly limited, and once you bought into a commercial stack, usually the vendor lock-in kicked in. It was clear that to accelerate innovation software should be more accessible, easy to try, understand and be more “open”. The only problem was with rewarding the authors of the software. Not to mention software engineers who spent their time developing it and should — just like us — be rewarded for the work they did.
Present
The time for open-source movement had come — the revolution has begun.
It started with GNU and Linux, of course, but then it spread like wildfire. The engineers inspired by noble goals led the open-source movement. However, it would not be able to reach its current size with just volunteers working on their spare evenings and weekends. Again — people who contributed to the open-source projects needed to be rewarded — fame is not always enough. Working on open-source projects is about gaining experience, recognition, or can be simply about contributing to something for the greater good. However, the long-term, stable contributions to big open-source projects done by experienced engineers require stable financing — people need to get paid for high-quality work, especially when such efforts involve team coordination.
Who pays the bills?
There are several ways of funding open-source work. A lot of commercial companies have been built around open-source software — for example, IBM-acquired RedHat, GitLab, and many many others. It’s a perfectly viable model to make extra services and provide additional, commercial features or dual license in an open-source project. Another option is to simply build your own open-source project so that you can use it in your commercial offering (this way, you can show your expertise in the given area). This is what Polidea did with its most prominent BLE suite of React libraries for iOS, Android, React Native and Flutter. Investing time of your people in building open-source projects might prove to be fruitful and many big companies are aware of it. They realize that open source is where innovation happens and that it is a good business model. Such projects changed the tech landscape in many respects — think about the Android Open Source Project by Google, React Native by Facebook, or Kubernetes developed originally in Google.
Many to rule them all
Still, who should rule in the open-software world? It’s evident that despite their open and distributed nature, the projects mentioned above have some governance, leadership, and vision. Many of them originally come from big players, but while getting more critical, their “ownership” may become a liability. You do not want to rely heavily on something that is controlled by your competition. That’s where the non-profit organizations like Apache Software Foundation, and more recently, Cloud Native Foundation and even more recently GraphQL Foundation come in (and these are just a few notable examples). They have been established to provide governance, structure, leadership, and, most of all, independence from the original owners of the projects.
One example of the projects that Polidea is recently involved in is Apache Airflow, which is a Top-Level-Project at the Apache Software Foundation. Initially developed by Airbnb and then donated to Apache Software Foundation, it already has more than 300 (as of today) happy, big customers such as Google, Tesla, Twitter and more. The contributors and core maintainers come from different companies all over the world. It’s now being used in the commercial offering as the Composer in Google Cloud Platform, and is becoming the first-choice solution for managing workflows in big data processing pipelines. There are also other projects we are involved in — like Apache Beam that originated from Google where the governance and structure are different but it still follows the same core principles of Apache Software Foundation — a so-called “Apache Way”. For many users, the Apache stamp of approval is an indicator that the project is stable, has a thriving community and is led in a neutral-vendor way. These are the most important factors when deciding which software you’re going to use as the foundation for your business.
The funding problem remains, however.
Making it work
Interestingly, the non-profit organizations governing such projects do not pay for creating the software. They provide brand, legal protection, common goals, vision way of work and governance structure, but they do not pay for the software itself (even if they own it eventually). Apache Software Foundation for one owns the software that is worth more than \\\\$20B, yet not a single person is paid by the ASF to create the software (nor to manage the Foundation). So the issue of funding the work remains. There are several ways open-source projects can be funded. When you are a company using such projects, you often want to develop new features or simply connect the project to your components and infrastructure. This means you pay your employees to develop it. You can also become the sponsor and pay external software houses to do it.
They should understand the open-source culture, know how to contribute and deliver the highest quality. Their teams can navigate autonomously in all the community discussions and contributions. In the end, they do not simply implement new features that are required, but they have to do it by following the community rules and agreeing with the rest of the contributors. They usually implement a number of generic improvements along the way, because they care about the community. This is a win-win-win situation — a software house makes the profit, people get paid for doing meaningful and interesting tasks, the community projects have new, useful features, while the sponsoring companies get what they need and more. This is just one of the models of funding an open-source work — the one that provides stability, predictability and clear benefits to all parties involved. There are other models worth checking out, however at Polidea we are the most familiar with this one as we follow it in our Apache Beam and Apache Airflow projects.
Future
Open-source (r)evolution
What will the open source look like in the future? Nobody can be sure, but just by looking at how dynamically it evolved since the beginning of the open-source revolution, the future will most likely surprise us! Nevertheless, there are a number of exciting projects and trends that are worth mentioning.
First of all, there is a big revolution happening right now in the sphere of communication and collaboration tools. Github and Slack have changed the way people take part in discussions, engage in arguments and reach agreements. Reaching consensus is one of the most time-consuming elements of the development process. The code is what you get at the end, but the road leading to it is crucial.
The communication had initially begun in the developers’ mailing lists. While Github and Slack (among many others) have definitely revolutionized the collaboration in general, their centralized and “proprietary” nature may not be enough. Keeping the discussion history, project plans, design decisions, and collaboration features still doesn’t work for big, well-governed open-source projects like those run under Apache Software Foundation. There is even a well-respected rule in the Apache community saying that “if it did not happen on the mailing list — it did not happen at all”. The mailing lists are publicly available, archived in the ASF and elsewhere and they are the single source of truth for anything that happens in these projects. There are many ways in which these modern communication tools might serve the needs of such projects by providing open-source versions of their software, making open standards, donating their projects while building their offering on commercial products. These actions may help to deal with the community’s reluctance to use proprietary solutions.
Additionally, Github introduced the “Github Sponsors” feature that allows sponsors to pay developers for their open-source contributions. This is just the beginning and in my opinion, more similar initiatives are going to emerge soon. However, for those changes to be universally accepted both for communication and funding, some values need to be met: vendor-neutrality, full transparency, and free-for-commercial use licensing. These are fundamental values that the open-source community builds on. You can be sure that no tool, service or standard will be widely accepted by the community if it lacks either one of them.
Blockchain on the horizon
If you look at the tech landscape, there is a new possibility just around the corner that might provide tools that are in-line with the fundamental values of the community: Blockchain. Satoshi Nakamoto published his paper about bitcoin currency 10 years ago and it seems that we are reaching the point when some useful solutions are finally being built. Most of the blockchain solutions are open-sourced and created with the purest open-source mindset. Actually, when you look closer — git, which is basically what powers Github — is a widely successful blockchain-based tool. Perhaps it reached the biggest adoption from blockchain-based tools. If so, maybe it’s time for the open-source software community to start using more of the blockchain elsewhere?
Blockchain solutions seem to be perfect for providing an independent, auditable and immutable track of design decisions and discussions. That’s not the end though — they can also solve the problem of giving financial rewards for the contributors. It’s easy to imagine a future of open-source projects managed by organizations like Apache Software Foundation, in which the developers will be financially rewarded for implementing features via smart contracts recorded in the blockchain. If a big company like Google would like to have some features developed, it could offer a contract with the automated fulfillment criteria check. It’s a bit out of reach yet, but there are already projects trying to do precisely this, with GitCoin being the most recognizable.
This is just the beginning. Companies like Slack and Github — or even communities themselves without any third parties — could make sure that the work done so far by the open-source community — like discussions, design decisions, issues, project plans, sponsorships, and donations — is recorded in the public blockchain and made available. Similarly, what git did to code, this could be the next step to make organizations like Apache Software Foundation use those commercially available tools and be sure that they can do it while following open source values. Wouldn’t that be something we can call “a truly open governance model”? If I were to say where the future of open-source lies, that would be my bet.
