How we maintain a healthy open source project
We could just say that an open source project is a public GitHub repository and an Apache License file. Boom — you can do it in 2 minutes. But that isn’t what we are going to cover here. Hosting providers like GitHub and Licenses like Apache are topics that deserve their own blog posts, and they are incredibly important decisions. But a healthy open source project requires some additional attributes, which is the focus of this article. We’ll start, briefly, on the definition of Open Source and the licensing that makes it possible.
Public Source Code — The textbook definition of “open source” is a project where “anyone can access the source code” — it doesn’t mean “free.” It simply means that you allow the whole world to look at how your software is constructed. I’ve always compared it to the ability to open up the hood of your car. You can look in and see how and why the steering works, where the pedals are connected, and what makes the wheels turn. Most people just want to drive the car and most people buy their cars pre-built. But everyone still has the right to open the hood of their car, understand how it works, take it apart, add to it, and change it should they choose to do so. That’s not traditionally been true of most consumer software, but it is true of open source. How you make this source code public is usually through a tool like GitHub, BitBucket, or GitLab. I’d recommend going to a good discussion forum or group like Reddit (use to be) and reading the comments to figure out the strengths and weaknesses. We use GitHub and like it okay. Its features are substantial and robust. I’ll say you could make worse decisions.
Open Source License — You need a clear well defined license that explains what others can do with your source code. For starters, sometimes all you can do is read it. To stick with our car analogy, you wouldn’t expect to have the right to make more Honda Civics and sell them. Some licenses allow you to make all the cars you want, but if you do, you have to give them away for free! And some licenses are highly permissive — more like free knowledge. It’s easy to add a license to your software — you just have to do it. You include the license with your source code and you don’t even have to write the license. Well-constructed and defensible licenses exist such as the Apache License, MIT, and BSD. When we took over maintenance of the core SpiffWorkflow library from Samual Abels (@knipknap), it was set to use the GNU Lesser General Public License or LGPL. With dozens of past contributors, it is VERY difficult to change the license now (we’d need every past contributor’s permission — that’s 44 people from around the world and across a decade). So we have continued to apply the LGPL to other projects for consistency and it is a very permissible license. Anyone can build any commercial application on top of our code. All this to say, the license you choose will have a profound long-term impact on the project and who is willing to adopt it. It is worth researching very carefully. A good place to start is with this Code Academy Article on Choosing An Open Source License.
The Real Requirements
The items above are obvious requirements for an open source project, but what are the less obvious needs that are still required to be successful? We’ve found it is a careful combination of Infrastructure, Community, and Funding. We’ll cover each of these in turn, based on ideas from major players in open source and from our own experience in creating SpiffWorkflow.
I had the pleasure of attending a webinar this week called “Monetizing Open Source” with some of the biggest names in the Python community: Andy Terrell, co-author of Conda, Wes McKinney co-creator of Pandas, Peter Wang co-founder of Anaconda, Travis Oliphant of NumPy, and Carol Willing of Project Jupyter. The ideas expressed here come in part from the excellent discussions from that meeting, related back to the discoveries and lessons from our own software development project, SpiffWorkflow. This group of successful open source developers are not wealthy tycoons of industry; they are the fathers and mothers of the underlying technology that makes our world turn. They aren’t about value extraction, but about the long-term sustainability of critical infrastructure. And that is the first thing a good open source project should strive to be.
Infrastructure
Successful open source projects tend to focus on generic infrastructure related issues. There are many critical problems that are not a part of a commercial company’s value proposition. An example might help. Consider a new AI startup that helps drive sales through targeted marketing. They should be spending their time thinking about how to create an ideal Language Learning Model (LLM) and focusing on the problem they are solving for their target audience. They should not be building a general purpose workflow tool to manage differences between each client’s established procedures or software choices. That workflow system is a rabbit hole; a massive distraction that will siphon off their best people at a time where they need to build marketable features directly related to the clients they are selling to. Most companies are under the clock: they have a certain number of months to become profitable before their funding runs out.
The webinar on “Monetizing Open Source” discussed the importance of targeting Infrastructure directly. The speakers pointed out the beautiful division line between “Product Solutions” that are the domain of commercial software and “Infrastructure” that is a natural domain of open source — a way to share the burden of building and maintaining solutions common to many organizations.
Our open source project, SpiffWorkflow, targets a general problem: how to model, execute and track complex workflows using a standardized diagram notation. It’s broadly applicable and it’s being used to improve workflows in universities and creating Web3 DAOs. It is useful for back office automation in large publicly traded companies yet also helpful for agile customer management at small startups. Being broadly applicable helps in the next major component to success: Community.
Community
“If a tree falls in the woods and no one is there to hear it, does it make a sound?” An open source project must have an engaged community. The fact that someone can access source code is very different from people actually doing it. It’s a bit like Schrodinger’s cat… If someone adds a license file to their GitHub project and no one ever looks at it, is it open source? I really don’t know the answer. A community takes dozens, if not hundreds, of individuals looking at the source code and getting involved. I’ve seen people build communities seemingly overnight — with 2k GitHub stars 3 days after it is published. I can’t tell you how to make such instant magic happen. For us, it’s taken many years of consistent effort. Here are the steps we’ve taken to amass 1.6k stars on GitHub, and build out a 300+ member discord channel. While those aren’t major numbers, they are evidence that we have created a small and active community:
- We invest heavily in documentation. We have a team member, Usama, whose primary role is ensuring every new feature is documented as it is released and published to our ReadTheDocs site. In addition to ReadTheDocs, we create videos and write articles (like this one) to help cover complex topics.
- We manage a Demo server where anyone in the world can test our project instantly to understand what it can do.
- We have a Discord chat room and watch it like a hawk. Every member of our team keeps an eye on that channel and responds to questions.
- We have a project website and advertise just a little bit to create some SEO. We keep a small trickle of funds aimed at a few key terms on Google that are very specific to us. This slowly introduces us to new people.
- We’ve published an article in a respected software magazine (D. Funk, “Creating a Low-Code Business Process Execution Platform With Python, BPMN, and DMN,” in IEEE Software, vol. 40, no. 1, pp. 9–17, Jan.-Feb. 2023, doi: 10.1109/MS.2022.3212033.)
- We ran a ProductHunt promotion to drum up interest from the entrepreneurial community.
- We’ve done a podcast on the supremely excellent Real Python Podcast with Christopher Bailey.
- We do Conferences (We’ll have a poster session at the upcoming PyCon in Pittsburg, so please come check us out).
- We host weekly Discussions on Discord and we have a way for anyone in the world to schedule a 45 minute meeting with a developer to ask questions.
Constant vigilance is how we are building our community. It’s politics, pure and simple — you kiss a lot of baby projects, shake a lot of developer hands. But how do you pay for these conferences and Google ads? And that’s just a small part of our monetary needs. The team of domain experts that our project requires all deserve to take home pay somewhere closer to the median of their profession. Funding is a real trick.
Funding
Startup Funds — We’ve been very fortunate to find a few key organizations that have helped us get started. Our initial funding came from the University of Virginia, a major Higher Education and Research institution. With their support we are able to solve a critical business need and breathe new life into an open source python library. You can read all about this project in this IEEE Software Magazine article I wrote last year: D. Funk, “Creating a Low-Code Business Process Execution Platform With Python, BPMN, and DMN,” in IEEE Software, vol. 40, no. 1, pp. 9–17, Jan.-Feb. 2023. In addition, Status (https://status.app) has helped fund the creation of an open source end to end platform on top of the original core library. Thanks to Status, multiple years of intense software development and QA effort are available under an LGPL license in the SpiffArena repository on GitHub. How did we land these first startup funds? We were a consulting company for many years and were able to build open source software as a part of our paid hourly efforts. I recommend you look for those kinds of opportunities: a paid position or contract where you are allowed to develop open source software. Build this into your negotiations for your next job or contract!
Sustainable Income — Part 1: These two organizations should not be responsible for supporting the infrastructure of dozens, if not hundreds, of the installations of SpiffWorkflow that exist today. We are working very hard these days to gather small bits of funding from many organizations. We are currently doing this through support contracts with companies that are acting in good faith. These are companies with capable leadership, smart enough to know they must fund the infrastructure on which they depend. I wish that all the world functioned in this intelligent forward-thinking way. We must search for ways to bring high value to these key organizations and give them all we can to allow them to differentiate themselves from their competitors. We are currently adding a set of extensions to SpiffWorkflow that we consider proprietary. We will make these extensions available under a permissive closed-source license to organizations that adopt one of our professional service retainers. This isn’t ideal — we don’t want to pull useful features from our open source product as it forces us to make difficult decisions. So we continue to look for alternative solutions.
Sustainable Income — Part 2: Our future is likely in the creation of “Product Solutions.” In this next phase of our development, we’ll be looking for ways to take our substantial knowledge of this open source platform and use it in partnership with domain experts to target specific industries. The commercial world is great at moving money, so we just have to play that game too. And the game is all about solving specific problems for profitable companies. If we can do this then we are “eating our own ice cream,” as one member of our community put it. We would be using our open source software to make a profit in the same way that everyone else is using our open source project. This removes the conflict of interest which is the difficult decision of what parts of our effort we open source and which parts we provide as closed source. I’m sure we’ll find issues with this approach as well, but it’s the best idea we have at the moment.
In Conclusion
It is fair to say that Funding, Community, and Infrastructure is not an exhaustive list, which is why I chose that modular origami design for the image in this post. There are certainly other key factors. Take transparency for instance — an important reason why we made time to write this article. We want everyone to understand what we are trying to do and how we are going about it. We want the community to know that our aim is to build a tool they can safely make a part of their own infrastructure. That’s a lot of trust, and transparency is a major part of building such trust.
All those other pieces that hold the big origami ball together? They are you. They are us. They are Elisabeth Esswein who now maintains the original SpiffWorkflow library, or Kevin Burnett and Jason Lantz who have built much of the end-to-end orchestration system on top of it. They are Alex Herron and Jon Herron, the father and son team that have introduced major features over the last year around Data Objects, and are now THE leading experts on BPMN in the Python world. They are Maudhurya Liyanage and Dinithi Jazeel who are solving the tangled knot of “test-driven” BPMN, and Ayoub Ait Lachgar who is shaping our diagram authoring environment. Just to name a few, among many.
We hope this was a helpful article and that it can be a resource to others out there who want to start their own successful open source projects. We love working on our own special project and seeing it take shape and take flight. It is possible.
If you have any thoughts to add about what makes an open source project successful, we’d love to hear from you in the comments.
About Our Open Source Project: SpiffWorkflow
Our open source project is called SpiffWorkflow, a low-code workflow orchestration platform. That’s a lot to unpack, but each word is important. Imagine drawing a detailed flow-chart as a part of a business meeting and it instantly works — your business rules change the moment you agree on the process. That’s basically the idea, and there are dozens, if not hundreds, of low-code systems that promise something similar. We are different in that we are building open source software (LGPL) based on open standards (BPMN) in an intuitive and well-loved programming language (Python). We’ve open sourced a whole platform — not just a library, engine, or extension. This is more on the level of a language implementation or a framework, which greatly increases the amount of documentation, testing, and planning efforts involved in the project. All of this takes a significant investment in time and effort and this article is about how to maintain that effort.
