SpiffWorkflow, A New Hope

Our progress towards a complete workflow management system

SpiffWorkflow is an open source workflow engine written in Python, and it is growing. This article will introduce some new projects we are undertaking to turn SpiffWorkflow into a full featured workflow management system. Our goal with this article is to find collaborators and supporters for the effort.

SpiffWorkflow is useful if you want your non-coding teammates to have some real control over your application’s business logic by creating and modifying flow-chart like diagrams. Rather than use these diagrams to “automate business processes”, why not use them to build strong collaborative teams that have shared purpose and empathy for one another? Such needs don’t just exist in large banks and corporations, they happen in every business, in every research project of every size, everywhere. SpiffWorkflow allows you to add a little empathy to your python code. With it, you can give your teammates some real control and understanding of the applications you build together.

Using flowchart diagrams to control software applications isn’t a new idea. Terms like “low-code” and “process automation” are used to describe similar projects, and there are a lot of them. Spiffworkflow is similar to the best of these projects in its rigorous adoption of the excellent BPMN 2.0 specification (the most broadly studied and implemented notation for diagramming business processes that exists). SpiffWorkflow is unique in that it is a library that can be fully integrated into your existing applications and it is 100% pure unadulterated Python.

There is another Medium article with a more complete introduction to SpiffWorkflow, and you can dig into the detailed documentation if you want to learn more.

You can go pick up SpiffWorkflow right now and integrate it into your application. You can trust that it will be well supported. With 1200 stars and growing on GitHub, there is clear evidence of a healthy community. Our consulting company, Sartography has multiple clients who are all actively integrating the library into their successful, well funded, and growing businesses. It is thanks to them that the effort we are describing here is possible.

Looking to the Future

We have a few specific goals in mind for the near future. Here is a very brief high-level overview.

Custom Diagram Authoring Tools

We want to make developing BPMN diagrams simple, intuitive, and deeply connected to your application. A custom editing environment will make it easy for people to draw their diagrams, while simultaneously editing their Python Scripts, creating web forms and surveys, managing data storage and communicating with external systems.

Process Management and Reporting

We are creating tools to handle the change management of BPMN diagrams through a tight integration with GIT. We will make it easy to edit files in place and manage multiple instances of your application (testing/staging and production).

We will also support the management and monitoring of running process instances. Record a history of actions as they execute. We will assure it is possible to pause running instances, move back to previous point, and make partial updates to running workflows.

Respect for the Standard

We are continuing to enhance SpiffWorkflow to meet more and more of the BPMN 2.0 specification. We have the unique opportunity to approach the standard from a fresh perspective while improving upon what so many others have done in the past. Data Management and External Messages will be the focus of two upcoming articles, which we hope to release in the next few weeks.

Well Composed Systems

We will ensure that we follow an “API First” approach, creating a clean and intuitive programming interface that will allow people to pick and choose which aspects of the system they need. Our applications will use an established modular web framework that will allow you to replace things like authentication and permissions to meet your specific needs. The “Frontend” and “Backend” will be separate applications that can be replaced in their entirety if you are integrating Spiffworkflow into an existing system.

Open Authentication and Permissions

SpiffWorkflow’s backend and frontend will work with the OpenID Connect Standard, integrating with common authentication providers like Google and Facebook if you are supporting the world, or systems such as KeyCloak or Active Directory for the more corporately minded.

Components

So what makes up a “Complete Workflow Management System”? We have the current SpiffWorkflow project on GitHub. Much that you can do with that core library will be common across all projects. So we are building a set of related and interconnected systems around SpiffWorkflow that you can adopt as needed to rapidly create your own workflow systems. Included below is a description of each, along with a link to the code on GitHub which is released under the LGPL.

1. Custom Diagram Modeler (bpmn-js-spiffworkflow): We have created a set of custom extensions to BPMN.js (one of the open source tools available at https://bpmn.io), the powerful, extensible, open source editor created by Camunda. These extensions allow you to access properties and features of SpiffWorkflow from within the diagram editor. This new editing environment will allow a number of critical features. Our first efforts here will focus on integrating the React Json Schema Form Editor, a powerful tool for building online web forms. We will also work on a native Python coding environment similar to Jupyter to make adding code quick, intuitive, and error resistant.
2. Process Engine (spiffworkflow-backend): As stated earlier, SpiffWorkflow is a library. To use it, you need to build an application that can handle things libraries typically do not do, such as database persistence, API endpoints, and scheduling. This new process engine will wrap SpiffWorkflow and provide an “out of the box” system that can execute arbitrary BPMN diagrams. It is built in Flask, and composed of a series of BluePrints so that it can be integrated into existing applications.
3. User Interface (spiffworkflow-frontend): We are creating a React application that can talk to the Process Engine above, allowing end users to navigate their way through a process instance, complete forms, and hand off tasks to other users. It will provide a powerful administrative interface as well, allowing administrators to monitor running processes and step in to make corrections to those processes when required. Think of this as the other side of the Custom Diagram Modeler — what you design there will be rendered in this end user interface.

Timeline

We will be dedicated to these goals throughout 2022. You should expect several releases of SpiffWorkflow during this time, with a significant major release in the next few months. Then 2023 will see these new code bases in active use by a few early adopters. As things solidify, we hope to get all of our systems officially published and released within the next year.

Thanks

This opportunity to expand SpiffWorkflow into a complete workflow management system would not be possible without a recent partnership with Status.io. We are deeply grateful to Jarrad Hope, for believing in our project and giving us this chance to create something new. We are also thankful to the University of Virginia which has provided so much support over the past few years in building out the core SpiffWorkflow library. And thanks of course to Samuel Abels (@knipknap) for originally creating SpiffWorkflow and maintaining it for nearly a decade.

Get Involved

These are exciting times! We would love to expand our effort beyond our current small team. Additional funding to help us accelerate our development internally, or collaborations where we work together on some common goals would be very welcome. We look forward to building something of merit. We hope that we can improve the way we work together, and make building things as a team fun, rewarding, and efficient.