Meta-Business Analysis, or Reviving Spreadsheets as Useful Tools

From Wikipedia:

The UNIX philosophy is documented by Doug McIlroy[1] in the The Bell System Technical Journal from 1978:[2]
* Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”.
* Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.
* Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.
* Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.

Could you build a business this way?

Replace “program” and “software” with “business process”.

How do you stitch your business processes together? In Unix, most programs expect a text stream as input, and output another text stream. Text streams are universal and very flexible, but they require a lot of work in rebuilding the context of the information (the meta-information) from the stream after every step.

I have a modest proposal for an interchange format for business processes: the spreadsheet.

Spreadsheets were the first “killer app” for PCs back in the 1980s (VisiCalc). Excel, which allowed the use of the mouse, crushed Quattro Pro and other pretenders, and is still the leader in the spreadsheet world. Google has introduced Sheets, and others have come up with spreadsheet replacements or extensions that allow similar functionality in a cloudy, SaaS-y world.

As an enterprise software developer, I have wished for the death of the spreadsheet many times, since the versatility of the tool leads to the spread of multiple versions of data, with no clear System of Record. Spreadsheets can be stored locally or in the cloud, edited by multiple authors, and stored in multiple revisions. Individual cells are not reliable as data stores, since the definition of the cell is entirely based on the context around it.

And yet…

Nobody wants to give up the freedom to experiment, calculate, modify, play “what-if?”, chart, graph, optimize, and learn that a spreadsheet affords.

Let’s use the versatility of the spreadsheet as an advantage.

How would this work?

  1. Templates: the basis of disciplined use of spreadsheets is frictionless access to a template that constrains the format of the spreadsheet, allowing some automated understanding of the content of the document. In this way, a spreadsheet template is inter-convertible with an XML or JSON or YAML document.
  2. Automated Spreadsheet reading and writing: almost every modern language has a well-developed library for reading and writing spreadsheets, especially in the .xlsx and .ods formats, which are open standards. These libraries make interacting with spreadsheets a relatively trivial problem, except for the lack of standards in spreadsheet content (but for that problem, see #1, Templates).
  3. Stable Spreadsheet Storage Location: This concept would depend on a reliable, well-known location for each spreadsheet. This implies cloud storage of some kind, which is a solved problem with many possible solutions.
  4. Spreadsheets for configuration: Instead of configuration files (which today are often in some unique text format, or in YAML, JSON or XML), a company could use spreadsheets (using a template). The benefit of this is the ease of editing of configuration. The downside of this is the ease of editing…
  5. Business Process Definition using a Spreadsheet: There are existing standards in this area: BPEL and BPMN. There is no need to re-invent the wheel, but simplifying the notation and tooling by using some spreadsheet templates could democratize the world of business process automation.
  6. Business Process Meta-definition: I have been describing a business analysis framework that breaks businesses down into event-driven processes. What we need is a set of templates that define the major types of event/process groupings, and then another set of templates that define the players (API connections, human roles) and the data packets that are communicated (via forms that are filled out, or communications from custom apps, or API calls and responses).
  7. Business Process Architecture: Given a definition from step #6, software could auto-generate an application that would run a process, including connecting to data sources and destinations, collecting input from users, and instrumenting the process for management visibility. This wouldn’t have the user-friendliness of a custom application, but it could act as the glue behind the scenes that keeps the business running.

Why not use a GUI for this? Because spreadsheets already have a GUI and are almost universally understood. And current GUIs are expensive and restrict you to their associated set of tools.

Why not use text files? Because spreadsheets can be extensively self-documenting, and templates can restrict input types and do basic reality checks on anything a user types in a field.

The end result would be 3 layers of business documents.

  1. One layer would define the process, connections, and data types of a process (the meta-data).
  2. The second layer would consist of data — spreadsheets that collect the raw data from the process (organized according to the definitions in the first layer) and allow analysis using the full power of the spreadsheet program.
  3. The third layer would be the actual operating software, including mobile apps, web forms, third-party SaaS apps, custom programs, boxed software, and even spreadsheets, that together make up the current state of the art of business software.

So business analysis would start with layer 3, figure out how to extract data to some format that can be inter-converted with spreadsheets (this requires a library of connectors to various software products) and then finally the meta-analysis of the first layer.

The benefit? Configurable business processes and applications, defined in a spreadsheet. Change a value in a cell, and that change propagates through the entire application.

It’s a nightmare. Too easy for disasters to happen.

So we add some disciplines from current software engineering — we use source control on the spreadsheets. We create sandboxes to test changes to metadata.

And the benefits! Complex business rules can be defined in a business rule spreadsheet, and changed as needed. New tools can be added to a toolchain by adding a spreadsheet with their configuration requirements, and then linking them in a process definition spreadsheet.

I don’t recommend this for a large corporation, as there will be a rat’s nest of spreadsheets that will be impossible to index and understand. But for small businesses, this would be an improvement on the status quo, which is no software architecture, fear of new tools, expensive integrations, and lost opportunities.

Someone somewhere has to have already built this system — please point it out to me in the comments. I will be building my version of this system over the summer of 2017. Let me know if you are interested and I’ll add you to a mailing list as the project matures.

Of course, please point out all of my mistakes and why it won’t work! Or connect with me and let’s build this together!