Automation: The Possible And The Reasonable
At Bench, we combine the power of humans and software to provide accurate and timely financial books to thousands of entrepreneurs. In Bench’s engineering department, we provide the software side of that equation. We build the tools and automation to support the workflow of Bench’s bookkeepers — the aforementioned humans — in producing those books.
In this article I’m going to talk about automation. We strive to automate as much as is possible and also reasonable:
- It’s possible if it can be achieved entirely by computer without human intervention,
- It’s reasonable if the automation produces results as good as or better than a human
Not all parts of the bookkeeping workflow fall under these categories, and there is a tension between the two. I’m going to describe a time when we over-emphasized possible to the detriment of reasonable — and what we learned.
Welcome to Bookkeeping
First, a little background. As mentioned, Bench produces financial books, which are known as ledgers (these used to be actual physical books, of course its all digital these days). These ledgers are records of all the financial transactions of a business, and are ultimately used to produce financial documents: income statements, balance sheets, tax returns, that sort of thing. But before they are used, they must be reviewed to be sure that they are balanced — that is, the sum of a ledger’s transactions equals a ledger’s current balance. If not, data is either missing or incorrect.
As you can probably guess, we want to automate as much of this as is possible and reasonable. Fortunately we can sync data from a business’ financial institutions (banks, credit cards) directly into our software. Automating this is a bit of a no-brainer.
Where it gets thornier is determining which accounts should be connected to which ledger — i.e. the ledger in which to record accounts’ transactions and balances. This is typically a 1–1 mapping, but it is often not obvious when dealing with corporate credit cards that have many actual physical cards for a business’s employees. And every financial institution represents this relationship differently.
This mapping can represent a large burden for a bookkeeper — so we asked the usual question — can we automate it?
T. Rex on the Loose
“[They] were so preoccupied with whether they could, they didn’t stop to think if they should.” — Ian Malcolm, Jurassic Park
So an algorithm was devised to do just that — it would find the combinations of accounts that balanced, when mapped to a ledger. All the bookkeepers had to do was tell our software which ledgers to point at which institutions. From there, the transactions would appear under those ledgers, in-app. The software did all this behind-the-scenes, without requiring any further intervention or confirmation from the bookkeeper (at least in theory).
Sounds simple enough, but there was a fundamental flaw in making this fully automated: there can be multiple possible mappings that appear correct. but only one that is. A mapping could appear correct even when it was not. And there was no way to determine which was which by algorithm alone. This was exacerbated by the algorithm having no memory of past mappings — the mapping could change over time, which meant a ledger could have transactions from multiple unrelated accounts.
This led to a bad situation. Some books generated by this algorithm would appear to be correct mathematically but could contain incorrect or missing transactions. For example, a purchase of an airline ticket could appear on the wrong employee’s card, or be missed entirely. And there was no straightforward way to tell which ledgers were affected (luckily, there are many layers to Bench’s quality controls and all ledgers are reviewed multiple times. Retroactive investigation confirmed all invalid data was corrected before books were finalized).
This was an algorithm gone wild.
The Aftermath
When the issue with the quality coming out of automation was raised, we immediately halted the production of financial statements to our clients until we could perform a full audit to determine the scope of the problem.
The problem was made worse by the fact that the transactions were stored in our databases in a way that over-emphasized the ledger relationship, and under-emphasized the account a transaction came from. This made it difficult to perform any forensics on the data in these ledgers.
The result of this is that we needed to build a new ad-hoc database so we could run some very involved analytics queries to mark potential problem ledgers. These ledgers then had to be manually reviewed in a process involving human eyes on database records, PDF bank statements, and Excel documents produced by our app. The total effort here was many person-weeks, across engineers, data analysts, bookkeepers, and internal support.
Our efforts here were successful, but like the survivors of Jurassic Park, we needed to take stock of what had happened.
The Lessons Learned
The overarching lesson here is to not put blind faith in automation. Always be introspective and ask yourself and your team “this is possible, but is it reasonable?” This means being sure of an algorithm’s correctness before it can run without careful human oversight.
In our case, we hid too much from our bookkeepers. They had no way of controlling or even seeing which accounts were mapped to which ledgers, leaving them in the dark. The human part of the equation was lost. If they had the proper tools, and could at least confirm the mappings, the situation could have been almost entirely avoided.
The other side of this was we had no database of the raw financial data, or any historical data on where and when it came into our system or how it was validated. This made it nigh-impossible to provide visibility to bookkeepers, proactively look for trends/patterns in the data, or perform post-mortem forensics when errors arose.
So here’re the key lessons, distilled:
- Make it clear to users what your software is doing
- Provide them the tools to use their knowledge when automation alone is not enough — software is better when it can accommodate human judgement and intervention
- Keep data persistent, visible, and queryable to engineers and analysts
Going Forward
So what are we doing about it now?
Well the first thing we did was to kill the automated mapping algorithm, and rely solely on bookkeepers to map accounts to ledgers. This gave us the breathing space in engineering to find the right solutions.
The next step was to build out an entirely new microservice for syncing and mapping financial data to ledgers, complete with a full database. While this is still in process, it has already paid dividends in allowing us to spot common trends in the data provided to us. We’ve been able to provide both better data to the bookkeepers and also better handle irregularities in the data in code. This raw data also opens up the possibilities of extending our product to support client and bookkeeper use cases that require real-time transactional data as opposed to fully reviewed financials.
None of this was possible without a good data store.
We are not yet doing any auto-mapping, and may never will, at least not 100%. If and when we do, it will only be in situations where we have can certainty the mapping is correct — first by manual confirmation, and then codifying what we learn from the manual process into the software.
As we continue, we can move the line of what is possible to automate farther into what is reasonable, by careful, measured steps. And we all can rest easy knowing that while no software is perfect, we’ve made our best effort to make it reasonable.
—
Written by Craig Warren, Backend Software Engineer at Bench
If you are interested in learning more about Bench Accounting or a career with our Engineering team, apply here: https://bench.co/careers/
Originally published at https://medium.com on December 10, 2018.
