Tracking State over Time with Snapshot Diffing

Every now and then something comes along and revolutionizes part of my development workflow. Something that’s actually fun instead of the typical laborious and dreary process of development. In the past year, that thing has been snapshot testing.

This is going to be a technical post. If you’re here for product posts, you might want to skip this one. I’m going to show how snapshot testing made it a piece of cake to develop and solidify the process of importing transactions in Actual.

I’ve talked before about how snapshot testing removes a major pain point with testing: updating mock data that tests consume force you to update all the tests as well. When using snapshots, adding a simple flag to Jest updates all the snapshots automatically.

But it goes much deeper than that. Let me start by explaining snapshots: all it takes is one line like expect(object).toMatchSnapshot(), and the current state of object will be serialized and stored away. Later, it sees a snapshot exists and will compare them, failing if the current test's snapshot differs. It's an amazingly quick way of seeing the state at a certain point in time.

That’s the real jewel of snapshot testing: it makes it trivial to see changes over time, because you can easily add as many toMatchSnapshot calls as you want as the test runs. You can track as little or much of the system as you want, at any point in time that you care about. With basically no effort.

Ok, let’s calm down a bit. Here’s what it looks like. I’ve recently been working on the transaction import process, which I’ll blog more about soon. There are many ways to add transactions: manually entering them, importing a file, or importing from an online service. Importing is a bit complex because of how it reconciles transactions; there are lots of edge cases. I need to make sure the transaction data is sane after an import in a specific context.

Ignoring a bunch of test setup, here’s a basic test for importing transactions from an online service:

monthUtils.currentDay = () => '2017-10-15';
expect(await getAllTransactions()).toMatchSnapshot();
await syncAccount(accountId);
expect(await getAllTransactions()).toMatchSnapshot();

First, it sets the date in the system so it’ll always get the same transactions back from the test syncing server. It saves the state of the transactions (which should be empty), imports transactions, and then saves the next state. With 4 lines of code I’ve guaranteed that transactions are empty at first, all of the new transactions are added, and all their data is correctly processed (data from the online service is in a different format and needs to be converted).

You could argue that the first expect should simply be something like expect((await getAllTransactions()).length).toBe(0). This is a valid point and I was starting simple to show how snapshots make it easy to track state over time. Writing tests is always subjective and it's up to you to decide which style is more beneficial (I address this point more at the end).

There are a lot of edge cases I need to test that involve importing multiple times. My tests actually do something like this:

monthUtils.currentDay = () => '2017-10-15';
await syncAccount(accountId);
expect(await getAllTransactions()).toMatchSnapshot();

monthUtils.currentDay = () => '2017-10-17';
await syncAccount(accountId);
expect(await getAllTransactions()).toMatchSnapshot();

This gets the initial transaction set as of 2017–10–15, then imports on the day of 2017–10–17 to add new transactions. The two snapshots will capture these states. More complicated tests involve up to 4 or 5 imports in various conditions, and I need to track the state between each one.

We start running into a problem though. The snapshots are too broad and contain too much unrelated data. The first snapshot is fine — on first import it should snapshot all the transactions. But I intend the next snapshot only to assert that new transactions have been added, and nothing else has changed. The snapshot is just a list of all transactions though. Looking at that list tells me nothing of what is actually being tested. That’s where snapshot diffing comes in.

Snapshot Diffing

The library snapshot-diff allows you to not only track state over time, but track how it changes. Instead of saving an entire snapshot, it just saves a diff of what’s changed.

Using this, my test would look like this:

monthUtils.currentDay = () => '2017-10-15';
await syncAccount(accountId);
let lastTransactions = await getAllTransactions();
expect(lastTransactions).toMatchSnapshot();

monthUtils.currentDay = () => '2017-10-17';
await syncAccount(accountId);
expect(
  snapshotDiff(lastTransactions, await getAllTransactions())
).toMatchSnapshot();

This saves only a diff of the transactions. The second snapshot looks like this:

exports[`Sync import adds transactions 2`] = `
"Snapshot Diff:
- First value
+ Second value

@@ -1,8 +1,26 @@
  Array [
    Object {
      \\"acct\\": \\"one\\",
+     \\"amount\\": -2947,
+     \\"category\\": null,
+     \\"date\\": 20171015,
+     \\"description\\": \\"lowes\\",
+     \\"error\\": null,
+     \\"financial_id\\": null,
+     \\"id\\": \\"one\\",
+     \\"imported_description\\": null,
+     \\"income_month_flag\\": 0,
+     \\"isChild\\": 0,
+     \\"isParent\\": 0,
+     \\"location\\": null,
+     \\"notes\\": null,
+     \\"starting_balance_flag\\": 0,
+     \\"type\\": null,
+   },
+   Object {
+     \\"acct\\": \\"one\\",
      \\"amount\\": -5093,
      \\"category\\": null,
      \\"date\\": 20171015,
      \\"description\\": \\"Transaction 51\\",
      \\"error\\": null,
@@ -123,6 +141,42 @@
      \\"location\\": null,
      \\"notes\\": null,
      \\"starting_balance_flag\\": 0,
      \\"type\\": null,
    },
+   Object {
+     \\"acct\\": \\"one\\",
+     \\"amount\\": -2947,
+     \\"category\\": null,
+     \\"date\\": 20171017,
+     \\"description\\": \\"macy\\",
+     \\"error\\": null,
+     \\"financial_id\\": null,
+     \\"id\\": \\"three\\",
+     \\"imported_description\\": null,
+     \\"income_month_flag\\": 0,
+     \\"isChild\\": 0,
+     \\"isParent\\": 0,
+     \\"location\\": null,
+     \\"notes\\": null,
+     \\"starting_balance_flag\\": 0,
+     \\"type\\": null,
+   },
+   Object {
+     \\"acct\\": \\"one\\",
+     \\"amount\\": -2947,
+     \\"category\\": null,
+     \\"date\\": 20171016,
+     \\"description\\": \\"papa johns\\",
+     \\"error\\": null,
+     \\"financial_id\\": null,
+     \\"id\\": \\"two\\",
+     \\"imported_description\\": null,
+     \\"income_month_flag\\": 0,
+     \\"isChild\\": 0,
+     \\"isParent\\": 0,
+     \\"location\\": null,
+     \\"notes\\": null,
+     \\"starting_balance_flag\\": 0,
+     \\"type\\": null,
+   },
  ]"
`;

Looking at this snapshot, it’s clear that it added 3 transactions, and nothing else changed. Which is exactly what I want to see. (Note: As part of my test setup, I also reduced the amount of transactions in the mock data. Reducing the size of the dataset makes snapshots easier to read as well.)

I found this API a little cumbersome though. I want to quickly test changes over time, so I made this utility:

function expectSnapshotWithDiffer(initialValue) {
  let currentValue = initialValue;
  expect(initialValue).toMatchSnapshot();
  return {
    expectToMatchDiff: value => {
      expect(snapshotDiff(currentValue,  value)).toMatchSnapshot();
      currentValue = value;
    }
  };
}

This makes it super easy to snapshot diffs over time. The above test would now look like:

monthUtils.currentDay = () => '2017-10-15';
await syncAccount(accountId);
let differ = expectSnapshotWithDiffer(await getAllTransactions());

monthUtils.currentDay = () => '2017-10-17';
await syncAccount(accountId);
differ.expectToMatchDiff(await getAllTransactions())

We can simply call differ.expectToMatchDiff to snapshot a diff, and the current value is replaced with that value, so only new changes are snapshotted over time.

This makes it amazingly simple to test a complex process: if you have manually entered new transactions, when importing transactions from online Actual should match transactions from the bank with the ones you’ve manually entered. Here is almost the whole test taken straight from Actual:

monthUtils.currentDay = () => '2017-10-15';
await syncAccount(accountId);
let differ = expectSnapshotWithDiffer(await getAllTransactions());

await db.insertTransaction({
  id: 'one',
  acct: id,
  amount: -2947,
  date: '2017-10-15',
  description: 'lowes'
});
//
// Add 2 more transactions manually...
//

differ.expectToMatchDiff(await getAllTransactions());

monthUtils.currentDay = () => '2017-10-17';
await syncAccount(accountId);

differ.expectToMatchDiff(await getAllTransactions());

The first snapshot lists all the initial transactions. The second one looks like the snapshot I pasted above: it clearly shows that only 3 transactions were added (full code manually adds 3 transactions). The third snapshot shows that all 3 transactions were matched and updated with data from the online import, and one new one was added. Here’s the whole snapshot:

exports[`Sync import matches multiple transactions 3`] = `
"Snapshot Diff:
- First value
+ Second value

@@ -1,16 +1,16 @@
  Array [
    Object {
      \\"acct\\": \\"one\\",
      \\"amount\\": -2947,
      \\"category\\": null,
-     \\"date\\": 20171015,
+     \\"date\\": 20171017,
      \\"description\\": \\"lowes\\",
      \\"error\\": null,
-     \\"financial_id\\": null,
+     \\"financial_id\\": \\"622f7b61-a6be-4ce5-bd2f-50eb14c12f42\\",
      \\"id\\": \\"one\\",
-     \\"imported_description\\": null,
+     \\"imported_description\\": \\"Lowe's Store\\",
      \\"income_month_flag\\": 0,
      \\"isChild\\": 0,
      \\"isParent\\": 0,
      \\"location\\": null,
      \\"notes\\": null,
@@ -71,10 +71,28 @@
      \\"starting_balance_flag\\": 0,
      \\"type\\": null,
    },
    Object {
      \\"acct\\": \\"one\\",
+     \\"amount\\": -2407,
+     \\"category\\": null,
+     \\"date\\": 20171016,
+     \\"description\\": \\"Transaction 32\\",
+     \\"error\\": null,
+     \\"financial_id\\": \\"753911ce-7b09-4cb3-8447-ac6eb74e727e\\",
+     \\"id\\": \\"testing-uuid-110\\",
+     \\"imported_description\\": \\"Transaction 32\\",
+     \\"income_month_flag\\": 0,
+     \\"isChild\\": 0,
+     \\"isParent\\": 0,
+     \\"location\\": null,
+     \\"notes\\": null,
+     \\"starting_balance_flag\\": 0,
+     \\"type\\": null,
+   },
+   Object {
+     \\"acct\\": \\"one\\",
      \\"amount\\": 9307,
      \\"category\\": 1,
      \\"date\\": 20171015,
      \\"description\\": \\"Starting Balance\\",
      \\"error\\": null,
@@ -148,13 +166,13 @@
      \\"amount\\": -2947,
      \\"category\\": null,
      \\"date\\": 20171017,
      \\"description\\": \\"macy\\",
      \\"error\\": null,
-     \\"financial_id\\": null,
+     \\"financial_id\\": \\"3591ad03-b705-42e0-945d-402a70371c49\\",
      \\"id\\": \\"three\\",
-     \\"imported_description\\": null,
+     \\"imported_description\\": \\"#001 fenn st Macy's 33333 EMX\\",
      \\"income_month_flag\\": 0,
      \\"isChild\\": 0,
      \\"isParent\\": 0,
      \\"location\\": null,
      \\"notes\\": null,
@@ -163,16 +181,16 @@
    },
    Object {
      \\"acct\\": \\"one\\",
      \\"amount\\": -2947,
      \\"category\\": null,
-     \\"date\\": 20171016,
+     \\"date\\": 20171017,
      \\"description\\": \\"papa johns\\",
      \\"error\\": null,
-     \\"financial_id\\": null,
+     \\"financial_id\\": \\"01a3a594-a381-49d1-bcf8-331a3c410900\\",
      \\"id\\": \\"two\\",
-     \\"imported_description\\": null,
+     \\"imported_description\\": \\"Papa John's\\",
      \\"income_month_flag\\": 0,
      \\"isChild\\": 0,
      \\"isParent\\": 0,
      \\"location\\": null,
      \\"notes\\": null,"
`;

This snapshot captures a ton of changes and thinking about manually testing all of them makes me want to cry in a corner. It updated the dates and other fields of the matched transactions, and added one that didn’t match. There are several other edges cases and I can capture them just as simply in additional tests.

Downsides

There’s only one real downside I can think of with snapshots: snapshots make it harder to read tests since you aren’t sure of the intent of the snapshot when looking at a test. expect(obj).toEqual({ foo: 3 }) certainly has more meaning than expect(obj).toMatchSnapshot().

This can be mitigated a few ways. I’ve seen extensions to Atom and VSCode that actually show the snapshot inline, which basically solves the problem. In Emacs I have a few shortcuts which allow me to jump to the snapshot. So while it’s a little annoying, it’s basically the same as expect(obj).toEqual(obj2) with obj2 existing outside the file, and tools can help make it easy to see them.

I personally don’t hit this as much as you’d think. Usually it’s pretty clear what it should be testing — in the tests above, the first snapshot should contain all the initial transactions. I don’t care what they actually are, but it shouldn’t change across tests.

However, there are times when it’s unclear what a snapshot is testing. In those cases, a simple comment goes a long way in documenting its purpose. For example, I’d document the diff that tests matched transactions like this:

differ.expectToMatchDiff(await getAllTransactions());

Another problem might be that snapshots are just too damn easy. This makes it easy to accidentally make snapshots too broad (contain way more data than is needed) which makes it a lot harder to read and deduce problems later on when things change. It requires a little experience to write good tests and shrink snapshots down to only what’s needed.

That’s It

Snapshots and diffing made it amazingly easy for me to build out transaction importing in Actual, with a full comprehensive test suite that makes me confident about change the code in the future. If you’re dealing with a lot of data (React components count as data, as they can be snapshotted) I’d highly recommend looking into snapshot testing. Jest has great support for them and a few other testing frameworks have added support for them.

Originally published at dev.actualbudget.com.