auToMaTioN

Monitoring and Logging at Quartet Health

Systems fail. From a subtle, periodic route handler refresh that cripples a web server to a syntax error that breaks a login form, failure is inherent to engineering. One natural way we deal with failure is to mitigate it through various technical and human means.

Increasing code coverage and backing up data are some examples of a technical approach to defending against software bugs and data loss. Transferring knowledge about a system and instituting policies and conventions involve more human components. At the end of the day, these measures do not prevent failure from occurring. Rather, they make it more predictable and simpler to reason about.

Here at Quartet, our applications are used by physicians and therapists to manage referrals and keep in touch with patients. Our users depend on these tools, so when a failure occurs we need to ensure the following:

1. Fall back to the best experience we can provide
2. Find out what went wrong and fix the underlying problem efficiently
3. Preserve the sensitive nature of protected health information (PHI)

This post will focus on how we tackle 2 and 3 for client-side failures.

We chose React to build our front-end web applications for its contained solution to the view. This enabled us to compose interfaces from smaller, simpler components.

For managing state and data flow, Redux made a lot of sense from a safety and predictability point of view. If data is isolated, and we can only change it by describing event-driven actions, we have a clear model of the runtime:

Actions describe state transitions, making data changes more predictable

From the diagram above, you can inspect the entire state of the application at any given point in time. In fact, that’s what React components do: deterministically render consistent UI from chunks of immutable data.

If we keep track of all actions going through the system, we can identify the interactions and events that took place before a failure occurred, simplifying the debugging process. This is where Automaton comes in.

Automaton is lightweight Redux middleware that intercepts actions and stores them in an immutable list:

class Automaton {
  constructor() {
    this.history = Immutable.List();
  }
  reduxMiddleware = store => next => action => {
    let filteredActionObject;
    if (!action._filtered) {
      filteredActionObject = {
        type: action.type
      };
    } else {
      filteredActionObject = action._filtered;
    }
    this.history = this.history.push(filteredActionObject);
    return next(action);
  }
}

Whenever an action gets dispatched, Automaton saves it before forwarding it down the pipe. When an error is thrown, we automatically append the action history for additional context:

log(data) {
  return request
    .post('/log')
    .send({
      ...data,
      actionHistory: this.history.toJS()
    })
    .end();
}

You may have noticed the middleware function attempts to grab the _filtered prop on the action object. By default, Automaton will only save the action type, ensuring that payloads carrying PHI and other sensitive information will not be sent over the wire.

There are cases when we want to save data associated with an action. Knowing that a number was used in an email address can help narrow down possible options when debugging errors during form submit, for example.

Obfuscator is a simple utility that Automaton uses to clean values, preserving structural properties like length, case, and type while replacing values with random ones.

Here’s an example of dispatching a notification action where the message property is cleaned:

import { cleanValue } from './utils/obfuscator';
import { createActionWithFilter } from './utils/create_action';
dispatch(createActionWithFilter({
  type: 'ADD_NOTIFICATION',
  message
}, (actionObject) => {
  return {
    ...actionObject,
    message: cleanValue(actionObject.message)
  };
}));

createActionWithFilter is a function that takes an action object, a filter function, and returns an action object with _filtered as a property:

createActionWithFilter = (actionObject, filter) => {
  return {
    ...actionObject,
    _filtered: filter(actionObject)
  };
}

Automaton owes its simplicity to the extensibility of Redux. We plan to open source the middleware and Obfuscator soon, but in case you want to get started earlier, the snippets in this post should provide enough context to recreate the functionality in your own application.

Primum Non Nocere is a blog we’re starting to share our experience with building modern health software for a rapidly changing ecosystem. The phrase is Latin for first, do no harm. While we may embrace bleeding-edge technologies and conventions to build out our applications, our primary focus remains on improving people’s lives. First, do no harm. Second, build awesome stuff!

If you enjoyed this post and are interested in finding out more about what we enjoy working on, let’s get in touch!