Creating a debounced email queue using FaunaDB and GraphQL

In my application, managers can communicate with their clients via messages, and then clients can reply with approve/disapprove. When a client responds, I need to send their response as an email to their manager.

The problem is that the Yes/No response can be easily changed. Not only that, but it feels great to click back and forth on those buttons. My goal is to prevent a bevvy of approve/disapprove emails from inundating managers’ inboxes.

I decided that I need to debounce client responses. I only want to send a response-email to a manager if a client hasn’t changed their response for five or more minutes.

My application is using as its back-end. Fauna is wonderful due to its global consistency and flexibility. I am using Fauna’s native GraphQL layer to communicate with the database.

One approach would be to check if an email had already been queued, and overwrite it if a newer, later response appeared before the email was sent. Due to Fauna’s global transactional consistency, this would be possible. The only downside is that the client would have to handle “Concurrent modification” errors if the same email was being updated within 200ms of itself. This could be avoided several different ways, but the easiest would probably be to debounce the client sending the emails.

I decided on a different approach to sidestep the concurrency issue. I would just queue all requests to send an approval-response email, and then worry about removing ‘duplicates’ at a later step. The email-sending would be broken down into two steps: 1) Query for the latest version of each response and 2) Mark all involved responses as sent. I would setup a scheduled function to run every 5 minutes to process the queue.

The Setup

First I had to define the problem more explicitly. I would queue up all responses before they were sent. Every response would have an key. If a later response with the same key existed, it would be sent instead of any earlier ones, which would end up being ignored. And I only wanted response whose most recent version was older than 5 minutes.

I created an EmailQueueEntry collection to store my email queue. I need to store an key to group entries together, a createdAt both to find the most recent entry and to determine if it was older than 5 minutes, a variablesPayload so that I knew what to put in my emails, and a sent flag so that I could query for entries that were unsent.

Here is my EmailQueueEntry GraphQL type:

The next step is to create an index so that I can query for unsent entries. In Fauna, indexes are crucial for being able to sort and query data stored in collections:

This was created using the node-js driver which is why there is a ‘q.’ in front of the Fauna function names. The source tells Fauna which collection to index. The terms are what properties from a collection will be used to search the index. The values determine what will be returned from an index search (called a match in Fauna), and how those values are sorted.

In this case, the entries in the collection would be grouped into ‘sent’ and ‘unsent’ sets (I only need the unsent set). And each set would be sorted first by key, then createdAt, and finally ref.

I store the reference to the original document so that I could query for and return the referenced documents. Fauna’s GraphQL layer expects documents to be returned when a resolver is defined as returning a GraphQL type. Also, I could use those references to update the documents’ sent property.

Next, I defined resolvers in my GraphQL schema for the entry-query and mark-as-sent mutations, and their respective UDFs (User Defined Function).

The timeKey argument would allow me to ensure that the lag between getting the entries and then marking them as sent wouldn’t cause me to accidentally lose track of entries created during the window between the two operations. FaunaDB has robust temporal querying capabilities which come in handy at times like these.

The Query: GetQueuedEmailEntries

For both getting the entries and marking them as sent, first I’d have to get the most recent message for a given key that was queued more than five minutes prior. I broke this requirement down into four main steps:

  1. Get all unsent entries
  2. Run a reduce operation on the entries, saving them to an object whose keys were the entries’ keys. I knew that since the index was saving entries by key and creation-date in ascending order, that as the same key was overwritten on the object, I would be left with only the most recent ones at the end of the reduce operation.
  3. Filter out entries that were queued less than five minutes ago.
  4. Gather up the entry data and return it.

Here is my getQueuedEmailEntries UDF written using the node-js driver:

First, I get all unsent entries using Fauna’s match function. By wrapping the function in Paginate, I can use those temporal capabilities that I mentioned earlier to only get results that exist at a specific point in time:

Next I use the results of the match in a reduce operation. The Lambda is an anonymous function that the Reduce uses on each element in the page returned by the paginated match function. I start with an empty object, which I updated each time the lambda is called. Each matched email entry is really an array with this structure:

[key, createdAt, ref]

This structure is due to how I defined the values property of the index. So I extract those values from the array (Using the Select function), and store them in local variables using a Let function. Ultimately I want to end up with an object that looks like:

{
"key1": {key: "key1", createdAt: Time, ref: Ref},
"key2": {key: "key2", createdAt: Time, ref: Ref},...
}

In this object, we will have all the keys of entries that should be sent, and the most recent values of those entries.

The problem is that there is no direct way to create an object with a dynamic key in Fauna. Instead, you can create such objects by first creating an array of key-value arrays (called finalObj below), and converting that into an object (which is done with the statement q.ToObject(q.Var(‘finalObj’) below). Lastly, I merge that newly created object with the reducer’s accumulator object, ordering the arguments so that my newly created key entry will overwrite any existing entry with the same key (and thus ensuring that only the most recent key for a given message exists in the final reducer result):

Next, I want extract the value objects from the result of the reduce function. Since my data is still typed as a Page, I have to first get the object produced by the reduce function out of the page so I can convert it to an array and map over its contents. Then I transform that array using Map, grabbing the value portion of each key-value pair and throwing away the key, since it exists as part of the value object anyway. Now I will have an array that looks like this:

[
{key: "key1", createdAt: Time, ref: Ref},
{key: "key2", createdAt: Time, ref: Ref},
...
]

Finally, I perform the time-based filter, remove entries whose createdAt is less than five minutes old when compared to the passed-in timeKey stored in the ts variable. This filter makes use of the GTE function (Greater Than or Equal), and TimeDiff (calculate the difference between two Time objects, and return the result in the specified units) functions:

And finally, I get the documents to return to the GraphQL layer by making use of the ref which is a reference to the stored EmailQueueEntry document, and the Get function which returns a ref’s referenced document:

The mutation

The mutation uses the same filter functions above, but then uses the results in a different way. After the filtered results, we have all the entry keys that should have been sent, and the most recent values of those messages. From there we have three basic steps to do:

  1. Extract the keys from that object (we only need the set of unique keys from the filtered results, we will throw away the values for now).
  2. Get all unsent entries again and filter them against the extracted keys. This way we can marks all entries which had a entry that was sent, even the ones that were ‘overwritten’ by newer entries.
  3. Actually mark them as sent.

Here are the parts:

First we start from the same getFilteredEntries function that the query used. We then transform those results so we have only the keys in an array:

Next we get the original matched set of unsent messages. This is keyed to the same time-entry as the original query, so that later queued entries won’t be accidentally marked as sent.

We then filter this set of queued entries by checking each one if its key exists in the array of keys that we know have already been sent. We do this by mapping over the array of sent keys, and using Equals against the key we are filtering on. Then we use Any on the resulting array. If any of the entries in the mapped array are true, then the Any function will return true, which means that the entry will be included in the filtered results.

Finally we extract the ref from the filtered results so that we can update the EmailQueueEntry document’s sent property to true.

Fauna Shortcomings

I’m pleased with how this system worked out. But the resulting code is more complicated than it needs to be due to the lack of some basic FQL capabilities.

Five FQL shortcomings that increased the complexity of the code:

  1. FQL has a Distinct function, but it isn’t flexible. I can’t ask for all the distinct keys from an array of email entries.
  2. Fauna has unique indexes. But rather than passively working to prevent duplicates, this index throws an error if you go to add one. This means I would be prevented from adding newer versions of entries.
  3. Document ids in Fauna’s collections can be arbitrary… as long as they are numerical. If I just wanted to overwrite documents by keys, then I would have to create a mapping scheme to map my alphanumeric keys to numbers and back again.
  4. The inability to directly create objects with dynamic keys, and to map over object properties meant lots of boilerplate to convert from objects to arrays and back.
  5. The Fauna collection types can be cumbersome. Having to convert Sets to Pages, and then have a Page with a single result come out from a Reduce function which then needs to be extracted to an array also adds unnecessary complexity.

Conclusion

Despite the shortcomings, it wasn’t too difficult to create a robust debounce queue for sending emails. Fauna’s temporal capabilities also made the queue more robust by preventing race-condition edge cases.

I hope that I showed how a complex system can be developed using Fauna. Such a system is scalable due to Fauna’s nature. And the code is flexible and maintainable due to its functional roots.

A fullstack developer that enjoys working with the latest serverless technologies and web frameworks to enable small teams to develop huge systems.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store