How we use DataFire in Production

While there are plenty of fun use cases for DataFire, we built it primarily for dealing with business logic. Our primary goal was to automate tasks that were taking too much time from core product development, such as managing users, tracking issues, and monitoring databases.

You can read more about our motivation for building DataFire here, but I wanted to share a few of the ways we’re already using it in production.

Extending GitHub Issues

One of the first DataFire Dataflows I set up was to help us manage issues in GitHub. GitHub’s issue tracking is simple and elegant, but it lacks a few features I’m used to in a bug tracker - in particular, we wanted to be able to add Priority, Severity, and Time Estimate fields. So I threw together a Dataflow that would:

• Grab all issues from the repository
• Create a row in a Google Spreadsheet for each new issue
• Update title/description/milestone/etc in the spreadsheet for any changed issues
• Remove any issues that had been closed

This Dataflow runs every minute, so now we have a spreadsheet that’s constantly up to date with the latest from GitHub. Add a few columns and some quick formulas, and now we can:

• Track the estimated number of hours to close each bug
• Track how many hours of work are left in a given milestone
• Quickly see whether or not we’re on schedule for a given milestone
• Craft contingency plans that drop low-severity and low-priority issues from a milestone

Additionally, I created a second Dataflow that syncs our GitHub issues to a Trello board, which allows us to track the progress of sales, marketing, and development efforts all in one place.

Engaging Users

Establishing a conversation with your earliest users and keeping them engaged is critical to getting a new service off the ground. Your first opportunity here is your welcome e-mail - you can reintroduce the product, and invite the user back to the site if they bailed after signing up. Too often, however, a signup flow goes like this

• I enter my email and password
• I see my shiny new account for the first time
• *bzzz* - my phone shows a new e-mail: “Welcome to our site!”
• I hit “archive” on the notification without bothering to look
• 5 minutes later I close the tab, never to hear from them again

This service has just missed a valuable opportunity to draw me back in. Why contact me when I’m already on your site?

Instead of blindly firing off a welcome email every time we see a sign up, we have a Dataflow which grabs the newest users from MongoDB. If the account is at least 30 minutes old, we’ll fire off the welcome e-mail; otherwise we hold back while the user continues to explore the site.

For our first 1000 users or so, this Dataflow will also send the founders an e-mail with a list of the newest signups. We’ll then reach out over the next couple days with a personal note thanking them for their interest and asking what we can do to help.

Down the road, we’ll also use Dataflows to fire off targeted, usage-based e-mails. For instance, if a user signs up but still hasn’t made a Dataflow after 24 hours, or if a user creates a Dataflow but never schedules it to run, we can automatically send a personalized message offering assistance.

Monitoring

One of the simplest use cases for DataFire is monitoring - when X changes, or Y hits a certain threshold, send me an e-mail or text.

The first monitor I set up was quite simple - I get an e-mail every morning with key statistics:

• Page Views and Sessions from Google Analytics
• Number of new users from MongoDB
• Number of new Dataflows from MongoDB
• Dataflow success rate for the last 24 hours

This keeps me informed on whats happening without having to visit several different sites or (worse) directly query MongoDB.

Monitoring can be more complex than a simple status update though. Early on we noticed that if one of our processes crashed or was restarted improperly, database entries could get into a bad state. While we put safeguards in place to ensure this wouldn’t happen in production, you can never be too safe - we have a Dataflow that queries the database every 15 minutes for entries that are in this bad state, and sends us an alert if it finds anything. Another Dataflow (only run on-demand) corrects them.

Future Work

We’re excited about the number of different ways DataFire is already saving us time and energy. We plan to continue moving sub-critical tasks from our core codebase into DataFire, and ideas for new uses continue to pop up. Stay tuned!

Are you using DataFire in production? Have ideas on how it could help? Let us know in the comments!