Scheduling recurring events. Lazy Jar — Part #2

This is part two of a series of article on Lazy Jar. A slack app for scheduling remote stand-ups and tracking participation in Node.js. Fork the to start experimenting. If you’re new to this project? Start with the .

How do we schedule notifications?

Our scheduler is a wrapper around : a library for scheduling recurrent events.

Our wrapper is responsible for

  1. Translating the time and frequency of an event to a Cron String
  2. Creating a Job that when fired sends a Slack message to the active members of an event (i.e. a stand-up)

Why Cron Strings?

Node Schedule supports RecurrenceRule — JSON object with fields such as year, month, and day — as well as Cron String to specify the recurrences of an event. However, with RecurrenceRule you can only specify one day in a week to fire an event — you can pick Monday but not everyday or weekdays. Because of this limitation, we had no choice but to convert the time and frequency information of an event to a Cron String.

We use the following table to map from frequencies specified by the user to the dayOfWeek segment of a Cron String. The result of the mapping is joined to the time of day information to create a complete Cron String with a formula similar to0 ${mm} ${hh} * * ${dayOfWeek}.

Frequency to Cron String map

Job Factory

A Job is a function that when called with the fireDate of an event, sends notifications to the active members of the event.

A Job depends on several pieces of information and helper functions that are available at different points in the scheduler. Hence we divided the dependencies into three groups,

  1. Helper functions that depend on the information that will eventually become available
  2. Information that is known when we schedule an event and
  3. Information that is only known when the Job is fired

Job Factory is a module that depends on async functions that interact with the database or slack such as getEvent, getSecret, and notifyUsers. These functions take in team_id and event_id as arguments and return a promise.

When we call Job Factory with its dependencies, it returns a function that takes as arguments team_id and event_id — the remaining dependencies for creating a Job — and returns a Job. We get access to team_id, and event_id when we want to schedule an event in the Scheduler (we will cover this soon).

fireDate is the only piece of information that we don’t have access to until Node Schedule fires the Job. Thankfully Node Schedule passes in the fireDate when it invokes a Job.

Job Module at High Level

Job delegates everything to a helper function that gathers all the information required by the notifyUsers to notify the active members of the team.

notifyActiveMembersdoes the following.

  1. Retrieves the bot access token, event name, and URL
  2. Finds the active members of the team based on their ignore flag and skip-until-date
  3. Serializes the fire date of the event
notify active members

Note that we catch notifyActiveMembers and log an error if anything goes wrong. This is important since otherwise the error will be hidden from us.

Scheduler offers an inerface with three functions, add, cancel, and move(reschedule). They all take a list of events as input. move uses add and cancel internally andcancel calculates an identifier for each event and delegates to rest to the Node Schedule.

add creates a unique identifier based on the team_id and event_id, maps the time and frequency to a Cron string and creates a Job function to be fired with every invocation of the event. The rest happens in Node Schedule.

Scheduler Add a new event

How does Node Schedule work?

It only uses a single timer at any given time (rather than reevaluating upcoming jobs every second/minute).

Before reading any further, I suggest you think about the problem and try coming up with an answer on your own. This helps you think about the challenges and design decisions involved in creating a scheduler.
How would you implement a scheduler to run an arbitrary number of jobs with different recurrence rules, using only a single timer?

Let’s think about this with an example. We have two jobs, A and B. Job A runs every hour starting at 12 am, and job B runs every hour starting at 12:30 am. It is currently 12 am and job A is firing. Let’s define the next job to be the first job that should execute after the current time. In our example, this is Job B. When job A finishes executing we should create a new timer to fire at 12:30 am when job B must run next.

How do we keep track of which job to fire next? My first thoughts were to use a min-heap and store the jobs with their next execution time as their key. I could then use extract-min operation to find the next job that should run and create a timer to execute it. When the timer fires…

  1. I can calculate the next time we should execute the job that was just fired (new key) and add the job back to the min-heap with this new key
  2. Next, I can use extract-min to find the first job that should run next and schedule it.

Now let’s dive into the codebase and see how Node Schedule is actually implemented.

Diving into the codebase

All the code snippets below are taken from the and have been modified and commented for brevity.

Let’s walk through what happens when you schedule a simple job like this one. Here we are logging a string every hour at minute 42.

an example from Node Schedule’s README

To understand how Node Schedule works we need to first define Job and Invocation. The two primary objects involved in scheduling.

Job is an recurrent task with many helper functions for scheduling, moving, canceling … itself. We will cover some of these functions in this walk through.

An Invocation is a Job that is scheduled to run at a point in time. An Invocation holds onto the Job and a Job holds onto its pending invocations. There is also a global invocations array that holds onto the Invocations of every Job. The pendingInvocations is used in methods likecancel, and reschedule. Both invocations and pendingInvocations are sorted arrays.

In our example, we have a Job that runs every hour at minute 42. The current time is 1 am. So the pendingInvocations might be holding Invocations with fireDate 1:42 am, 2:42 am …

Since we only have one job in our example the globalinvocations list (the list holding onto all the invocations of all the jobs) will have the same members as the job’s pendingInvocations.

Job and Invocation

Calling scheduleJob creates a new Job object. It then calls the schedule method on the job object, passing it the cron string that defines the recurrence rules for the job. It returns the job if it can successfully schedule it.

scheduleJob

When a Job is created, pendingInvocation is initialized to an empty array. Then schedule is called. schedule supports different forms of specifications such as Date, RecurrenceRule, and Cron String. Because of this schedule is about 100 lines of code with lots of conditional statements.

In our example, schedule delegates parsing the cron string to thecronParser and then calls scheduleNextRecurrence with the parsed expression.

Cron Parser as its name suggests parses a cron string and returns an object with a next method to get the date of the next invocation of a specification. In our example calling next returns 1:42 am.

schedule

scheduleNextRecurrence gets the next invocation date after prevDate (current system time) to create an Invocation. It then calls scheduleInvocation. scheduleInvocation only adds this new invocation to the global invocations list and delegates everything else toprepareNextInvocation.

scheduleNextRecurrence

preparenextInvocation is where scheduling happens. It is the only place where a timer is created. Here is where we can see what happens when a job is executed.

currentInvocation is the earliest invocation in the global invocations list.

We first check if an invocation that is earlier than the currentInvocation has been added to the global invocations list. If so we need to clear the timer for executing the currentInvocation, update it to the first invocation in the invocations list (remember the list is sorted by time) and create a new timer to execute this new earliest invocation. We pass in a anonymous function to the timer that when the timer expires and the function is invoked it

  1. removes the first invocation from the invocations list
  2. invokes the callback if you specified any when creating the job
  3. maybe reschedule the next recurrence of the job
  4. and finally invoke the job itself

This is a recursive function that calls itself when the timer expires later in the future to create a new timer for the next invocation accross all the jobs.

In our example, there is nocurrentInvocation so we use the newly added invocation to the global invocations list as the currentInvocation. We then create a timer to execute the job at the invocation’s fireDate.

When that timer expires and the anonymous function is called, we reschedule the next invocation of the job and call prepareNextInvocation to create a timer for it.

prepareNextInvocation

Next

The scheduler logs when a user is notified. We also log when a user clicks on a link to visit the stand up marking them as participated. Next, we will use these logs to provide statistics on the participation of the members in each event.

.

Artris Code

This publication was an experiment in distributed teams and writing

Thanks to Grazietta Hof.

Alireza Alidousti

Written by

Working on a CubeSat. Student @ SFU. Previously intern @ Shopify, Visier, Menrva, and Samsung.

Artris Code

This publication was an experiment in distributed teams and writing