How Clara suggests times

This is part of a series of posts on How Clara’s human-in-the-loop system works behind the scenes.

Overview

One of Clara’s core competencies is negotiating a time with others on your behalf. A key aspect of this negotiation is the suggestion and confirmation of meeting times. Unlike a human assistant, Clara leverages any relevant calendar information throughout our service network, as well as all in-flight meeting activity in our system.

To do this, we’ve constructed a supervised system overseen by our workers that takes into account all known information to produce meeting times. This system currently generates the majority of times suggested across all Clara emails.

An overview of the flow of this system can be viewed in this diagram.

Supervised automated suggest times system.

The times Clara suggests are produced through the following sequence of steps:

1. Determine Candidates Window
2. Generate Time Candidates
3. Score and Rank Candidates
4. Human Review & Feedback

In the next few sections of this post, we’d like to walk through these steps and share how we formulate the times that we propose or confirm.

Determine Candidates Window

Let’s say a new email comes in from a customer with the following request:

Clara, schedule a meeting with Alice early next week.

Before selecting any specific times to suggest we need to determine the range of possibilities, or window in which to suggest.

The time candidates window defines the range of time that the meeting should occur within and is determined by context provided in the email thread. A time window can be a single occurrence or recurrence of times.

• “Clara, schedule a meeting between me and Alice this Friday”
• “Clara, find an afternoon for us to do happy hour next week”
• “Clara, find a good time for a board meeting in early January”

The text “early next week” maps to a candidate window of Monday and Tuesday.

In the degenerate case, a specific meeting time proposal by an individual may be one-to-one with the suggestion window. If no window is provided, by default Clara will automatically choose a one week window.

Generate Time Candidates

Now that we’ve limited the search space of time candidates to a smaller window, we must divide the window into candidates to consider and further explore.

As an example, “early next week” might be broken up into candidates of 30, 60, and 120 minutes, as depicted below:

Time candidates of 30, 60, and 120 minutes within the window “early next week.”

These candidates can be any length greater than the duration, allowing us to consider both meeting length time candidates and spans of a longer duration (ranges) that we could suggest or confirm.

For any given time window hundreds to thousands of time candidates are generated to be further considered as a meeting time.

Score and Rank Candidates

Once time candidates have been generated we must evaluate each candidate and determine the optimal candidates.

The candidates are scored and ranked to determine the times to suggest or confirm for a meeting. In our system today, a candidate c is scored as a weighted linear combination of features or “weak scoring” functions f over all attendees a on the meeting,

Feature functions are parameterized by any information we’ve annotated and collected, such as attendee preference, attendee availability (given to us), or customer calendar information. Some interesting feature functions in production today are

• Attendee local time — Given all known timezones for attendees, Clara can prioritize time candidates that occur during typical business hours. This allows Clara effectively handle meetings across multiple timezones.
• Outstanding suggested times — Clara has visibility into outstanding suggestions for attendees and customers across multiple meetings being scheduled. As suggested times accumulate, the probability of any given time becoming unavailable increases. We can bias against time candidates likely to become unavailable in the near future, increasing the efficiency of our proposed or selected times. This is a powerful network effect of having so many customers on our platform.
• Location — Given a customer’s locations throughout the day, Clara can bias towards time candidates that minimize the total amount of daily travel required. If a customer is meeting an individual at a coffee shop between 1 and 2pm, then 2pm is potentially a great candidate for the next coffee meeting at the same location.

Example weak scoring functions. (a) Attendee local time score is computed as a piecewise function where tau denotes start and end business hour boundaries and t is the start time of candidate c in timezone z of attendee a. In this example, the scores outside of business hours decrease linearly but they can easily drop off much faster (or be set to zero) if desired. (b) Outstanding suggested times score is computed as negatively proportional to the number of candidates that intersect any suggested ranges S_a for attendee a. (c) Location score is computed as negatively proportional to travel time, for example where M_a are all meetings for attendee a and l denotes the location for a time candidate or meeting, denoted by l_c and l_m, respectively.

These are just a few of upwards of 20 scoring functions currently in use today. Once we’ve generated a score for each candidate, we rank them and return the top scoring candidates to be considered as potential meeting candidates.

Because the weak scoring functions are easy to understand and interpret, it’s trivial for anyone of the team to propose and implement an extended behavior. Further, we can run experiments with new features against historical suggestions and incorporate our learnings back into the system.

Human Review & Feedback

If the highest ranking time candidates do not meet a predetermined threshold, we present them to a worker for further consideration with any helpful local context such as customer preference and calendar information. Here the times are presented in a simple, easy-to-use interface with two possible actions:

• Approve Time — The time is appropriate and meets the constraints of the meeting.
• Reject Time — The time does not make sense and violates the constraints of the meeting.

A simple interface for collecting binary “looks good” / “looks bad” feedback about a generated time.

The resulting worker feedback can be incorporated into future recommendations, for example, by training the scoring weights w. When a time is rejected, we can display another high ranking candidate for consideration or allow the worker to select their own time using a calendar-like interface.

The current interface collects positive or negative feedback on time candidates, but as the system evolves we are exploring alternative interfaces to collect feedback.

Measuring Success

The ultimate measure of success for an individual candidate is whether or not that time is confirmed by one of our workers. Any confirmed or rejected time candidate on its own can be used as feedback into how we determine weights for different features.

Feedback for a single time candidate is useful information, but it’s also important we measure performance at the system level. The most frequent action Clara takes is suggesting meeting candidates, so it has a big impact on the overall quality of the product.

Some of the higher level metrics we currently measure and evaluate are:

• Number of time proposals — How many different time candidates did we have to present before confirming a meeting time.
• Number of back-and-forths — How many emails needed to be exchanged before confirming a meeting time.
• Downstream complications — Did a confirmed candidate ultimately need to be moved or rescheduled and was that preventable.

As we change product behavior, modify features, and tune their corresponding weights, all of these metrics are tracked and monitored. Interestingly, Clara’s global view of performance means we can optimize our scheduling negotiation tactics unlike any individual human assistant.

Conclusion

In the early days of Clara, workers would manually select available times using a calendar-like interface. All times were individually selected by a worker. Algorithmically generating times allows us to drive product improvements through software and enables increased scalability of our workforce. Today, the majority of times you see in an email from Clara came from this system.

Clara is growing

We’re excited to grow our platform and team. Find out more about open roles here: https://jobs.lever.co/claralabs

Learn more about our CEO Maran Nelson here:
https://www.forbes.com/sites/clareoconnor/2017/04/18/clara-labs-wants-to-save-your-from-your-inbox-with-cyborg-assistants/

and read about our recent $7M series A announcement here:
https://techcrunch.com/2017/07/19/claraseriesa/