How to develop a serverless chatbot (for Hangouts Chat) — Process incoming chat messages

Visit http://www.mikenikles.com for my latest blog posts.

Introduction

This post is part of a series where you can learn how to develop, monitor (and later debug and profile) a serverless chatbot (for Hangouts Chat). Please refer to the other posts linked below for context.

• Intro & architecture
• Process incoming chat messages (this post)
• Send messages to users
• Find reminders & notify users
• Monitoring

This post focuses on the highlighted part of the architecture

Setup (PR #1)

NPM: First things first, let’s initialize an npm package in an empty git repository with npm init.

Lerna: We use independent mode and initialize the monorepo with npx lerna init --independent. Lerna is added to devDependencies at the root, let’s install it with npm install.

Set a reminder (Steps 1 to 6)

Step 1 & 2 — Set up Hangouts Chat API

Two endpoint types are supported, HTTP and Pub/Sub. We use Pub/Sub since it works with bot implementations behind a firewall, but also makes the architecture more robust. Two services are responsible for sending messages to a user, one when a reminder is set and another one when a reminder is due. By using Pub/Sub, we can abstract the sending part into a separate service (reminder-bot-sender).

In the Google API Console, enable the Hangouts Chat API by doing the following:

1. In the navigation, click APIs & Services > Dashboard.
2. In the Dashboard, click Enable APIs and Services.
3. Search for “Hangouts Chat API” and enable the API.

Once the API is enabled, click the Configuration tab. In the Configuration pane, set the following values and click Save:

• Bot name: Reminder Bot
• Avatar URL: https://goo.gl/yKKjbw
• Description: A bot that reminds you of things.
• Functionality: Select “Bot works in direct messages” and “Bot works in rooms”
• Connection settings: Select “Cloud Pub/Sub” and enter “tbd” as the topic ID. We will find that ID once we publish the reminder-bot-messages-in Pub/Sub topic.
• Permissions: Select an option that is acceptable for you.

We also have to grant permissions to the Hangouts Chat API to publish to Pub/Sub. Please follow the instructions in the documentation.

Step 3 & 4 — Create the reminder-bot-messages-in Pub/Sub topic and the reminder-bot-receiver Cloud Function (PR #2):

By configuring a Cloud Function whose trigger is a Pub/Sub topic, Google Cloud Platform creates the following automatically:

• A Pub/Sub topic with the name we specify.
• A Pub/Sub subscription which triggers the Cloud Function for each event published to the Pub/Sub topic.

In other words, all we need to do is create the reminder-bot-receiver Cloud Function, configure it properly and we’re all set to receive messages.

If you don’t have the Cloud Functions API enabled, do this first. In the navigation, click Cloud Functions and if you see “Enable API”, click it and wait until the API is enabled.

At the project root in your terminal, we can use Lerna to create a new service with npx lerna create reminder-bot-receiver. When prompted, use the default values. This creates the packages/reminder-bot-receiver directory and initiates it with some boilerplate code. Edit the packages/reminder-bot-receiver/lib/reminder-bot-receiver.js file and replace the existing code with the following:

// packages/reminder-bot-receiver/lib/reminder-bot-receiver.js
exports.reminderBotReceiver = (event, context) => {
 const pubSubMessage = event;
 const name = pubSubMessage.data ?
  Buffer.from(pubSubMessage.data, 'base64').toString() :
  'World';
console.log(`Hello, ${name}!`);
};

Next, let’s deploy this function and configure it to be triggered via Pub/Sub events. In the service’s package.json, add the following script:

// packages/reminder-bot-receiver/package.json
{
  ...
  "scripts": {
    "deploy": "gcloud functions deploy reminderBotReceiver --runtime nodejs8 --trigger-topic reminder-bot-messages-in",
    ...
  }
}

This is where the magic happens. This commands deploys the function, creates a Pub/Sub topic named reminder-bot-messages-in and configures a subscription on that topic to trigger the reminder-bot-receiver Cloud Function. It’s really that simple.

For simplicity, we can add the following script to the root-level package.json to deploy the reminder-bot-receiver service with a single command:

// package.json
{
  ...
  "scripts": {
    "deploy:reminder-bot-receiver": "lerna run deploy --stream --scope=\"reminder-bot-receiver\"",
    ...
  }
}

Lastly, npm run deploy:reminder-bot-receiverdeploys the Cloud Function. Once completed, let’s make sure everything worked as expected.

Validation: We want to make sure the Cloud Function, a topic and a subscription are available. If any of the following steps fail, please double check you followed the above instructions correctly. If you’re stuck, please leave a comment and I can help troubleshoot the issue.

1. In the navigation, click Cloud Functions.
2. Click on reminderBotReceiver name.
3. Select the Testing tab, located below the function name and version dropdown.
4. Click the Test the function button and wait for the logs to show up below. Make sure you can see “Hello, world!”.
5. In the navigation, click Pub/Sub > Subscriptions.
6. Ensure a subscription is available and its name contains “reminder-bot-messages-in”.
7. On the left-hand side, click Topics.
8. Ensure a topic is available and its name contains “reminder-bot-messages-in”.
9. Copy the full name of the topic, e.g. projects/YOUR-PROJECT-NAME/topics/reminder-bot-messages-in.

Update the Hangouts Chat API topic: In the navigation, click on APIs & Services > Dashboard. Click on the Hangouts Chat API link in the table. Once loaded, click Configuration and update the Cloud Pub/Sub Topic ID in the Connection settings section. Replace tbd with the topic name copied in step 9 above. Click Save at the bottom.

At this point, it’s time to test the bot. Navigate to https://chat.google.com, click “Find people, rooms, bots” at the top left and type “Reminder Bot”. Select the one where it says “A bot that reminds you of things”.

Select your bot, the one with the green avatar

Send a message to your bot. It won’t respond yet, but we can check the logs to validate the message was received by the reminder-bot-receiver Cloud Function. You should see logs similar to the following (Note: centralised logging is available in the GCP navigation under Logging):

ADDED_TO_SPACE and MESSAGE are good signs that the bot is correctly configured

Step 5 — Persist the reminder data in Cloud Firestore

Next up, we need to process the incoming event. This requires us to do the following in the reminder-bot-receiver Cloud Function:

• Determine the type of event the Hangouts Chat API sends. We will care about ADDED_TO_SPACE and MESSAGE.
• For type MESSAGE, we want to make sure the user sent a proper reminder. We do this by validating the message contains a date and a subject.
• Convert the date (e.g. “tomorrow at 5pm”) to a proper timestamp.
• Deal with incomplete or invalid messages such as when they don’t contain a date.

Lastly, we will persist the reminder data in Firestore, where the reminder-bot-checkerfunction will look for reminders that are due to be sent to users — more on that in steps 7 to 10).

Step 5 — Foundation to determine the type of event (PR #3)

Let’s start by setting up a foundation for the event handlers by creating files that match the following directory structure:

./packages/reminder-bot-receiver/
└── lib
    ├── event-handlers // New directory and files
    │   ├── added-to-space.js
    │   ├── index.js
    │   └── message.js
    └── reminder-bot-receiver.js // Existing file

For now, let’s add the following code to the two handlers, event-handlers/added-to-space.js and event-handlers/message.js:

// packages/reminder-bot-receiver/lib/event-handlers/added-to-space.js
module.exports = () => {
    console.log('TODO: Process event type ADDED_TO_SPACE')
}
// packages/reminder-bot-receiver/lib/event-handlers/message.js
module.exports = () => {
    console.log('TODO: Process event type MESSAGE')
}

To tie it all together, the event-handlers/index.js file should look like this:

// packages/reminder-bot-receiver/lib/event-handlers/index.js
const addedToSpace = require('./added-to-space')
const message = require('./message')
// Keys represent event types.
// See https://developers.google.com/hangouts/chat/reference/message-formats/events#event_types
module.exports = {
    ADDED_TO_SPACE: addedToSpace,
    MESSAGE: message
}

Step 5 — Call event handlers (PR #4)

With the event handler foundation in place, it’s time to call the appropriate event handler. The event type is part of the payload we receive from the Pub/Sub topic. Let’s parse the event, look for the event type and call the correct event handler in the reminder-bot-receiver.jsfunction:

// packages/reminder-bot-receiver/lib/reminder-bot-receiver.js
const EVENT_HANDLERS = require('./event-handlers')
exports.reminderBotReceiver = async (event, context) => {
  console.log(`Input received:
    Event: ${JSON.stringify(event)}
    Context: ${JSON.stringify(context)}`)
// See https://developers.google.com/hangouts/chat/reference/message-formats/events#event_fields
  const chatEventBody = JSON.parse(Buffer.from(event.data, 'base64').toString())
  console.log('Chat event body:', chatEventBody)
try {
    EVENT_HANDLERS[chatEventBody.type](chatEventBody)
  } catch (error) {
    console.error(new Error(`Couldn't process event due to: ${error}`))
  }
};

Step 5 — Process MESSAGE event — parse date (PR #5)

Now that the correct handler gets called based on the event type, the next step is to parse a user’s reminder to determine the reminder’s date. For example, “Go to sleep in five minutes” needs to be turned into a date that takes the current time and adds 5 minutes.
The good news is, chrono-node is a NPM package that does exactly that. So let’s use it:

// packages/reminder-bot-receiver/lib/event-handlers/message.js
const chrono = require('chrono-node')
module.exports = (chatEventBody) => {
  const userInput = chatEventBody.message.text
  console.log(`Processing event type MESSAGE for message text: ${userInput}`)
const reminderDate = chrono.parseDate(userInput);
  console.log(`Found date: ${reminderDate}`)
}

Validation: Deploy the function by running npm run deploy:reminder-bot-receiver from the project’s root directory. Once deployed (check the logs), open Hangouts Chat and send “Go to sleep in five minutes” to the Reminder Bot. In the logs, you will see the above console.log statements. Verify the Found date: xyz log statement shows the correct date, 5 minutes from the time you sent the message to the bot.

Step 5 — Process MESSAGE event — parse subject (PR #6)

Now that we know the reminder’s date, let’s figure out the subject. In the “Go to sleep in five minutes” example, the subject is “Go to sleep”.

Warning in advance, this is not the prettiest solution and I’m happy to approve pull requests that clean this up. For brevity, here’s the function that takes care of finding the subject. Please refer to the pull request on how to use it.

// packages/reminder-bot-receiver/lib/event-handlers/message.js
const determineReminderSubject = (userInput) => {
  // Object reference at https://github.com/wanasit/chrono#usage
  const parsedChrono = chrono.parse(userInput);
  const dateTimeText = parsedChrono[0].text;
// Remove the date / time part of the user's message
  let subject = userInput.replace(new RegExp(dateTimeText, 'g'), '');
// Remove "@Reminder Bot" if the bot is used in a room
  subject = subject.replace(new RegExp('@Reminder Bot', 'g'), '');
return subject.trim();
}

Step 5 — Process MESSAGE event — persist reminder in Firestore (PR #7)

Fantastic, we’re in good shape. We have a reminder date and its subject. In order to persist it, we need some more information, such as who sent it, in which room, etc. The good news: This information is provided in the event the reminder-bot-receiver function receives from the Pub/Sub event.

Let’s first create a db.js file. It provides helper functions to deal with Firestore.

// packages/reminder-bot-receiver/lib/db.js
const Firestore = require('@google-cloud/firestore');
const db = new Firestore({
  projectId: process.env.GCP_PROJECT,
  timestampsInSnapshots: true
})
const persistReminder = async (spaceName, threadName, userInput, reminderDate, reminderSubject, userName) => {
  try {
    await db.collection('reminders').add({
      parent: spaceName,
      raw_msg: userInput,
      remind_at: reminderDate,
      status: 'new',
      subject: reminderSubject,
      thread: threadName,
      user: userName,
    })
  } catch (error) {
    console.error(new Error(`Reminder could not be persisted due to: ${error}`))
  }
}
module.exports = {
  persistReminder
}

To make this work, we need to enable Cloud Firestore. Open the Google Cloud Console, enable Cloud Firestore by doing the following:

1. In the navigation, click Firestore.
2. On the Get started screen, three options are presented. Click SELECT on the first one, “Cloud Firestore in Native mode”.
3. On the next screen, choose a suitable database location. Keep this close to where you deploy the Cloud Functions and where the majority of your users are located. The best option based on this tutorial is us-central (United States). Then click CREATE DATABASE.

Lastly, the following changes are needed to persist the reminder (refer to the pull request for all changes):

// packages/reminder-bot-receiver/lib/event-handlers/message.js
module.exports = async (chatEventBody) => {
  ...
  console.log(`Found subject: ${reminderSubject}`)
console.log('Persisting reminder...')
  await persistReminder(chatEventBody.message.space.name,
    chatEventBody.message.thread.name,
    userInput,
    reminderDate,
    reminderSubject,
    chatEventBody.user.name)
  console.log('Reminder successfully persisted.')
}

Validation: We’re making great progress. You can now send a message to the Reminder Bot, it will get processed and persisted in Firestore. Test it with “Go to sleep in five minutes”, then make sure the information is persisted in Firestore:

1. Open the Google Cloud Console.
2. In the navigation, click Firestore > Data.
3. Click on the reminders collection, then on the only document that is listed.
4. The document fields display on the right. Make sure its values look ok.

Step 5 — Process ADDED_TO_SPACE event (PR #8)

When the bot is added to a 1:1 message or a chat room, the Hangouts Chat API sends a ADDED_TO_SPACE event. There is nothing to be persisted in Firestore, so the event handler is a simple no-op.

// packages/reminder-bot-receiver/lib/event-handlers/added-to-space.js
module.exports = () => Promise.resolve()

Step 5 — Process REMOVED_FROM_SPACE event (PR #9)

This too is a no-op event handler and follows the same pattern as the ADDED_TO_SPACE handler above. It’s here for the sake of completeness and to avoid errors when the Reminder Bot is removed from a room.

Step 6 — Prepare foundation to send a response (PR #10)

To get started, let’s first create a pubsub.js file where we handle the publishing of messages to Cloud Pub/Sub:

// packages/reminder-bot-receiver/lib/pubsub.js
const { PubSub } = require('@google-cloud/pubsub')
const pubsub = new PubSub()
const sendMessage = async (message) => {
  try {
    const messageId = await pubsub
      .topic('reminder-bot-messages-out')
      .publisher()
      .publish(Buffer.from(JSON.stringify(message)));
    console.log(`Message ${messageId} published.`);
  } catch (error) {
    console.error(new Error(`Message could not be published due to: ${error}`))
  }
}
module.exports = {
  sendMessage
}

Note: Running this code currently fails because the reminder-bot-messages-out Pub/Sub topic does not exist yet. You can either manually create it, or wait until we create the reminder-bot-sender Cloud Function. Once we deploy this function, the corresponding topic will be created automatically.

Step 6 — Develop the response handlers foundation (PR #11)

The pattern here is the same as for the event handlers above. The response sent back to Hangouts Chat depends on the event type. The pull request lists the changes for this step in detail.

Step 6 — Send a response for ADDED_TO_SPACE events (PR #12)

When a user adds the Reminder Bot to Hangouts Chat, it’s a good idea to reply with a warm welcome and a quick introduction on how to use the bot. Thanks to the foundation already in place, this is very little effort to implement.

// packages/reminder-bot-receiver/lib/response-handlers/added-to-space.js
const { sendMessage } = require('../pubsub')
const helpMessage = `Hey! I can remind you of stuff. Tell me what to remember and when to remind you and I will send you a direct message at the right time. For example, "@Reminder Bot buy milk tomorrow at 6pm".`
module.exports = async (chatEventBody) => {
  console.log('Processing response for event type ADDED_TO_SPACE')
await sendMessage({
    spaceName: chatEventBody.space.name,
    message: `Thanks for adding me, ${chatEventBody.user.displayName}. ${helpMessage}`
  })
}

Step 6 — Send a response for MESSAGE events (PR #13)

For each reminder a user sets, the system responds with a confirmation message and the reminder’s formatted date. This ensures the system understood the user’s intent.

// packages/reminder-bot-receiver/lib/response-handlers/message.js
const chrono = require('chrono-node')
const { sendMessage } = require('../pubsub')
const formatReminderDate = (userInput) => chrono.parseDate(userInput).toLocaleString();
module.exports = async (chatEventBody) => {
  console.log('Processing response for event type MESSAGE')
await sendMessage({
    spaceName: chatEventBody.space.name,
    threadName: chatEventBody.message.thread.name,
    message: `Got it. I will remind you on ${formatReminderDate(chatEventBody.message.text)}`
  })
}

Step 6 — Send a response for REMOVED_FROM_SPACE events (PR #14)

Upon removing a bot from Hangouts Chat, we can’t respond because the bot has already been removed. See the pull request on how to implement a no-op for this use case.

Summary

At this point, we have a Hangouts Chat bot users can use to persist reminders. It’s basic, doesn’t have much error handling and certainly has no tests — bad! One important part of any chatbot is to respond to users. We’re actually quite close since the reminder-bot-receiver Cloud Function already publishes messages to the reminder-bot-messages-out Pub/Sub topic. Remember, that topic doesn’t exist yet, but that’s exactly what we’re going to do next.

👏 ❤️

Next Steps

The third part of this blog series is where you’ll learn how to send a message back to Hangouts Chat. I highly recommend to have a look at it. Once you read it, you’re in a great position to develop your own chatbot.

Alternatively, head over to the monitoring post to learn more about Stackdriver error reporting, custom log metrics, etc.