Chatbase: Analytics for chatbot 🤖️

When it comes to user behavior understanding in website, Google Analytics (GA) comes to mind. GA offers abundant kinds of tools for analysts to get insight into their viewers based on page transitions. Website shows different pages to handle variant users’ requests, hence transitions between pages become an effective way to understand your users on websites. However, there is no page transition in chatbot, and how to track the chatbot’s users becomes a problem.

In my opinion, there are two main facets that we want to know about our bot users.

1. Metrics measuring user engagement like Daily Active Users (DAU), Monthly Active Users (MAU), DAU/MAU Ratio, User Retention across different periods.
2. The messages sent by users but our bot can’t handle properly. Ideally, we want our bot to understand any user intents by means of natural language understanding methods, but this is hard to achieve without sufficient text data. So what we need is to collect the text of the unhandled messages to help us optimize our bot.

Chatbase targets the above two concerns and provides colorful visual reports which make user experience researchers easier to optimize their products.

In this article, I will briefly introduce some essential terms in order to fully utilize Chatbase as a analysis tool for chatbot.

Message Types

• User Message
• Agent(bot) Message

Intent & Sub-Intent

Intents define the purpose of a user message relative to the context of one’s bot. An intent is the essence of a user message and can be used to label a set of user messages with the same meaning.

It’s important to label each user message with correct and suitable intent for the reason that we could understanding how users are interacting with your bot, whether or not users are following optimal dialog paths by your design.

With intent being set, Chatbase will calculate metrics such as intent frequency and user exit rate for analysts/UX designer to better optimize your bot’s user experience.

Furthermore, Chatbase supports sub-intents in case some intents need multiple steps to be fulfilled. For example, if your bot is designed for hotel booking, then it’s needed for users to input the check-in date, duration, room type and even personal information. In such case, the general intent could be set to ‘book-hotel’, and the intermediate sub-intents might include ‘book-hotel-date-selected’, ‘book-hotel-room-selected’, etc.

The more detailed we provide in sub-intents, the more insight we could get from session flow report.

Handled and Not Handled Messages

When sending user messages to chatbase, one optional property ‘not_handled’ can be supplied to indicate whether this user message is understood by bot.

It’s especially useful when we want to optimize our bot with respect to not handled rate, which has positive correlation with users’ discontent with bot.

Session

Chatbase offers two ways to define your bot’s sessions

1. Automatically calculated by chatbase: session starts when a user sends a message to your bot and ends after 15 minutes of inactivity.
2. Custom session ID: you can specify your custom session ID for each message. The way you specify will affect how metrics are evaluated.

Take the following interactions between user and bot for example:

[1:00pm] user: Hello 
[1:01pm] bot: Hi, how can I help? 
[1:02pm] user: Show me my account info 
[1:03pm] bot: Here is your account no. ########## 
[1:04pm] user: Thanks 
[1:05pm] bot: Do you have any more questions? 
[1:30pm] user: No, thank you
[1:31pm] bot: Goodbye

We will acquire different metric values by different strategies of defining session ID. By automatic strategy, the first session lasted 4 minutes from 1:00pm until 1:04pm (the last user message of the session) and the second session lasted for 1 minute.

+-----------+----------+--------------------+------------------+
|  Feature  | Sessions | Avg Session Length | Msgs Per Session |
+-----------+----------+--------------------+------------------+
| Automatic |        2 | 2 min 30 sec       |                2 |
| Custom    |        1 | 30 min             |                4 |
+-----------+----------+--------------------+------------------+

Sending Messages to Chatbase

Messages can be sent to Chatbase by Restful API call or wrapped libraries in .Net, Node.js or Python.

Sending via API call

URL: https://chatbase.com/api/message
HTTP Method: POST
Request Body (JSON):
{
  api_key:
  type:              "user" or "agent"
  user_id:     
  time_stamp:    
  platform:          "Facebook", "iOS", "Line", .....
  message:          
  intent:       
  not_handled:
  version:
  session_id: 
}

Sending via Node.js

const chatbase = require('@google/chatbase');
chatbase.newMessage('my-api-key')
 .setPlatform('Line')
 .setMessage('I want to book hotel')
 .setUserId('UID-00000000')
 .send()
 .then(msg => console.log(msg.getCreateResponse()))
 .catch(err => console.error(err));

Link Tracking

Chatbase offers Link Tracking feature which enables you to inspect whether and when users tap/click on the links you displayed in bot messages.

Note: All query parameters per path are removed to maintain privacy.

There are two ways to do link tracking:

1. Link Redirection

URL: https://chatbase.com/r?
Parameters:
  api_key:    
  url:        
  platform:   
Example (Wrapped URL):
  https://chatbase.com/r?api_key=102303&url=http://myURL.com/some-page&platform=slack

2. Telling click event by API

URL: https://chatbase.com/api/click
HTTP Method: POST
Parameters:
  api_key:     
  url:         
  platform:    
  (user_id):     
  (version):

Reports

Please allow up to one hour for newly uploaded messages to be processed and appear on the Messages Report.

The most beneficial feature for user behavior researcher is the visual reports. Chatbase gathers statistics from the messages you sent and gives you quantitative reports about user engagement like Daily/Weekly/Monthly Active Users session metrics.

Quantitative reports

• Messages

• Active Users

Active Users Report Generated by Chatbase

• Session Metrics

User Engagement Report — Sessions Analysis

• Retetion Cohorts

Retention Cohorts Report Generated by Chatbase

• Funnels (Drop Rate)

“Funnels help you identify points of user drop-off toward a goal that has multiple steps.”

Qualitative Reports

• Transcripts

• Session Flow

Session Flow Report Generated by Chatbase

Discussion

• Metrics are not immediately updated in the Chatbase UI (reports).
• Take care of message out-of-sequence problem if you send user message and agent message at the same time.