The 4 Most Important Chatbot Metrics to Track

Over the last two weeks, we have worked with the awesome people at Botanalytics to put together a list of the key metrics to track for any chatbot. Botanalytics is a company which provides an analysis tool to help improve human-to-bot interactions and conversational UI through data.

In this quick read, you’ll find a break down of each of the four metrics supported by an example of bots that adhere well to each of the metrics and for you to get an idea of where perhaps your bot fits.

When developing a chatbot, your work is never done. Releasing it into the wild is only the first step; from there, you’ll need to pay close attention to the conversations your users have with it in order to iterate upon and improve the bot. Just like when releasing an app or website, you need to know what’s working and what’s not. The difference: the metrics you should track.

Not only does this allow you to make your chatbot more engaging, it helps you discover unexpected or new patterns in user and bot behavior as well.

There are a lot of metrics to track, some more important than others and most varying in importance dependent on the category of bot you’re building.

This guide takes you through which metrics to chat, why they’re important, and real-world examples of of how chatbot developers put those metrics to good use.

Average Session Length

Your average session length is an important metric for determining how much time people spend talking to your bot. What it means for your bot will depend on what purpose that bot serves.

Customer service chatbots should aim for a shorter session length; a long one suggests that too much time is taken to arrive at an answer that resolves their issue, indicating a potentially frustrating experience. An entertainment-oriented chatbot, meanwhile, will aim for a high session length to show that its users are highly engaged and don’t want to stop talking.

English with Edwin is a great example of a bot that’s looking for a high average session length. This English-learning bot must keep users engaged to ensure they develop a habit of practicing the language and retaining what they’ve learned. To keep users talking and enjoying the learning experience, Edwin has implemented a gamified experience point system. As users continue practicing English with the bot, they gain points to “level up,” providing them with trackable validation that encourages them to keep going.

Sentiment Score

Sentiment is important for two reasons: on the one hand, emotional understanding allows your bot to respond appropriately to a user’s attitude. On the other, you can quickly determine moments and interactions that led to user frustration or delight.

In either case, your sentiment score helps you understand how users feel in real time throughout a conversation, which you can use to make your bot more engaging and emotionally intelligent.

Christian Grey — a bot based on the eponymous 50 Shades of Grey character — is a bot whose makers pay close attention to user sentiment. After all, the bot is all about sex and romance, where emotional cues are important. Christian Grey aims to provide a fantasy scenario based on the racy book and film series, flirting with users to help them discover their latent kinks. Obviously, users must be titillated and thrilled throughout the conversation; if they seem bored, upset or annoyed by the exchange, then the bot isn’t successful in accomplishing its goal.

Likewise, it’s important not to be too pushy and annoying. Sometimes bots might have difficulty catching what the user means but they still repeat themselves. It’s better to hand the issue over to a human agent when such things are encountered.

Most Active Hours

The peak times that people are talking to your bot is essential for content strategy. If you know when users are active, you know when to better target them with push notifications for increased engagement or click throughs to other sources, like a new blog post or video. This one is big for all the marketers amongst us!

Champ Bot pays close attention to most active hours, because two of its most important features are time-based: game betting and keeping users updated on their favorite players and teams. Because betting is centered around live games, peak activity during games makes it easier to encourage users to participate and bet on the game they’re watching right now. The bot can further take advantage of peak usage by pushing notifications on a followed team’s activities, informing them of ways to earn in game credit, or asking a user to share when it knows the user is most active and likely to respond.

Retention

Perhaps the most important metric of all is retention, or the rate of users who continue using your bot at a given point in time since their first interaction. Common metrics are 1 day, 7 day, and 30 Day retention. By paying close attention to the metrics listed above and making improvements, you’ll likely increase retention rate, too. It’s most important to assess retention when your bot first launches and when you make major changes to it, but it’s always worth checking out.

However it is worth noting that high retention doesn’t make sense in all use cases. For bots that act as customer support agents, high retention means your users are experiencing a large amount of issues. Unless ofcourse you use the same bot for customer support and marketing.

Let’s look at the Christian Grey bot again. On the surface, this chatbot emotionally invests the user with a scenario mimicking that of the 50 Shades franchise. But it obviously has another purpose: to market the 50 Shades movies. After users have chatted for a while with Christian, he asks them to grant permission to receive future messages and updates from him.

Users can also follow his Facebook page, emulating a Facebook friendship. By retaining its users and keeping them invested in the brand over time, the chatbot keeps them within easy reach for promotions and updates on the film series.

BONUS TIP: Importance of Onboarding

We all know that first impressions matter, and it’s no different with your chatbot. Onboarding affects other metrics like average session length, engagement and retention rates. Your onboarding strategy should not only teach people how to use the bot, but also show why it’s worth using at all. Failure to manage user expectation sets them up for a disappointing experience, which will be verified by lowered retention rates.

Art Chatbot wants to make it as easy as possible for users to discover and buy works of art, because of this it’s important their onboarding process puts art at the user’s fingertips from the start.

When the user reaches out, Art Chatbot immediately offers a handful of options for discovering artists and viewing featured works. With button-press menu options, users can quickly scan what the bot does. Once the chatbot has demonstrated its value, it invites the user to tell it their tastes or what it is exactly they’re looking for. This strategy is important for retail bots that need to surface solutions for a quick purchase, while also encouraging users to delve deeper and browse if they feel like it.

Check out Botanalytics to learn more about the work they do and their useful analytics tool. If your bot has been interacting and has a good number of users, perhaps it’s time to get some hard data to work off of in order to keep improving it.