The practical guide to chatbot metrics and analytics

These days I get the feeling that wherever I turn, I encounter a chatbot: some great, some OK, many completely pointless.

There’s no way around it: chatbots are everywhere, and it seems (almost) everyone wants one.

According to research, 80% of businesses aim to implement one by the time 2020 rolls around.

Yet many of these very same companies confess to not being sure how to gauge the efficiency of their chatbot in the first place.

Perhaps it’s no wonder that the rise of the chatbot has been a bumpy ride for some. While some companies have been hugely successful in adopting chatbots, others have stumbled or failed in their attempts.

And while some chatbots have sailed smoothly, others have nosedived and sunk.

So how do you ensure that the chatbot you’ve invested in becomes a hit rather than a miss? Well, you approach it in the same way you would any other strand of your business: you measure, measure, measure — then adjust accordingly.

Here are the key metrics and KPIs you need to pay attention to:

Total number of users

This is your bread and butter metric. It will give you the total number of users and show the amount of data that your chatbot has been exposed to. This will, in turn, provide crucial information about market size and your chatbot’s overall success.

You can drill this metric down by dividing chatbot users into further groups, like active users, engaged users and new users.

Revenue growth

Here’s another priority: is your chatbot actually making you any money?

There are a number of ways to evaluate your bot’s impact on revenue, depending on your bot’s purpose. You will want to keep in mind that a poorly-performing bot can have a knock-on effect on the rest of your finances.

The most obvious way to measure a customer service bot’s revenue growth is by identifying the amount of money you save compared to employing a 24/7 customer service team.

But that in itself is not enough: you also need to consider how well the bot is completing the customer service assigned to it.

That’s where self-service rates and NPS scores come in.

Self-service rate

This metric helps you identify the number of users who get what they want from the chatbot without any human input. For example, if your chatbot’s goal was to sell a particular product, you will measure the percentage of user interactions that achieved that goal.

Your self-service rate will correspond closely to your revenue growth — i.e., how much money is your chatbot saving you by doing what it’s intended to do? The higher your self-service rates, the better.

Satisfaction rate

Are your users happy with your chatbot’s performance? Why not ask them?

You can do this by using what according to many is ‘the only metric worth knowing,’ viz. the Net Promoter Score (NPS): “On a scale of 1–10, how likely is it that you would recommend our chatbot to a friend/colleague?” The NPS gives you a key to understanding your customer’s experience, and therefore your chatbot’s performance.

Another way to work out user satisfaction is through a tailored exit survey. You can keep this simple (“Did the bot perform well?” — Yes or No) , or invite more detailed answers.

User interactions

The total number of user interactions (rather than users) offers a basic yet solid metric for getting a better grasp on your chatbot’s performance.

A chatbot might have lots of users but a minimal amount of interactions, or it may have a small number of users that interact with it frequently.

If a conversation isn’t continuous, you can’t guarantee its effectiveness; a conversation that goes on for a while, on the other hand, will allow you to measure the number of in- and out-messages, giving you an indication of whether the chat was useful or not.

Conversation statistics offer another key way of tracking the performance of your chatbot. The number of different types of conversations (e.g. new conversations, total number of conversations) reflects the usage of the chatbot over a certain period of time.

This helps you determine users’ re-engagement times and behavior. You can then use these statistics to enhance your chatbot and par it with users’ requirements.

Organic users

It’s super-annoying when a chatbot keeps popping up, begging us to use it. So if users come back of their own accord without being prompted, that’s a great sign — and a metric worth counting. ‘Organic’ users come in with a purpose; you can measure their number through messages initiated by the user, not the bot.

Goal completion rate (GCR)

Ideally, a chatbot should have not only a goal, but a specific purpose. Remember, a chatbot that does one thing very well is infinitely more useful than one that does many things poorly. T

GCR captures the percentage of engagements that are successful on this basis. It does so by tracking how many conversations achieve your bot’s goal. For example, if your chatbot featured on an e-commerce site, the GCR might relate to how many conversations resulted in sales on the site.

Activation rate

This is the rate at which a user responds to a chatbot’s message with a question or answer that relates to your business goal. For example, a chatbot designed to give makeup tips would receive an activation rate when a user gave their eye color.

Fall back rate (FBR)

Chatbots are expected to fail sometimes — but is this happening occasionally or regularly? This is what FBR measures. FBR is the percentage of times a chatbot fails at delivering, or comes close to failing.

Confusion triggers

Even bots with the most sophisticated NLP capacity are unable to understand everything a user says. Confusion triggers are a helpful indicator for working out how and where a chatbot needs to be improved. There are different types of triggers: for instance, the chatbot can’t understand a comment; or the user sends a one or more messages that are beyond a chatbot’s remit; or the bot needs to delegate the task to a customer service agent after a failed interaction.

Each of these triggers will tell you something about a chatbot’s performance. The confusion rate is measured as follows:

Confusion Rate = number of times the chatbot had to fall back / total number of messages received.

Retention rate

Retention rate represents the percentage of users who return to a chatbot over a specified period of time. This timespan depends on the bot’s purpose. For example, a fitness chatbot would require daily interaction and would benefit from analysis of its day-by-day retention.

Artificial intelligence and machine learning rate

How strong is the AI in your chatbot? You can measure this by checking the percentage of user questions that your bot has understood correctly.

An agent with robust machine learning will be able to continually run its own gap analysis to highlight potential areas for improvement.

The most successful chatbots are the ones who are constantly revising, adapting and iterating their conversation flows in response to their users.

That’s where Hubspot’s Chatbot Builder comes in handy: you can qualify your leads, allow your prospects to book a meeting with sales or ensure your customers are always happy through 24/7 online support with chatbots.

The best tools for bot analytics

Dashbot

Dashbot is one of the largest and best-known chatbot analytics platforms. It lets you track a wide range of metrics in terms of retention and engagement, conversational analytics and user behavior.

Dashbot also lets you use bot-specific metrics for a deeper understanding of the conversations your users are having. For example, sentiment analysis gives you a high-level view of user moods, while the conversation funnel shows which kinds of questions your users are asking at different intervals, and how the conversation tends to progress from there.

The short version: Dashbot is strong in terms of better understanding conversations.

Botanalytics

Botanalytics is the best tool for tracking individual users. Its dashboard displays the user lifecycle, charting the length and date of each conversation and the number of conversations per user. This is helpful for figuring out which of your chatbot’s users are most active.

The platform also gives deep user logging by giving transcripts of each conversation.

The short version: Botanalytics is best for tracking user lifecycle

Chatbase

At Google I/O this year, Google quietly introduced a new chatbot analytics platform called Chatbase, developed within the company’s internal R&D incubator, Area 120.

Google’s chatbot analytics platform Chatbase offers tools to analyze and optimize chatbots more easily. This includes giving bot builders the ability to understand what works to increase customer conversions, improve the bot’s accuracy, and create a better user experience. This data is available through an analytics dashboard, where developers can track specific metrics like active users, sessions, and user retention. These insights give an overall picture of the bot’s health.

The dashboard also lets bot creators compare the bot’s metrics across platforms, to see if some platforms need additional optimizations.

The short version: Chatbase is best for a range of easy-to-access optimization tools.

Originally published at blog.growthbot.org.