Analytics for Actions
Once your Action is deployed, your analytics in the Actions Console are a valuable resource to understand how your Action is used and how well it is performing.
If you are using Dialogflow to implement your Action, it has an analytics page that gives you insight on how well your agent is performing. The Actions Console also has an analytics section that covers usage, health, and discovery.
In this post, we use the analytics of several deployed Actions to learn how well they are performing and what improvements might be needed.
Let’s start with data
One of our sample apps, Number Genie, was one of the first Actions on Google. This Action is mainly used as a reference implementation of our design principles. Here is its session flow view in the Dialogflow analytics page:
We can see that there are two main ways the Action is invoked. It is mostly triggered via a deep link. Number Genie has a useful demo feature that allows you to set the target number which we use in our presentations when we want to show particular dialog paths as the user approaches the answer (try it yourself: “talk to number genie about 45”). The other main entry point to the Action is the welcome intent, start_game, that handles the GOOGLE_ASSISTANT_WELCOME event.
We can also see from this graph that a very low percentage of user interactions are mismatched (see the No match intent). These mismatches happen when your agent fails to match the user utterance to a particular intent, which is usually due to not finding matching intent training phrases or matching entity values.
We can drill deeper into the conversation paths to see the subsequent intents invoked. We can see a low number of users choose to play the game again (see the play-again-yes intent), so we may want to improve the design to keep the user playing, but it’s not much of a concern for our demo number guessing game.
Actions console health data
Here is the Actions Console analytics for one of our sample Actions called “Quotes Sample”:
This Action looks very healthy. There are no errors, and the latency is very low (well below the maximum response time of 10 seconds allowed for Actions). This Action’s fulfillment is implemented with Cloud Functions for Firebase using Dialogflow’s inline editor.
Here is the view of another Action that isn’t considered healthy:
This Action is regularly unresponsive and also occasionally times out. See our recent post about debugging Actions on how to find and fix errors. Unhealthy Actions are a bad experience for users, and if Google detects that the Action is experiencing an outage, the Action is suspended.
Discovery
Also in the Actions Console, the “Discovery” tab shows the various ways the Action is invoked. Here is the data for Number Genie:
Both “number genie” and “number genie game” are explicit invocations. This kind of invocation occurs when a user tells the Google Assistant to use the Action by name.
“AUTO_MATCHED_BY_GOOGLE” refers to implicit triggering, which is when the Assistant tries to match a user’s intent with invocation phrases that have been declared by the Action (for example, “Hey Google, I want a number guessing game”).
Number Genie also supports the actions.intent.PLAY_GAME built-in intent, which is a way for an Action to declare that it is suitable to fulfill a specific category of user requests, such as playing games.
Number Genie has also configured an Action link, which is a unique URL that allows users to invoke the Action by clicking the link.
Retention
An important metric for Actions is the number of active users over the prior 28-day window. Here is the data for another one of our deployed sample Actions, and you can see that the number of returning users is quite low:
Retention could be improved by using notifications and supporting daily updates. See our recent post on how to implement these for your Action.
Another important metric is the abort rate, which is the percent of conversations in which a user aborted or canceled:
A high abort rate may indicate that users are unhappy with your Action. You can use the history feature in Dialogflow to determine what users were doing before they left your Action. Take a look at our recent post on mining your Action’s history.
Analytic tools
Various analytics tools providers can be integrated with Actions on Google. One of these is Chatbase, which provides an API to analyze conversational experiences. Chatbase provides a convenient Node.js client library that you can use in your Actions fulfillment. The Chatbase GUI provides a much more advanced analysis of conversations than Dialogflow.
The Actions on Google Node.js client library has a convenient middleware layer that allows you to invoke a function before any of the intent handlers are invoked. This is a convenient way of adding analytics for all the intent handlers.
Below is a snippet of code that invokes the Chatbase API using their Node.js client library:
const chatbase = require('@google/chatbase');app.middleware((conv) => {
if (!conv.isHealthCheck) { let platform = 'AUDIO';
if (conv.screen) {
if (conv.voice) {
platform = 'SCREEN_VOICE';
} else {
platform = 'SCREEN_KEYBOARD';
}
} if (!(conv.intent === 'Default Fallback Intent' ||
conv.intent === 'No-input')) {
chatbase.newMessage(process.env.CHATBASE_KEY, some_user_id)
.setAsTypeAgent()
.setTimestamp(Date.now().toString())
.setMessage(conv.intent)
.setIntent(conv.intent)
.setCustomSessionId(conv.id)
.setPlatform(platform)
.setAsHandled()
.send()
.then(msg => console.log(msg.getCreateResponse()))
.catch(err => console.error(err)); chatbase.newMessage(process.env.CHATBASE_KEY, some_user_id)
.setAsTypeUser()
.setTimestamp(Date.now().toString())
.setMessage(conv.input.raw)
.setCustomSessionId(conv.id)
.setPlatform(platform)
.setAsHandled()
.send()
.then(msg => console.log(msg.getCreateResponse()))
.catch(err => console.error(err));
} else {
chatbase.newMessage(process.env.CHATBASE_KEY, some_user_id)
.setAsTypeUser()
.setTimestamp(Date.now().toString())
.setMessage(conv.input.raw)
.setCustomSessionId(conv.id)
.setPlatform(platform)
.setAsNotHandled()
.send()
.then(msg => console.log(msg.getCreateResponse()))
.catch(err => console.error(err));
}
}
});
Note that we are ignoring requests that are health checks by checking conv.isHealthCheck. The code assumes you obtained a key from Chatbase and is storing it in process.env.CHATBASE_KEY.
We create a platform identifier so we can track the nature of the user interaction (voice only, screen with voice, or screen with keyboard). Also, note how the code tells Chatbase about user responses that are handled by intent handlers and those that are mismatched and invoke the fallback handlers, such as the Default Fallback Intent.
Once the API is invoked, you can use the Chatbase GUI to analyze the user engagement and learn about ways to optimize the conversational design:
Tracking events
Analytics tools like Chatbase are great for tracking the conversational aspects of your Actions, such as prompts and related user inputs, but that might not be enough to get deep insights on how your users are interacting with your Actions.
Your Action might be tracking other metrics that can also give you insight on how well your design is performing. For example, let’s say you have a game that is tracking the number of steps the user takes to solve a puzzle. It would be very important to track that total, but since it’s not conversational, you need another way to track data like this.
A good (free) option to consider is the Google Analytics Measurement Protocol, which allows you to track events and associated data. The data can then be analyzed in the Google Analytics web GUI.
To be able to use the API, you need to create a new Google Analytics property (even a web site property will work) and then obtain the tracking ID.
There isn’t a client library, but the protocol is quite simple and involves posting data to a URL endpoint. The following code uses the urlencode and request Node.js modules to invoke the API:
const request = require('request');
const urlencode = require('urlencode');const GOOGLE_ANALYTICS_URL =
'https://www.google-analytics.com/collect';
const GOOGLE_ANALYTICS_TRACKING_ID = 'UA-xxxxxxxxx-x';
const VERSION = 'v';
const TRACKING_ID = 'tid';
const CLIENT_ID = 'cid';
const HIT_TYPE = 't';
const EVENT = 'event';
const EVENT_CATEGORY = 'ec';
const EVENT_ACTION = 'ea';
const EVENT_LABEL = 'el';
const COUNT = 'count';const totalAnalytics = (some_user_id, context, count) => {
const options = {
url: GOOGLE_ANALYTICS_URL,
json: false
};
let body = VERSION + '=' + 1;
body += '&' + TRACKING_ID + '=' + GOOGLE_ANALYTICS_TRACKING_ID;
body += '&' + CLIENT_ID + '=' + some_user_id;
body += '&' + HIT_TYPE + '=' + EVENT;
body += '&' + EVENT_CATEGORY + '=' + urlencode(context);
body += '&' + EVENT_ACTION + '=' + COUNT;
body += '&' + EVENT_LABEL + '=' + urlencode(count);
options.body = body;
request.post(options, (error, response, body) => {
if (error) {
console.error(`post: ${error}`);
return;
}
console.log(`analytics response: ${response.statusCode}`);
});
}
We defined our own event called count, which has the value of the number of steps. You can then invoke this function in your intent handler logic at the appropriate time (when the user has solved a puzzle):
totalAnalytics('anonymous', 'steps', counter);In the Google Analytics web GUI, go to the Behavior/Events/Overview menu items to create reports based on the event data:
Keep measuring
Keeping an eye on the analytics of your Actions is a valuable way to ensure your users get the best experience. Analytics can help you improve both the conversational design and technical implementation of your Actions.
These improvements will ensure you have happy users and can help with your ratings in the Actions directory.
Want more? Head over to the Actions on Google community to discuss Actions with other developers. Join the Actions on Google developer community program and you could earn a $200 monthly Google Cloud credit and an Assistant t-shirt when you publish your first app.
