Using Custom Events to Improve Retention on Voice
Engagement and retention is the top priority of many developers and brands for their voice applications. Without a clear picture of your users and their behaviour, it’s difficult to make impactful product decisions.
In this post, I’ll be detailing what “custom events” are, how custom events managed to answer my burning questions for my skill, State Quiz Skill, and the actions I’ve taken to address these issues. I’ll also show you how you can set these custom events in under a minute!
What are custom events?
Custom events are flags that you define depending on how you want to measure your skill. These events are based on the incoming and outgoing request payload.
For example, you can create a custom event to capture sessions with a certain value within sessionAttribute, or sessions that last over 10 turns, or sessions that have more than two FallbackIntents.
With these flags, a developer can answer more in-depth questions about how users are interacting with the skill.
Using Custom Events to Answer Questions
How long does it take to finish the quiz?
One question that arose after observing various users interact with my skill is that some blaze through the skill, and some take their time. I wanted to know what the upper bound was in terms of session lengths, and what were the behaviours within.
Using VoiceHero, I was able to segment sessions based on their relative session length, meaning I was easily able to identify and tag sessions whose lengths are in the top 10% of all my sessions.
With this information, I can dig into the sessions that tend to go longer, and identify any friction points.
Furthermore, I can dig into the types of users who have longer sessions, and determine if their retention and engagement is higher. After digging through these users using custom cohorts (another useful VoiceHero feature!), I’ve found that the users with longer sessions tend to have less retention, so maybe shortening the interaction might drive stickiness.
Is the quiz too hard?
After finding out that people can take up to 13 minutes (!) to finish a quiz, I decided to dig into the question: is the quiz too hard?
I wanted to see the conversion rates as users go through the quiz. How many people get some amount of questions right? What are the average scores of users who finish the quiz?
For this, I set up 2 custom events: kicking off the quiz, and getting more than 3 questions right.
After I did this, I can see that there was a huge discrepancy between total sessions captured by these two events. Let’s take a look at this visually, with VoiceHero’s funnels.
On the dashboard, I setup a new funnel, with the two custom events I just made, and some I had already.
I can see that there seems to be a dropoff between an “Active” session and a session that “Started the Quiz”. I’ll take a look, and make sure I’m not losing users in that first interaction!
Secondly, I can see that a sliver of people get even 2 questions right, which indicates that the quiz might need to be easier. Furthermore, I can dig into the users who do get questions right, and compare their retention vs. other cohorts (once again using VoiceHero’s cohorting tool).
Debugging — How many people hit a bug?
Recently, it has come to my attention that the data in one of the questions is incorrect, and users are reporting a bug with it. I decided to take a look — how many people have hit this specific bug? Because VoiceHero’s custom events can filter based off session attributes, I was able to pluck out the sessions that were relevant: those that asked a question about Tennessee.
Setting up New Custom Events
How many people get asked a type of question?
After digging into sessions using VoiceHero’s Session Explorer, I found that most people get the “Abbreviation” questions right. I wanted to verify if that’s true, so I set up a custom event to pluck out: “Sessions that asked for an Abbreviation”.
As you can see, it is very quick to set up a custom event, and immediately I can see the results. In fact, VoiceHero backfills data, identifying sessions in the past that have triggered this event. Furthermore, in this flow, you can see other condition types that are available: intent name, number of turns, etc.
Conclusion
In conclusion, custom events gave me huge amounts of insight into my skill. By understanding top session lengths, correct answer rate, and erroneous sessions, I have actionable next steps that I can test to ultimately drive retention on my Alexa Skill.
If you want to join our closed beta, shoot me an email (jacob@voicehero.ai) and I’ll be in touch.
About VoiceHero
VoiceHero is an analytics platform that helps developers understand user behaviour to improve engagement and performance of voice applications. Backed by Amazon Alexa and Techstars.
