Testing Alexa Skills — The grail quest
You almost finished your Amazon Alexa Skill and are now started the quest for the Holy Grail of Alexa Testing. Now, you are desperately searching for a way to automate it. Even, googling it gave no obvious outcome.
Fortunately, your search is over.
Sometimes, you leave testing until the end, and you always regret it. This time is even worse, as you are developing your killer Alexa Skill and before releasing it into the wild, you want to be confident that the most basic use cases work.
Just google a bit, search StackOverflow and nothing seems to stand out.
There exist several ways of testing Alexa Skills, however, either they are mostly manual or require the introduction of other languages to define the tests, sometimes not even programming languages, so your spectrum will be somewhat limited.
Here, we are introducing a way of automating part of the process using python (Node counterpart will be available in the future) .
It has been discussed many times that testing a voice-based UI is entirely different than any other test because there are millions of ways of saying the same thing and each user would do it differently, correct, but this is only an excuse to avoid testing your Skill.
The manual ways
The skill builder’s new dialog model reduces the amount of handling required in the skill endpoint (backend) to manage slot filling, prompts, and confirmations for a customer intent. All you have to do is respond with a Dialog.Delegate directive if the status is not complete. Alexa will manage the dialogs for you, significantly reducing the amount of handling needed by you to create a great experience.
Amazon provides excellent tools to test this interaction with your VUI and Skill manually.
Alexa Simulator can interact with your Skill either sending audio or text. There are other features like sending Manual JSON request. However, it is somewhat limited as each operation is a single turn operation or one-off exchange (no “dialog”).
Cute, but we can’t automate our tests, and there’s even no way of resending the same input, so it involves a lot of texting or talking.
The semi-automated ways
Amazon recently announced the Alexa Skill Management API and Alexa Skills Kit Command-line Interface in the Alexa Skills Kit. They provide us with some tools to start automating some steps, but it soon becomes very laborious.
The following example shows ask api invoking a skill providing a previously crafted JSON which contains the info to launch the Skill. The output, which is also JSON is sent to jq to beautify and colorize it.
Lex Bot Tester
Lex Bot Tester is a framework aimed to simplify test automation of Alexa Skills and Lex Bots. It’s written in python (a Node version is in the making as mentioned before).
The main idea is to be able to automate part of the process. Aimed principally at functional tests and integration tests, as, contrary to other products that use a local server this uses the real Alexa back-end.
The dialog model identifies required slots and the prompts to collect and confirm the slot values as a way of simplifying your code for collecting and confirming slot values.
To use these prompts in a dialog with the user, respond to an incoming IntentRequest with the Dialog.Delegate directive. Alexa then determines the next step in the dialog and uses the appropriate prompts to ask the user for the information.
We will leverage this to automate our tests. We provide a data structure containing the steps of our conversation and the Slots to be filled, and at the end, we will be able to verify if all the Slots for what elicitation was required have already been filled and eventually you can also check those values.
Having explained that, you could take a look at lex-bot-tester implementation on Github, but what’s better than a real-life example to understand how the tests can be simplified amply.
Having explained that, you could take a look at lex-bot-tester implementation on Github, but what’s better than a real-life example to understand how the tests can be simplified amply.
This example uses Alexa Skill Blueprint:
- BookMyTrip
and the test created with lex-bot-tester looks something like this (see full source)
Which, when run, gives a conversation like the one below
And the interesting point to notice is how the conversation is maintained with Alexa, which is providing the values corresponding to the utterances given by the test. In the highlighted case, we are sending “five days from now,” and Alexa converted it to the real date “2018–01–28” (which is five days from the day I’m writing this).
I hope this article helps you improve the way you are testing your Alexa Skills.
Next, we will cover some improvements to these examples, a more sophisticated example using the High Low Game Skill, how to automatically provide some values for the Slots in our tests and other related concepts.
Continue reading
If you are interested in creating tests automatically for your skills check Testing Alexa Skills — Autogenerated tests