Effective Integration Testing of Alexa Skills
What are Alexa Skills?
Alexa Skills are applications that extend the out-of-the-box capabilities of your Echo home assistant - similar to an app for your phone.
Before we get into the testing approach, I want to provide a short refresher on Alexa terminologies:
1. Skill: An application for Alexa - e.g. a trip planner
2. Intent: Representation of a particular action within a Skill - e.g. my next train
3. Utterance: Different ways of triggering an Intent - e.g. when does my next train depart
4. Slot: A parameter that can be used together with an Utterance to trigger an Intent - e.g. when is my next train from {Central Station} to {Town Hall}
Why Do You Need to Test Alexa Skills?
If your Alexa Skill does not understand the user’s intention, it’s only a matter of time until the user gets frustrated and stops using your Alexa Skill. Therefore, you want to make sure that you have good test coverage to ensure the user receives a good response for all possible outcomes. Don’t forget to include negative tests.
Why Do You Want to Automate Your Tests?
Testing voice-based apps can be very time consuming and regression testing becomes boring very quickly. The testing effort increases with every added Intent or Utterance - therefore, test automation is your friend.
How Do We Get Started?
I am a big fan of test-driven-development and writing unit tests before writing the actual code. However, we are focusing on end-to-end integration in this write-up. Therefore I assume you have already implemented your Skill.
When I write a new Skill, I start with a simple Intent without Utterances or Slots. I leave it in a Skill to have a very basic test Intent available. If there are any surprises or issues - e.g. the account linking between AWS and Developer Portal, or a wonky internet connection - then I can at least test a simple happy path.
First of all, we want to run some manual tests. After a successful manual voice-based test, we repeat the test in the Developer Portal. We can either do that by using our voice and the microphone button, or by typing the words in the text field.
Moving Towards Test Automation
Running the manual test in the Developer Portal will show the request and response in JSON format. For our testing approach we will use the request as our test input and we will store the request from the above “JSON Input” text area as a JSON file in our testdata directory. Then we implement a test method using the Golang built-in Test framework that reads the test data, unmarshals it and sends it to the actual Intent. The Intent replies with a response that we compare with the expected result as illustrated below:
The example below shows a test method that takes the request input data from a file (const spanishTestFile). The byte slice from the file is then encoded by invoking the Unmarshal() method. We then send the test data to the HandleTranslateIntent that handles the translations into several languages. As function parameters we pass on the request information and the language that we want to translate our message into.
This is a very simple Intent that takes an AMAZON.SearchQuery as an Utterance.
As a next step we write an assertion to validate that the actual response equals the expected response. To simplify our assertions we are using the github.com/stretchr/testify/assert library. Using the github.com/arienmalec/alexa-go helps us to simplify our Alexa requests and responses. The test is then kicked off with go test command, which runs files ending with _test.go.
When you have complex Intents, I would recommend testing several Utterances and include negative tests around your Utterances and Slots. This approach does not cater for edge cases, like dealing with different accents. Therefore, you will still need user acceptance tests, but that is a different topic on its own.
In a Nutshell
Everyone - including your users, product owners, UAT testers - wants a repeatable outcome. Automated integration tests help you to compare the actual response to the expected response. Treat your test data and tests as code and keep them in your version control repository like all your other code. Use whatever language you like - the described approach will work with most languages and testing frameworks. Last but not least: make sure to hook the integration tests into your CICD pipeline.
