Beginners Guide to Automated Voice App Testing

Florian Treml
Feb 3 · 5 min read

This guide suggests best practices, infrastructure and tools to ensure your voice app continues to deliver outstanding user experience.

Questions When Testing Voice Apps

  • Is my voice app following the designed conversation flow ? Is the conversation flow working as expected ?
  • How does my voice app work under real-life conditions ? Does is handle low audio quality ? Does it handle slow network connections ?
  • Is my voice app available 24x7, or are there any interruptions in service ?

The Art of Challenging Chatbots

The platforms behind powerful voice applications are still evolving and are subject to constant improvements — which means that developers have to rely on components that they do not own and the possible influence is limited.

Testing the Voice Conversation Flow

We will use Bring! Shopping List as an example of a voice app to test. It is published as Alexa Skill, and we can use the Botium Connector for Amazon Alexa with AVS for simulating voice input and output with Botium.

For details about the presented steps and tools please take a look at the Botium Wiki and our Blog!

Record Test Cases

Depending on the technology of your voice app, both text and audio response are shown or either of them.

Botium Box Live Chat — Recorder

You can save the conversation as test case and make some changes afterwards.

  • Refining input and output text and audio
  • Using wildcard matching or utterance lists instead of full text
  • Add additional test steps or asserters
Botium Box Voice Test Case

Synthesize Test Cases with Text-To-Speech

Test cases are showing plain text now instead auf audio input:

Botium Box Live Chat — Text Input

Eliminating Flakiness — Homophone Mappings

In this example, you can see that instead of okay milch ist auf deiner liste the transcription says okay milch is auf seiner liste — this one character difference will make a test case fail:

Transcription problem

Botium provides the option to specify homophone mappings to deal with audio snippets that are often misinterpreted by the Speech-To-Test engine.

Specifying Homophone Mappings

Test cases use these mappings to qualify transcription results as success or failed.

Transcription Problem — Homophone Mapping applied

Testing Real-Life Scenarios

  • Add background noise on various levels
  • Pitch volume up or down
  • Simulate various levels of distance
  • Simulate technical restrictions like GSM phone line or low bandwith
  • Simulate otherwise bad audio quality like interruptions or various levels of silence
  • … you name it …

In Botium Box you can apply various effects for simulating real-life usage scenarios to your own clean recordings or synthesized audio samples.

Botium Box Voice Effects

Continuous Monitoring

  • a smoke test for checking basic behaviour (for instance, just sending a simple hello to the voice app and listing for a response)
  • a scheduler to run the smoke test every few minutes
  • a notification mechansim to inform you in case of failures

With Botium Box, everything you need is coming out of the box.


  • Record your own voice or use synthesized voice
  • Apply audio effects for real-life simulation
  • Conversation flow testing with Botium

See this article in spanish here! 🇪🇸

The Startup

Get smarter at building your thing. Join The Startup’s +786K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +786K followers.

Florian Treml

Written by

Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +786K followers.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store