Testing telephony with Twilio, AWS, Node.js and Cucumber

At carsales, we offer numerous ways for customers to contact sellers. One of these ways is calling. Telephony is our custom built system responsible for handling all incoming calls. In most cases when you call a number on our site (we call it the virtual number), you’re actually calling our phone system directly. We handle the call in order to gather useful information like which car you’re enquiring on and then dial the seller and connect you both.

As the QA for this system, I’ve designed and implemented a suite of automated end-to-end checks which monitor basic functionality 24/7 to ensure we have the maximum amount of uptime. This article is a “How-to” for creating telephony end-to-end test automation.

What’s Involved In Telephony?

There are a number of events that contribute to handling your call experience. Here’s a general idea of what goes into it:

Example call flow from start to end

1. You dial the virtual number on carsales
2. We receive your call and play you a series of interactive voice response (IVR) audio messages. The IVR messages include a greeting and a request for the item code. 2.a. At the same time, we generate a unique call ID.
3. Once the code is validated, we are able to map the seller’s ad to your call, and you’re added to a shared call. You hear wait music.
4. On the seller’s side we check if they’re willing to accept a call right now, or if they have opted to receive a voicemail at this time.
5. If the seller is accepting calls, they are added to the shared call
6. Once you hang up, we give the seller accurate information about the call so that they can follow up on your enquiry. This metadata is stored under a unique ID and is passed to the seller via their UI Portal.

What’s Being Automated?

To automate this flow, you’ll need to mimic how the customer places the call to the virtual number, presses dial tones, waits, talks and hangs up, as well as how the seller receives the call. You’ll also need access to log details which in my case is AWS Cloudwatch Logs.

Which Tools?

1. Cucumber to structure and run test scripts
2. Twilio to model call behaviour
3. AWS SDK for Node.js to query Cloudwatch Logs

Cucumber.io is an open-source BDD test automation tool. The bread and butter of Cucumber are feature files and step definitions.
A feature file is comprised of plain text instructions which detail the test “steps” and expectations. It’s like Simon in a game of ‘Simon Says’.
Step definitions map each instruction in your feature file to code blocks which control how a step is executed.

Before deep diving into the code, here’s an outlook for what your directory structure will look like when you finish this tutorial.

Step 1: Create your feature file

Install cucumber npm install cucumber. In your root dir, add the folder src. Under src, create feature and step folders.

In feature, create the feature file and write out the steps of the test. We’ll call it answered.call.feature.

Step 2: Calling using Twilio

Twilio is a cloud based communications service which enables developers to make calls (amongst other things) programatically. We’re going to use it to model the customer’s behaviour when they place a call to one of our sellers. We’ll also use it to mimic the seller’s behaviour when they receive the call.
First up you need to claim two numbers; one to act as the customer and another to act as the seller.

You’ll need to sign up with Twilio to access the console. At the time of this article local numbers in Australia cost $6 each. Placing a call costs $0.0240 /min and receiving a call is $0.0100/min. The estimated cost of this particular test $12.34.

Once you’re set up and logged in go to Phone Numbers → Buy a number. In Capabilities, select Voice, since the numbers are only for calling.

Hit Search, select a number, claim it and make a note of it. We’ll come back to it later.

Repeat the same process to claim a second number.

Now you’ve got the numbers needs to specify what they do when they are used to make or receive a call. Twilio Functions is a beta service which enables you to do exactly this with a few lines of code. Using code you can control behaviour such as playing audio and setting pauses, call recording and sending dial tones.

To set up a Function, go to the sidebar and click Runtime → Functions. Create a new function and select a blank template.

Create a call function in Twilio

Name it Customer Caller and give the URL the path customerCaller. Next, provide this code:

The final result will look like this…

Hit save.

Make a note of the URL of the function. It’ll look something like http://{yourCustomDomain}/customerCaller. We’ll come back to this one later. Let’s move on to the seller.

Step 3: Mimic the seller’s behaviour

Create a new function, just like you did in Step 2. Use this code in your configuration:

You want this code to be triggered by an actual inbound call. In order to configure this, save the function to the phone number you just claimed.

Go to Phone Numbers → Manage Numbers → Click the number provisioned for the seller → Under A call comes in, provide the URL to your function for the seller. You can either copy and paste it or click from the drop down.

If you wanted to test this out, all you have to do is dial the number you just configured. You’ll hear an automated voice say “Call answered”, then 20 seconds of silence and so on.

Step 4: Trigger the customers call from your step definition

Great! Now that the customer and seller are set up it’s time to write the script that will trigger the customer’s call. This is where twilio-sdk becomes very useful. The SDK opens up many APIs that you can call **excuse the pun** from your cucumber step file. You can find docs here:
https://www.twilio.com/docs/voice/api.

npm install twilio-client --save

The aptly named call resource does exactly what’s written on the box. The catch is that for it to work, you’ll have to authenticate the request using your accountSid and authToken. You can find both these values on the console dashboard under Project Info.

Remember when you created your Customer Caller function? Well now you’re going to use it! Grab the URL and pump it into common.steps.js:

Step 5: Get the call ID from Cloudwatch Logs

During the call, various messages are published under a unique call ID. Those logs are stored in AWS Cloudwatch Logs.
Imagine though, that it’s peak time and there are multiple calls running at once. You can’t just wait for the next call ID to be logged because you can’t be sure that it’s the right one.

The good news is that Cloudwatch Logs can be filtered by strings as well as object structure. You can use the virtual number, the customer’s number and the seller’s number to locate the relevant ID. Say for example, there is an object logged that has this structure:

{
   "callId": "12345678-BBBb-cCCCC-0000-123456789012",
   "virtualNumber": "0311111111",
      "session": {
         "caller": "0322222222"
         "callee": "0333333333"
      }
}

Cloudwatch Logs query syntax lets you search using this query:

{ ($.virtualNumber = "0311111111") && ($.session.caller = 0322222222) }

Using the Cloudwatch Logs API means you can make this query from your step definition. Please note that setting up permissions or accessing resources using roles can be a little fiddly but is not within the scope of this article.

First up, install the SDK

npm install aws-sdk

The filterLogEvents method is perfect for retrieving logs according to your specified query. The method takes a log group name, filter pattern, optional limit to cap the number of logs you get back and an optional start and end time if you need to get logs for a specific time duration.

Step 6: Check the call details are accurate

So now you’ve executed the call and you’ve got the unique call ID. All that’s left is to verify that the details associated with the call are correct! Carsales system stores all details against a unique ID and makes this information accessible internally via API.

You can use request-promise to make API calls from your common.steps.js file. This package allows you to execute CRUD operations with promise support from your code and also has features for various authentication types. Since this example is fairly simple, we’ll only need to execute a GET request.

npm install request-promise

Next step is to execute a GET request to get the call data that was stored when the call ended. Since the retrieval of the information is a separate step from the verification step, you’ll need to store the call information in a shared context. Luckily, Cucumber exposes an isolated context for each scenario via the this keyword.

In Conclusion…

You’ve created a simple call test! Wasn’t that easy?!

You’ve covered automation of caller and callee behaviour, retrieving logs and GET-ing data from your internal system. Depending on your use case, you might find it worthwhile investing time in calibrating your caller and callee behaviour. One of the major difficulties I came up against was identifying when and for how long the telephony system experienced delays. The minor delays threw off my timing for waits, audio and dials and as a result tests failed when there was in fact nothing wrong with our system. One of the solutions what was suggested to me by our amazing devops Jesus Tovar was to run a dummy call before my test. This proved highly effective and reduced the false negatives greatly.
Another challenge was to determine which parameters were best for filtering logs is very important. I started filtering just by string and subbing in the virtual number. As my test suite grew and calls were overlapping I quickly realised that I had to find a better method for deducing which logs were relevant. The most robust method I found was to query log objects, rather than just strings. This method has produced consistent and reliable results. I only need to calibrate the time range a little before it was squeaky clean.

And Thanks To

A big shout out to Kaveh Azad, Philippe Roy, Jesus Tovar, Craig Peebles and Nam Duong and for editing assistance and to Echo dev team for sharing their knowledge and experience.