Adding call routing to your on-call rotation with Twilio and Lambda
- How to actually call your on-call engineers
In a previous article I went over how to create a Slack bot with Serverless Framework, to get the person on-call from a PagerDuty on-call rotation and respond with name and number in Slack. The bot described in that article is actually preceded by another app we have in Schibsted, one that we’ve had in many different versions and flavors over the years. Its purpose is to direct calls to the on-call engineer via a Twilio Voice Webhook. In this version I’ve chosen to use the Serverless Framework and deploy the function on AWS Lambda, but the exact same code might as well be hosted on Twilio Functions.
A few years after our first version, PagerDuty launched their Live Call Routing feature that basically does the same thing. It’s a great feature as it’s fully managed and you have less moving parts, but we found it a bit too expensive for what we actually got and countries for which they had supported phone numbers were too limited. We decided to stick with our own solution.
Let’s go about setting this up! For those who read the Slack bot article, this setup is way easier as you only need to fetch users based on the given schedule id. No transforming team names into schedules etc. This is what we’re essentially building.
Setting up your function
As with every Serverless project you start with the serverless.yml file. For this project we need one handler that accepts a PagerDuty Schedule ID as input and has tokens to communicate with PagerDuty and verifying Twilio signatures.
The Lambda function uses the Twilio SDK for two things: verifying signatures and generating the dial response. If you want to reduce the size of the lambda, these functions can easily be added by one self. Unlike Slack however, Twilio does provide a handy utility function to verify the signature. Using it in a Lambda function required some extra code though, as you don’t have access to the requested url needed to verify the signature. The url needs to be stitched together using data from the event.
Another part that required some debugging was the use of params in the signature, as the body needed to be parsed and provided as an object to later be transformed to string again by the SDK. It only worked with regular objects too, and not UrlSearchParams. Once I got that sorted it was smooth sailing.
To get the phone number of the person on-call for the given schedule, the same code as described in Slack bot with Serverless Framework is used. First it uses the schedules.listUsersOnCall(schedule_id, options) function to get the user and the users.listContactMethods(user_id) function to get the phone number of that user.
Last step is to generate the XML response with the Twilio SDK Object twiml.VoiceResponse() and the dial(phone_number) function.
When you have it all ready, with the PagerDuty and Twilio tokens set in Parameter Store you can deploy the solution with
$ npx sls deployAll code for this is available at https://github.com/schibsted/sls-oncall-twilio.
Hooking it up with Twilio
To hook this up with Twilio you need to create a Twilio account. There is a free trial and each app you create, that Twilio also calls Accounts, you get $15 credits to be used to test out functionality. Some features are turned off with trial accounts, like buying certain phone numbers and such, but all things needed for this demo are there for free. If you are to run this for real however, then you need to start a subscription.
Once in, buy a phone number using the trial credits. There are some limitations to buying phone numbers in certain countries, like several countries requiring a verified physical address to be able to buy or that some phone numbers require you to upgrade the account before proceeding. Having tried Sweden and failed, I went for a US based number in Swedesboro, New Jersey. A lovely little borough with a population of 2,584 according to the 2010 census.
When buying the phone number you agree to add an US based emergency address associated with this phone number. I’m not suggesting you break any laws or terms and conditions, or suggesting you set any US based address, like the lovely The Red Hen restaurant in Swedesboro, New Jersey, as the emergency address. I would suggest no such thing! When setting this up for real, I suggest you pick a phone number local to you and jump through all the hoops needed for that.
There are a few more things to do before we can test the full flow. Let’s start with the webhook and token. The token can be fetched from the General Settings or from the app Dashboard and you need it to verify signatures as explained above. This is generally good to have before deploying the solution. Once deployed, you’ll get an endpoint address back that you set as the webhook address under incoming calls. You’ll find this view by going to Active numbers and clicking the number you purchased.
Two more things before we’re done. With the trial account you can only call/route phone calls to pre-verified phone numbers called Verified Caller IDs. Make sure the number you are trying to call is in the Verified Caller ID list. The last thing is changing the Geo permissions to enable the countries you wish to call to. This is a security mechanism Twilio has to help reduce toll fraud. You will find this setting under Voice > Setting > Geo permissions. The on-call numbers I added to the Verified Caller ID list were both Swedish, which meant I just enabled Sweden as approved for now.
All set! Call the number you bought and be routed to the person on-call! If it does not work, Twilio provides a great Call Log with clear error messages of what went wrong and what you should do to fix it.
I hope you enjoyed this write-up and that it might help you set up a call router for your on-call rotation. If you have any questions or suggestions, feel free to reach out on Twitter or submit an issue on Github.
PS: We’re hiring and have exciting positions in all our locations across the Nordics and Poland. Check out our open positions at https://schibsted.com/career/.
