Build Your RingCentral Virtual Voicemail Assistant for Business — Part 2

Virtual Voicemail Assistant

In part 1, I explained the voicemail capabilities of the RingCentral cloud communications system, and AI (Artificial Intelligence) solutions that can be employed to build an effective virtual voicemail assistant for your telephone customer services. I also showed you how to create and setup a dedicated extension for taking only voicemail messages, and the overall workflow of a virtual voicemail assistant.

In this article, I will walk through the essential steps to develop a Web app — a demo of virtual voicemail assistant for RingCentral Developers support, which can listen for new voicemail messages and perform the following tasks.

Voicemail Data Analysis

1. Receive voicemail messages
2. Detect spammy voicemails
3. Auto reply to a caller with an SMS message
4. Transcribe voicemail messages
5. Detect urgency of a voicemail
6. Categorize voicemail content
7. Assign a support ticket (voicemail) to a designated support engineer (agent)
8. Let an agent read the voicemail transcript or listen to the original voicemail message
9. Allow agents to easily call back by click-to-dial or send an SMS to the caller

The associated demo application is built using Node JS, Express Web application framework. Thus, for conveniences, I will use the Node JS SDKs provided by RingCentral, Monkey Learn, and Rev AI to access their services. You can easily build this Web app backend in any programming language you like with the client libraries provided by the services providers mentioned above.

In order to build and run the demo app, you will need to setup the following accounts, get their API keys and login credentials:

• A RingCentral developer account. Click here to create a free developer account.
• A MonkeyLearn developer account. Click here to create.
• A Rev.ai developer account. Click here to create.
• A WhitePages Pro (a.k.a Ekata) developer account. Click here to create.

Note: The code snippets shown in this article are shorter and just for illustration. They may not work directly with copy/paste. I recommend you download the entire project from here.

Get new voicemail notification

Let’s get started with the critical function of a voicemail assistant — Listening for new voicemails arriving in the voice mailbox. This feature can be implemented using either PubNub notification or Webhooks notification.

I will use the RingCentral webhooks notification method to subscribe for new voicemail event notification.

var eventFilters = ['/restapi/v1.0/account/~/extension/~/voicemail']
platform.post('/subscription',
  {
    eventFilters: eventFilters,
    deliveryMode: {
      transportType: 'WebHook',
      address: 'https://[c1969441.ngrok.io]/webhookcallback'
      }
  })
  .then(function(response) {
    console.log("Ready to receive voicemail notification.")
})

See the webhooks.js module for complete code. I will skip the explanation on how RingCentral webhooks work, but if you are not familiar with RingCentral webhook notifications, please read this blog to learn more.

Since I expect to run the app on my local machine, I use the ngrok tool to get the callback address [https://c1969441.ngrok.io]. If the event notification subscription was subscribed successfully, I will receive notifications in a post request to the address specified above.

When I receive a notification, I parse the data to get the body JSON object, the ownerId (which is the extension Id of a user who subscribed for the notification) and the subscriptionId. Then I call the processVoicemailNotification() function to process the data. I use the extensionId to find the user who should process the voicemail notification and I use the subscriptionId to verify if it is the event I subscribed for.

app.post('/webhookcallback', function(req, res) {
  if(req.headers.hasOwnProperty("validation-token")) {
    res.setHeader('Validation-Token', req.headers['validation-token']);
    res.statusCode = 200;
    res.end();
  }else{
    var data = []
    req.on('data', function(chunk) {
      data.push(chunk);
    })
    .on('end', function() {
      data = Buffer.concat(data).toString();
      var jsonObj = JSON.parse(data)
      var body = jsonObj.body
      var extensionId = jsonObj.ownerId
      var subscriptionId = jsonObj.subscriptionId
      processVoicemailNotification(body,extensionId,subscriptionId)
    });
  }
})
function processVoicemailNotification(body, extId, subscriptionId){
  var index = getUserIndexByExtensionId(extId)
  if (index < 0)
    return
  if (users[index].getSubscriptionId() == subscriptionId)
    users[index].processVoicemailNotification(body)
  else
    console.log("not my subscription")
}

Let’s move on to parse the body of a voicemail notification to extract some essential data:

{
  "body": {
    id: 1054xxxx,
    from: {
      phoneNumber: '+1650224xxxx',
      location: 'Mountain View, CA'
    },
    type: 'VoiceMail',
    readStatus: 'Unread',
    attachments: [ 
      {
        id: 1054xxxx,
        uri:
'https://media.ringcentral.com/restapi/v1.0/account/~/extension/~/message-store/1054xxxx/content/1054xxxx',
        type: 'AudioRecording',
        contentType: 'audio/mpeg',
        vmDuration: 11
    },{
        id: 2406yyyy,
        uri: 'https://media.ringcentral.com/restapi/v1.0/account/~/extension/~/message-store/1054xxxx/content/2406yyyy',
        type: 'AudioTranscription',
        contentType: 'text/plain',
        vmDuration: 11,
        fileName: 'transcription'
    } ],
    lastModifiedTime: '2019-06-16T22:17:41.584Z',
    vmTranscriptionStatus: 'Completed'
  }
}

I extract the data and keep them in the item object as shown below:

var item = {}
if (body.from.hasOwnProperty("phoneNumber")){
  item['fromNumber'] = body.from.phoneNumber
  if (body.from.hasOwnProperty('name'))
    item['fromName'] = body.from.name
  else
    item['fromName'] = "Unknown"
}else{
  // Just for demo purpose. Use predefined scam numbers
  if (index >= samplePhoneNumber.length)
    index = 0
  item['fromNumber'] = samplePhoneNumber[index]
  item['fromName'] = "Unknown"
  index++
}
item['toNumber'] = body.to[0].phoneNumber
item['toName'] = body.to[0].name
var timestamp = new Date(body.lastModifiedTime).getTime()
item['date'] = timestamp
item['id'] = body.id

I need to detect if the caller’s phone number exists. This is necessary because the “from.phoneNumber” could be missing if a call is anonymous. In this demo, I predefined a list of scam phone numbers and use it for simulating scam calls if a call is anonymous. In a real application, we still can proceed the voicemail analysis for anonymous calls and try to detect if a caller leaves a call back number in the voicemail message.

Identify a caller

Using the caller’s phone number, I make a query to my customer database to find a customer and retrieve the customer’s information such as the first and last name, and the type of phone number (i.e. mobile). In a real application, you can connect to your CRM database and pull any necessary customer information that an agent should be aware of when he or she makes a callback to the customer.

var query = "SELECT first_name, last_name, phone_number_type FROM customers WHERE phone_number='" + phoneNumber + "'"

For the demo purpose, I created a small customer database with just basic customer information below:

customer_id — first_name — last_name — phone_number — phone_number_type

If the caller is a registered customer, I move on to analyze the voicemail message. Otherwise, I will proceed to detect if it is a spammer or not.

Scam voicemail detection

As I mentioned earlier, I use WhitePages service to detect a phone number reputation.

var url = "https://proapi.whitepages.com/3.0/phone_reputation?"
url += "api_key=[WHITEPAGES_PHONE_REPUTATION_APIKEY]";
url += "&phone=[phoneNumber]"
request.get(url, function(err, res, body){
  if(res.statusCode == 200 ){
    // parse the body to get phone reputation info
    var jsonObj = JSON.parse(body)
    var numberInfo = {}
    if (jsonObj.hasOwnProperty("reputation_level"))
      numberInfo['reputation_level'] = jsonObj.reputation_level
    ...
    if (jsonObj.hasOwnProperty("reputation_details"))
      numberInfo['reputation_details'] = jsonObj.reputation_details
    ...
  }
});

See the number_analysis.js module for complete code.

At this point, I should have enough information to identify if a caller is a spammer or not, based on the reputation level. I also classify a voicemail source using the “category” tag from the “reputation_details” returned from WhitePages. And for a registered customer’s voicemail, I use the “Customer” tag.

If the reputation level value is greater than 1, I flag the voicemail as spam with a label (Likely, Highly or Risky) based on the reputation level, and I will stop analyzing the voicemail.

If the reputation level value is 1, I will label the voicemail as “Clean”, and move on to get the voicemail transcript.

Get voicemail transcript

If the voicemail was automatically transcribed by RingCentral, I can access the voicemail transcript via the attachment URI from the notification event body. However, I need to check whether the voicemail transcription status is completed and if the attachment type is “AudioTranscription” before getting the transcript.

if (body.vmTranscriptionStatus == "Completed"){
  for (var attachment of body.attachments){
    if (attachment.type == "AudioTranscription"){
      platform.get(attachment.uri)
        .then(function(res) {
          return res.response().buffer();
        })
        .then(function(buffer) {
          var transcript = buffer.toString()
          item['transcript'] = transcript
        })
      break
    }
  }
}

For testing on the sandbox environment, I use the Rev.ai speech recognition service to transcribe the voicemail. I use my own SDK to call the Rev AI transcription service. However, you can use Rev AI’s official SDKs to access their service. Either way, we must attach a valid access token to the voicemail attachment URL to make it publicly accessible (as long as the access token is valid).

for (var attachment of body.attachments){
  if (attachment.type == "AudioRecording"){
    var vmUri = platform.createUrl(attachment.uri, {addToken: true})
    // Call Rev.ai transcription service API
    transcriptionist.transcribe(vmUrl, function(err, transcript){
      if (err){
        console.log(err)
      }else{
        item['transcript'] = transcript
      ...
     }
    break
  }
}

See the transcription_engine.js module for complete code.

Getting ready for voicemail analysis

I decided to use MonkeyLearn AI services for urgency detection and categorization. It is very easy to train your own dataset with MonkeyLearn’s data model creation tools.

For urgency detection, I used their demo data model for the demo purpose, as it can detect the urgency based on keywords like “as soon as possible”, “as soon as you can” “this is urgent” etc.

For categorization, I trained my own dataset to build a data model for detecting the following predefined categories:

Messaging — Voice — Meeting — Data — Authentication — Configuration — Notification — Integration

First, I collected a couple hundred technical questions from the RingCentral Developers Forum and put them in an Excel sheet and tagged them appropriately, then saved them in a .CSV file. It’s worth noting that, the more data samples you provide, the better your results will be for categorization.

Then I uploaded the file to MonkeyLearn to start the training process.

Below are the main steps to train a data model with MonkeyLearn:

1. Choose a Model Type
2. Select a type of classification
3. Import sample data from a file
4. Build the model
5. Test the model
6. Copy the model Id and use it in your code

const MonkeyLearn = require('monkeylearn')
const ml = new MonkeyLearn(process.env.MONKEYLEARN_APIKEY)
let urgency_model_id = 'cl_Aiu8dfYF'
let categorization_model_id = 'cl_zBbUZ6dU'

Analyze the voicemail message

Now I have the voicemail transcript and my own data model, the next task is to detect the urgency and categorize the voicemail. But before analyzing the voicemail content, I do a quick check to see if I have enough information from the voicemail for analyzing it. I decided to check the word count from the voicemail transcript and set the threshold at 10 words. I also make sure that the voicemail is not a spammy one.

var wordArr = transcript.split(" ")
if (wordArr.length > 10 && reputation_level == 1){
  analyzeVoicemail(transcript)
}

To detect the urgency, I call the MonkeyLearn classifiers API, passing the “urgency_model_id” and the “transcript”, then I parse the API response to extract the urgency status (“Urgent”, ”NotUrgent”), convert the confidence scale from 1 to 10.

let data = [transcript]
ml.classifiers.classify(urgency_model_id, data).then(res => {
  var body = res.body[0]
  var result = {}
  if (body.error == false){
    var classification = body.classifications[0]
    result['status'] = classification.tag_name
    var scaled = (classification.confidence * 10)
    if (classification.tag_name == "Urgent"){
     result['confidence'] = Math.ceil((scaled / 2) + 5)
    }else{
     result['confidence'] = Math.ceil(scaled / 2)
    }
  }else{
   console.log("Error: " + JSON.stringify(body))
  }
})

To categorize the voicemail, I call the MonkeyLearn classifiers API, passing the “categorization_model_id” and the “transcript”, then I parse the API response to read the categories. There could be multiple categories, and each category is associated with a confidence score. I iterate through the “classifications” array and pick the category with the highest confidence score.

let data = [transcript]
ml.classifiers.classify(categorization_model_id, data).then(res => {
  var body = res.body[0]
  var result = null
  if (body.error == false){
    var confidence_score = 0
    for (var item of body.classifications){
      if (item.confidence > confidence_score){
        result = {
          category: item.tag_name,
          confidence: item.confidence
        }
        confidence_score = item.confidence
      }
    }
  }else{
    console.log("Error: " + JSON.stringify(body))
  }
})

See the content_analysis.js module for complete code.

Assign support ticket to a designated support engineer

Suppose there are several support engineers (agents) in a developer support team, I assign their duty based on their technical skills that match the categories I defined above (the assignment can be done from the settings page of this demo app).

var agentName = "Unassigned"
for (agent of settings.assigned_agents){
  for (var cat of agent.category){
    if (cat == result.category){
      var table = "voicemail_" + agent.id
      agentName = agent.name
      item['assigned'] = agent.name
      addSupportCaseToAgentDB(table, item)
      if (item.status == "Urgent" && item.confidence > 6){
        var text = "You have an urgent callback request from "
        text += item['fromNumber']  + "\n"
        text += (item['transcript'].length < 150) ? 
                 item['transcript'] : 
                 item['transcript'].substr(0, 150)
        notifyAgentBySmsMessage(thisUser, agent.id, text)
      }else{
        console.log("Not urgent. No need to alert an agent")
      }
      break;
    }
  }
}

In the code block above, I iterate through the “assigned_agents” list, compare the voicemail category with each of the agent’s assigned category. If there is a match, I assign a support ticket to that agent then add the ticket information to a database. I also send an SMS to notify the agent if the voicemail is detected as urgent.

Auto reply SMS message to a customer

To enhance the customers experience, we can instantly send a reply SMS message to notify the callers that we are working on their questions and will call them back as soon as we can. Of course, we are not going to reply to scammers nor try to send an SMS message if the caller’s phone number is not a mobile phone number.

To send a reply SMS message, all I need is to make sure that the caller’s phone number is a mobile phone number, then compose a relevant message and send an SMS message using RingCentral SMS API.

var text = "Hi, thank you for your voice message! We will get back to you as soon as possible. For your reference, here is your case number 1234567890"
var params = {
  from: { 'phoneNumber': ourServicePhoneNumber },
  to: [{ 'phoneNumber': item['fromNumber'] }],
  text: text
  }
platform.post('/account/~/extension/~/sms', params)
  .then(function (response) {
    console.log("sent SMS")
  })
  .catch(function(e){
    console.log(e.message)
  })

Demo of a Virtual Voicemail Assistant

That’s it for the demo for now — you should be able to get started building your virtual voicemail assistant the way you want and feel free to further develop this app to make it useful for your business.

Learn more about our Developer Program here: https://developer.ringcentral.com/