Advanced chatbots using Dialogflow, Sentiment Analysis, Telephone Gateway and Firestore

Demo displayed at Next 18 in Japan

Calling a chatbot from your phone, is that even possible?

And the answer is yes!

Jon Skeet and I presented a mentor search demo in C# and node.js, respectively.

If you give our conversational app a call, you would be able to talk to the conversational app via phone, as it is integrated to Dialogflow Phone Gateway Beta.

A bit more context about the demo — you are able to search the mentor by name, location or piece of technology you’d need mentorship.

Sentiment Analysis

Ok, so you are in a call with our bot, and in case you say a negative query, the conversation can be escalated to a real human being who can talk to you real time, leveraging both sides of AI and Machine Learning in one computer-human interaction platform.

Dialogflow Enterprise Edition has a sentiment analysis beta feature, that we use the sentimentAnalysisResult field to check the score and the magnitude of the queries.

  • score - is a number from -1.0 to 1.0 that shows how positive or negative the statement is, the overall emotional leaning of the text.
  • magnitude - is a number ranging from 0 to infinity, it represents the weight of the statement.

Making Phone calls

How we have done it:

  • We have bought a US phone number to call the dialogflow agent.
  • The “transfer calls” are forwarded to our own numbers.
Demo at Next 18 in San Francisco

Telephone Gateway

In terms of telephone gateway, I have been excited to try and test this new feature — it provides a telephone interface to your agent.

I’ve done a number of tests using the simulator but before this demo, not that many real calls, as of now (January 2019) it is only available for US phone numbers

So I bought a US phone number and from that moment onwards I began to see errors that could not be seen in test mode

It was a challenge to come up with a solution to make our demo work for a real-life use case, as the resources and questions answered online are really scarce at the moment.

Demo at Cloud Next 18

Found a bug in production 🚫

I’ve managed to get the functionality working that I could not find the answer anywhere else.

The error was the following: when creating the workflow of the conversation, in the Dialogflow console both in the default text or the telephony gateway tab we could see the conversation flow working neatly.

However, when you did make a real phone call it would act very glitchy.

It would sometimes ignore the message being said, even if we had correctly specified it on the web browser framework response under “SSML or Text response” and the Telephony tab synthesised responses and commands.

So instead of using the web browser functionalities or even the workaround of adding a “setFollowupIntent” and adding an event — we can write the JSON in full with the texts expected in the platform and whether it is voice or text.

As you can see on the code snippet below.

response.json({
"fulfillmentMessages": 
[
  {"text": {
      "text": [
         "I am sorry you feel this way, let me transfer you to a real      person!"
              ]
}},
  { "platform": "TELEPHONY",
    "telephonySynthesizeSpeech": {
      "text": "I am sorry you feel this way, let me transfer you to a   real person!"}
},
  {
    "platform": "TELEPHONY",
    "telephonyTransferCall": {
      "phoneNumber": "<<ADD_THE_PHONE_NUMBER_HERE>>"
    }
  },]});

Jon’s fulfillment in .NET on github:

My fulfilment in node.js on Github:

More specifically the index.js file

The node.js demo available on the browser

Setting up your dialogflow agent:

Intents

Default Welcome Intent

Default Fallback Intent

Mentor Search

Training phrases
Actions and Parameters
Default Tab
Telephony Tab

Transfer Call Escalation to Human

Goodbye

Entity

@ topic

“VR”,”VR”
“Advertising”,”Advertising”
“angular”,”angular”,”Angular JS”
“GCP”,”GCP”,”Google Cloud”,”Google Cloud Platform”
“Java”,”Java”
“firebase”,”firebase”
“Blockchain”,”Blockchain”
“Data analytics”,”Data analytics”,”data analyst”
“python”,”python”
“public speaking”,”public speaking”
“Actions on Google”,”Actions on Google”,”AoG”,”Google Actions”,”Google Assistant”,”NLP”,”Conversational apps”,”conversation design”
“Business intelligence”,”Business intelligence”
“PHP”,”PHP”
“Play Store Optimization”,”Play Store Optimization”
“SQL”,”SQL”
“C++”,”C++”
“app script”,”app script”
“Android”,”Android”
“Javascript”,”Javascript”
“Agile”,”Agile”,”scrum”
“Kotlin”,”Kotlin”
“data privacy”,”data privacy”
“CSS”,”CSS”,”CSS 3"
“React”,”React”
“AI”,”AI”,”machine learning”,”TensorFlow”,”artificial intelligence”
“AR”,”AR”,”augmented reality”
“Backend”,”Backend”
“developer relations”,”developer relations”,”developer advocate”
“Product manager”,”Product manager”,”product managment”
“security”,”security”
“XML”,”XML”
“design”,”design”,”graphic design”
“objective C”,”objective C”
“Usability”,”Usability”
“UX”,”UX”
“Community Management”,”Community Management”
“Community managment”,”Community managment”
“data mining”,”data mining”
“UI”,”UI”
“Marketing”,”Marketing”
“Systems planning”,”Systems planning”
“design thinking”,”design thinking”
“event managment”,”event managment”
“game design”,”game design”
“HTML5”,”HTML5",”HTML”
“network security”,”network security”
“online marketing”,”online marketing”
“Social media”,”Social media”
“Front end developement”,”Front end developement”
“Students in Bootcamps”,”Students in Bootcamps”

Firebase Cloud Firestore Beta Database

We're using the very famous Tracy Lee | ladyleet’s #fempire list, she kindly allowed us to use.

All the kudos to the software developer and Google Developer Expert to Jason Berryman for pairing with me late at night to make the conversion from firebase real-time database to Firestore. If you’re interested in getting started with Firestore and Dialogflow you should definitely check Mandy Chan’s article.

Screenshot of the database “mentors” containing the mentors records

Was this article useful to you? Let me know in the comments!

Thank you for reading this, I hope it encourages you to create your own agent! I try to be active on Stack Overflow answering questions regarding actions on google and dialogflow, you can find my profile here.

To me personally, this has been a very exciting journey as I’ve had the opportunity to create something that has potential to be very meaningful in real life, and definitely doing it collaboratively with Jon, who has kindly mentored me in the past months, it makes it worth for me to want to pass it forward so that other people can believe more in themselves while being guided.