Retrieving conversation history from Dialogflow

What is Dialogflow?

According to the official documentation by Dialogflow,

Dialogflow is a natural language understanding platform that makes it easy to design and integrate a conversational user interface into your mobile app, web application, device, bot, interactive voice response system, and so on. Using Dialogflow, you can provide new and engaging ways for users to interact with your product.

Dialogflow can analyze multiple types of input from your customers, including text or audio inputs (like from a phone or voice recording). It can also respond to your customers in a couple of ways, either through text or with synthetic speech.

Hey, if you haven’t trained your own chatbot on Dialogflow, please visit their documentation to get started. It’s easy and fun!!

If you have integrated a Dialogflow chatbot to your website, mobile app or social media page and you want to retrieve the conversation history, so that you can perform some kind of analysis and understand your customers or users, this is for you!!

Dialogflow Analytics

Dialogflow consists an analytics page which gives you insight into how well your agent is performing, so you can work to further improve the user experience you’re providing. It shows two types of data related to the agent and the conversations it’s been a part of:

• Usage data: Number of sessions and queries per session.
• NLU data: Most frequently used intents and exit percentages.

Clicking on Analytics in the left side menu will take you to UI Dashboard. Here, you can review statistics relevant to the specific agent.

Logging in Dialogflow

Dialogflow does not provide any API to retrieve conversation history directly. You will have to log conversations to Google Cloud and retrieve from there. To enable logging in Dialogflow go to chatbot settings by clicking the ⚙️ icon on the left side menu of the console and under LOG SETTINGS enable Log interactions to Google Cloud. This will write Dialogflow logs to Google Stackdriver. Clicking on the project id under GOOGLE PROJECT will take you to Google Cloud console. In the cloud console go toLogs View .

This will show you the logs from Dialogflow and you can filter results by running your queries.

Exporting logs to local machine

Now your Dialogflow logs are in Google Stackdriver. You can either store these logs in cloud itself or export the logs to a third party server. To enable this you will have to configure a Log Router. According to Google cloud documentation,

All logs, including audit logs, platform logs, and user logs, are sent to the Cloud Logging API where they pass through the Logs Router. The Logs Router checks each log entry against existing rules to determine which log entries to discard, which log entries to ingest (store) in Cloud Logging, and which log entries to export using log sinks.

The routing of log entries is illustrated in the following figure:

Each log entry that is received by Cloud Logging is compared against the exclusion filters and the inclusion filters. These comparisons are independent.

To determine if a log entry is exported to a destination, the log entry is compared to the log sink query of an inclusion filter. When a match occurs, the log entry is exported to the sink destination. A log entry might match multiple inclusion filters.

To determine if a log entry is discarded or saved in Cloud Logging storage, the log entry is compared to the exclusion filters. If it matches any exclusion filter, the log entry is discarded. Otherwise, the log entry is saved in Cloud Logging storage.

For further information visit the documentation page.

To have more control over the logs, you will need them in files or databases in your local environment. Therefore, you’ll have to route them to Google Cloud Pub/Sub. Click on Logs Router -> CREATE SINK .

In the next view, give a name to your Sink and select Pub/Sub as the Sink Service. In the Sink Destination dropdown select Create new Cloud Pub/Sub topic. Give a name to your Pub/Sub topic and click Create. Then click on Create Sink. This will route your logs to Cloud Pub/Sub.

From the cloud console menu select Pub/Sub -> Topics .

You can see that your topic has been created. The topic I created here is chitti-logs. To pull the logs to your local environment, you’ll need a subscription for your topic. Let’s create a subscription. Click on ⋮ next to the topic and select Create subscription. Give a name to your subscription, leave everything else as default for now and select Create. Your logs are ready to be to be pulled to your local machine. But before that, follow the tutorials here and setup Google Cloud SDK in your machine.

In your command line run gcloud init to initialize the SDK.

./google-cloud-sdk/bin/gcloud init

Once everything is setup, the following command will print your logs in the command line or terminal. Here, chitti-sub is my Pub/Sub subscription id.
gcloud pubsub subscriptions pull --auto-ack chitti-sub

The purpose of exporting Dialogflow logs to your local machine was to perform some kind of analysis and understand your customers or users. In my case, I wanted to develop a dashboard for my chatbots, so that I can view all my analyzed information in one place. To achieve this, I had to write the Pub/Sub messages to Elasticsearch. I’ll be explaining how I implemented this with Spring Boot GCP in the next article. Stay tuned…