Comparing Language Understanding for Indian Languages on Dialogflow and NeuralSpace

Introduction

We, at NeuralSpace, are back with another comparative study. This time with Google Cloud’s Dialogflow ES for Language Understanding with automatic model training.

In Part 1, we performed a similar study for AutoNLP Entity Extraction services of Hugging Face and AWS Comprehend. Check out the article for more details.

In Part 2 we assessed the NeuralSpace and Azure Cognitive Services AutoNLP platforms comprehensively for entity extraction in a multilingual setting. In this part, we had also attempted to benchmark Google Cloud’s Dialogflow ES service for the same but had decided against it due to user experience constraints associated with the service (more on this later). However, since Dialogflow ES is a popular platform, we at NeuralSpace, decided to get on top of these issues and perform a thorough analysis.

In this article, we evaluate the Language Understanding services of Google Cloud’s Dialogflow ES and NeuralSpace, exclusively for Indian languages. For this evaluation, the Amazon MASSIVE dataset has been used. Check out our previous blog on Evaluating NeuralSpace on the Amazon MASSIVE Dataset.

Benchmarking Results

We wouldn’t like to keep you waiting for the results so let’s get right into it! The table below shows the intent accuracy of Google’s Dialogflow ES and NeuralSpace’s Language Understanding app, on the Amazon MASSIVE dataset.

Since the release of the Amazon MASSIVE blog post, NeuralSpace has worked on improving the architecture of the Language Understanding app. This has led to a further improvement in language-wise performance. For this reason, we compare Dialogflow with the current as well as the previous benchmark results on NeuralSpace. For our results, we tried to emulate how a customer would use our platform. So, we trained 5 models for each language and picked the best one.

A quick note on intent accuracy for the uninitiated:

The higher the accuracy, the better!

As can be seen above, NeuralSpace performs on average 9.6% better than Dialogflow across all Indian languages at identifying user intents.

But these results barely scratch the surface of the actual differences between the two platforms, both feature-wise and usage-wise. Setting up and then using Dialogflow was not super intuitive and there were quite some hoops that we had to pass through.

Feature Comparison

Let’s take a detailed look at Dialogflow’s features (or a lack thereof) and compare them with NeuralSpace.

Entity recognition

Right off the bat, you must have noticed that we did not have strict and partial F1 scores in our comparison table above. The reason for that is, entity tagging (where we can input an example sentence, and get the sentence with the entities tagged automatically as output) is not available on Dialogflow. When providing the training examples, the entities can be annotated. But while querying for a test example, only the intent can be obtained via their APIs and the UI.

We, at NeuralSpace though, have support for entity tagging and it can be achieved rather simply by just training the data with the entities annotated.

Trainable entities

We suspect that the entities on Dialogflow are not trainable, and use only regex and synonym-based lookups. There are a set of pre-trained entities available though, but that is about it. But due to a lack of explanation in the documentation, this inference is inconclusive.

The NeuralSpace platform supports proper entity recognition including training as well as tagging of entities.

Import dataset

One of NeuralSpace’s highlight features that we would like to keep reminding you of is ‘Click, Train, Chill’! In achieving this, our Import Dataset feature plays a key role.

Import dataset feature on the NeuralSpace Platform.

The Import feature allows you to directly import popular datasets into your projects using a no-code way. Without having to convert the data yourself into the required formats, you can easily import data at the click of a single button! The Amazon MASSIVE dataset, on which we have run this benchmarking comparison, has been available to import to NeuralSpace’s Data Studio since almost as soon as it became public. That’s one extra step less for you! A similar feature is not available on Dialogflow, though.

Language Understanding dataset upload page on NeuralSpace Platform.

But even if you have to manually upload a dataset to the NeuralSpace Platform, doing that is not very difficult either. You could upload in multiple file formats and we have even provided examples for all of them that you could refer to!

Supporting Docs and other materials

Language Understanding Docs page on the NeuralSpace Platform.

NeuralSpace has easy to understand documentation that is also very beginner friendly. We have provided quickstarts to get you started in just a few minutes with usage examples. We have also provided Colab Notebooks and Postman Collection where you can try things out without even having to do anything on your local machine. On top of that, we have made a lot of quick and handy tutorial videos to help you navigate around and get things done! We believe that all of this makes the NeuralSpace Platform a lot easier to use compared to Dialogflow.

Chatbots

While NeuralSpace as an NLP platform provides state-of-the-art NLU services which would be the brains of your chatbot (or any other app that you so desire), you would have to use a framework like Rasa (which is open-source) that integrates seamlessly with our APIs. Dialogflow on the other hand though, is the complete solution to build a chatbot, even if with not so smart a brain. 😉 Though, the training time on Dialogflow for NLU is very low, at just a few seconds.

Language Support

Dialogflow UI and UX hurdles

We have saved the best (or in this case the worst?) for the last. We would not shy away from saying that Dialogflow does not really have the most intuitive or the most beginner-friendly UI. The UX story is also similar, with any developer requiring quite some experience with the platform to be able to use it. We have categorised all the pains of using Dialogflow into the following points (there are quite a few, which is why we have had to delay this blog by so much):

Docs

Dialogflow docs page with its multitude of options for just one section and one tab.

Let’s start with the docs, where any developer would first start with as well. To put it simply, the docs are complicated. There are 3 sections in the docs, one that is common for both Dialogflow ES and Dialogflow CX, one only for ES, and one only for CX. But there is quite some overlap between all these three sections. There are multiple tabs in each, and multiple sub-sections and pages that one can navigate to from the sidebar. To get a hold of even the structure of the documentation requires deciphering and is a task in itself. If that wasn’t all, weirdly enough, navigating to certain tabs makes the docs switch sections automatically!

File upload

After having all of that figured out and going through the setup steps, the quickstarts page shows what the different elements (like entities, intents, etc.) are, and how to use the UI to add examples. Thus, we tried using the UI. But of course, manually creating each example was not a viable option. In the UI, there was an option to export the current training data and to import from a zip file as well. We converted all the required data in a format similar to the export. And then imported the zip with all the training examples, intents and entities. Unfortunately, we were only able to get the intents and the examples for them but not the entities using this method.

Upload data on Dialogflow via the UI.

There was an entity file upload option on a different page which wanted a separate csv each for all the entity types. But even after uploading all of them we realised that the annotations were missing in the training examples, and the only way to fix them was doing them manually.

Upload entities on Dialogflow via the UI.

Using the APIs

After all this trial and error, we had to reach back again to the docs for a proper solution and a way to upload all the training examples in the correct format. Under the ‘Reference’ tab was the documentation for all the APIs, which we then tried to implement next. But these were no less complex either. For some APIs, to even figure out the required format of the request body json, we had to go into upto 6 interlinked pages for each API! After collecting information from all of those pages, we could finally create our own sample example of the API request body, since none were provided.

The docs page for the batch update Intents API.

It took a long time for us to figure out that IntentBatch is clickable and that it goes 4 pages deep to arrive at the innermost object.

We then converted all of the data in these required formats, which were a bit complex and quite unnecessarily so in our opinion.

An example snippet of an API request body. Notice how many layers are there and how each example has to be broken down into multiple parts for annotation.

Authentication woes

Using the API was not so straightforward as having the POST/GET request and having created the request body. There were quite a few steps after that:

• The APIs first had to be enabled on a project basis from the APIs library page.
• Then a key had to be generated from the APIs and Services page.
• A service account that had the proper permissions to use the APIs had to be created.
• Its keys had to be generated, downloaded and set as environment variable from the IAM and Admin page.

Now, all of these happen from different parts of the Google Cloud console which gets quite messy and you are very likely to get lost unless you are already familiar with it.

A lack of examples caused more trouble here as we had a hard time figuring out how to pass authentication steps while calling the API. It turned out that there was an API explorer on the API page’s right-hand side, which only when made full screen, displayed a cURL request. Using this we were finally able to figure things out and get the API running.

All of these things above make Dialogflow as a platform almost unusable for a new user unless they are ready to invest quite some time and effort into it. The only redeeming part is that everything is in some way or the other available in the docs. But you need to be very persistent and keep doing a lot of trial and error, to eventually get things running.

Conclusion

We at NeuralSpace, believe that technology, and especially the supporting documentation to use it, doesn’t need to be this complex. NeuralSpace’s platform is built around the concept of “no-code” at its core. We strive to make things super easy for even someone not from a programming background to understand and use our services. We hope to have been able to convey our commitment to this by the comparisons drawn above.

NeuralSpace also performs quite better for all Indian languages if you were to go by pure model performance, performing on an average 9.6% better.

We hope that this blog has provided you with insights and will help you to choose the best language-agnostic NLU service for your specific use case.

At NeuralSpace, we will be happy to connect with you if you would like a demo or have any questions or feedback to help us improve our product. We aim to provide a powerful resource to accelerate your pipelines and empower the next billion users to use the Internet in the language and mode of their choice. Together we can contribute to the engineering and research community.

Try out NeuralSpace’s Language Understanding service now.

Check out our Documentation to read more about the NeuralSpace Platform and its different Apps.

Join the NeuralSpace Slack Community to connect with us. Also, receive updates and discuss topics in NLP for low-resource languages with fellow developers and researchers.