Dialogflow and Beyond
Nu Echo has been using Google Dialogflow for some time now and would like to take a moment to share our thoughts about using the platform for chatbot and intelligent virtual agent projects within enterprise organizations. In this blog post, we will describe what Dialogflow is good for, its current limitations, and how we work around those limitations to get the most out of our development. We will also explore our plans for the future as part of our IVA Solutions practice. This article will solely focus on an engineering perspective to kick off this series but please watch for future posts which will touch on dialogue management and conversational aspects to be considered for great IVA development and more!
What makes it great!
Browsing around Dialogflow’s website, you can easily find the benefits. Some are accurate, others are less evident as true benefits or are lacking as of yet. Let’s start by considering what some of the true benefits are that we have been able to experience and take advantage of:
Quick and easy to start building: Yes! You can truly create your first bot within a matter of minutes. I think my mother did one in, like, 16 minutes, while cooking her famous apple jelly. You login to the console, create an agent project, create one or two intents, and then try it out within the console. Instant reward. A few more clicks to activate the Phone Gateway integration and then you can, yes, call in (assuming your bot is en-US for now). Small dopamine rush.
Built on Google infrastructure: From within a project, you can tap into the rich Google Cloud ecosystem. Conversation logs get pushed over Stackdriver, fulfillment is only a few Google Cloud Functions or Firebase Functions away, security and containment are directly handled as part of IAM, while speech-to-text and text-to-speech relies on Google Cloud respective APIs. And frankly, although we are using some beta features, the platform is quite stable.
Easy to scale: Seriously! This is an additional benefit to the point above. No need to worry about any sort of viral effect your bot might experience. There’s not even a dial or knob to tweak or turn on. It’s essentially all done behind the scenes. Coupled with Google Cloud Functions for serverless fulfillment stage and you’re golden.
Strong natural language understanding (NLU) capabilities: So far so good on this front. We definitely have yet to push the limits but its context-based approach appears to be robust. More on that to come in future posts.
Where it might fall a little short…
Now, on to the not so great part. And yes it is singular so technically this is a good thing. But this one claim is a rather bold one, with several implications.
Good developer experience: For any toy, prototype, or proof-of-concept type of project, the experience with Dialogflow is indeed good enough. Good is however far from excellent and there is room for improvement. For serious projects, the word we would think of to best describe the Developer Experience would be underwhelming. Let’s break it down.
For one, the Console itself is not very collaborative; a team of a few developers easily find themselves stepping on each others toes editing resources. Moreover, collaborative work goes beyond the dialogue implementation itself. It typically implies tasks around testing, deployment and operations. While the Console offers versioning and publishing over environment capabilities (as a Beta feature) it remains a manual task. In an age of everything-as-code and continuous deployment, we’d like to automate the process of publishing any given version over multiple environments (i.e. dev, staging, prod) alongside the fulfillment webhooks functions in a reproducible and consistent fashion.
Since the Console is missing a few key features such as copy/paste or undo/redo, some of the operations remain tedious.
When you think “Developer Experience”, refactoring is always a big thing:
the ability to easily swap intent order or even transform an intent into a follow-up is just not there (yet) and at the end of the day, whenever a developer wants to reshape the dialogue flow, they have to trash the some previous work done, and start over, with some painful dialogue tree navigation. Let’s not even venture into the composability and reusability, which are simply inexistant. On top of that, the Console seems to experience important memory leaks…
When you’re dealing with a simple bot featuring just a few intents with a limited set of follow-ups and standard slot-filling steps, functional testing is less of a concern. On the other hand,
when an agent gains in complexity, you want to make sure adding new intents or follow-ups, changing input/output contexts and such do not break anything else.
This is typically performed through regression testing. Done manually, it is a tedious task. We’d like to see the ability to define and run periodical test scenarios easily, and ideally with some test phrase variations. At this stage, we are not concerned so much about NLU performance, we are more focused on dialogue management performance and results.
Circumvention Kung-Fu
Now, in the face of those engineering and developer experience related limitations, and because we still need to deliver rock-solid IVA solutions to our large enterprise customers, we do what great engineering practices dictate: we rolled up our sleeves and started working on a toolchain to fill in the gaps. Unlike other platforms, Google sort of adopted an API-first approach which serves us well.
We started from humble beginnings, creating our gush command line interface (CLI):
gush: To flow or send out quickly and in large amounts. To express a positive feeling, especially praise […].
For a start, we added support to export/import a Dialogflow agent so we can have intermediary snapshots. Coupled with Google Cloud Scheduler, nightly snapshots are a breeze.
Next, we went on to add support for basic regression testing: given a set of test scenarios featuring a sequence of user input, we can now automate regression testing, essentially leveraging detectIntent from the API. We’re planning on beefing up this critical part of our practice.
Finally we wanted to address composability and reusability, while pretty much leaving aside the Console for anything related to agent creation itself, going through offline intent/entities modeling to avoid the aforementioned limitation. The idea, and this is something we’re actively working on right now, is to have a set of Yaml definition files, containing intents, entities and response definitions, and then have our toolchain perform some integrity checks while pushing those resources over our agent project, again leveraging the web API. In the process, we can automatically deploy serverless fulfillment functions tied to our agent project directly to Google Cloud Functions. And boom! We have a fully managed continuous delivery pipeline offering everything-as-code across the board.
Conclusion
While Dialogflow is a chatbot platform that is going to mature and increase in functionality overtime, in the meantime, it offers just enough power and facilities to create tooling around it so we can deliver customer projects with confidence.
As for the dialogue management itself, which has its own specific set of capabilities and limitations, stay tuned for an upcoming blog post series. Big thanks to my hard working colleagues Guillaume Voisine and Karine Déry, masters of our Dialogflow practice, and the rest of the team for gushing around.
