Testing and the Future of NinjaChat

This article is part four (and the last part!) in a series of articles (#1, #2, #3, #4) about how we set up a chatbot — NinjaChat — to serve our shippers and consignees here at Ninja Van. This final section talks briefly about the testing setup for our chatbot, and the new features in the pipeline for NinjaChat.

A Brief Note on Testing our Integration

Building something entirely from scratch means we also need assurances that it would work as intended when we finally deploy. This is especially so for an application based on a conversation tree where we should strive to cover as many possible pathways as possible. As such, we wanted a good foundation that covers testing the application at different levels.

• Unit Testing

One advantage of writing our intent service classes such that each matched intent carries an action value which executes a fixed handler method, is that writing units for them involves testing solely the handler method and expecting a given output in the form of a DialogflowQueryResponseBean. Here’s an example that tests for an error response when one of our internal services throw an exception:

@Test
public void givenNoOrders_whenHandleActionY_thenProcessAction() {
    //Arranging the error from an internal api call
    when(orderApi.searchForOrders(/* ... */))
        .thenReturn(completedFuture(Optional.empty()));
//Asserting on the response as an error bean
    assertErrorBean(intentService.processMatchedIntent(
        buildCustomerRequest(DEFAULT_SYSTEM_ID),
        buildQueryResult(CUSTOMER_FETCH_ORDERS_FOR_ISSUES_SEARCH)),
        DF_FETCH_ORDERS_FAILURE.getKey()));
}

*Note: buildQueryResult() mocks a matched intent result from Dialogflow with the given action CUSTOMER_FETCH_ORDERS_FOR_ISSUES_SEARCH. DF_FETCH_ORDERS_FAILURE refers to the error message key shown to the user.

• Dialogflow Integration Testing

Naturally, once we’ve made sure that the individual handlers for each matched intent action behave as expected (i.e. returns the right response to the user, makes proper api calls, and parses responses from intermediate calls appropriately) via unit tests, we also use an inhouse framework that allows us to test the chatbot experience from a more integrated, user-relevant perspective.

What this means is we run integration tests that tests on the level of the main Dialogflow service class. This class is responsible for accepting a query string from a given user, converting it to a Dialogflow request bean, processing it and churning out a Dialogflow response bean, and finally returning the response as per what the user expects to see on any given platform.

Here’s a sample of one such test:

@Test
public void givenTrackOrder_whenUserChatsWithBot_thenProcessConversation() {
    AbstractModule customModule = configureCustomerProfileModule(
        new DialogflowCustomerIntegrationTestProfile()
            .buildDefaultProfile());
DialogflowConversationContainer con = 
        newConversation(customModule, CUSTOMER);
con.assertThat(whenUserSays("hello there")
        .thenBotRepliesWith("Thank you for using Ninja Van!" 
            + "How can I help you today?")
        .andShowsOptions("Track Orders", 
            "Reschedule Delivery", "Other Options"));
con.assertThat(whenUserSays("i want to track my order")
        .thenBotRepliesWith("Here is a list of your orders...")
        .andShowsOptions("NVSGD123456", 
            "NVSGD135790", "NVSGD246810", "Return to Main Menu"));
/* ... and so on */
}

This test covers a series of interaction starting from the user saying hello to our bot, to tracking their order and finally returning back to the main menu.

Stuff to note:

• We created a class DialogflowCustomerIntegrationTestProfile that is initialized for each test with details about the customer, including the number of orders they have and their details, addresses, etc. This data is then used in mocked responses returned from external api calls.
• DialogflowConversationContainer is a container for this service class that abstracts away the setup of the service class, and involves setting environment variables, injecting the dependencies into the service being tested, etc, via a custom module.
• The container also has helper methods that allows us to process assertions, generated by whenUserSays(…).thenBotRepliesWith(…).andShowsOptions(…) which returns a DialogflowConversationTurn object. By processing assertions I mean building the request bean, hitting the actual Dialogflow url for detecting intent matches with that request, and asserting on the parsed response.

• End to End Testing

Lastly, we engaged our QA team to write end-to-end automated tests via a mock endpoint that makes actual calls to other internal services, and returns the actual HTTP request that is sent to a given platform as the response. This way, we can guarantee that the behavior of our chatbot running with dependencies on other services are more reliably covered.

Simply put, these tests involved firing requests that contain a String query and a given platform (e.g. Messenger), and expecting that the response is consistent with the payload SNS sends to Messenger.

*Oh as an interesting sidenote, we’ve also added an intent that matches a secret code, which when uttered to the bot would export the agent configured for a given environment (e.g. local, qa, etc.), and display statistics such as how many intents there are, how many are lacking training phrases in different languages, the average number of training phrases provided per language per intent, etc. Howver, the actual utility of this feature at the moment is still undetermined.

What’s In Store for the Future

Looking ahead, our team plans to use Dialogflow as the bridge between product and tech, as a space where both product and tech can specify the requirements and structure of new and modified flows.

Given that Dialogflow already has an easy to use and intuitive (for the most part) user interface in its console, it perhaps bears no significance noting that with a little getting used to, product designers can freely work on new prototypes by creating these intents directly. This would involve a workflow that revolves around them approaching tech only when new actions, parameters or message keys need to be handled on the backend.

In fact, it requires close to no developmental effort to simply insert a new intent that reuses an existing action, say CONFIRM_SELECTION, between two existing intents. SNS would continue to process the new matched intent blissfully unaware of this additional step in our flow. And deployment to different environments involve merely updating said agent on other environments.

We are currently already experimenting with steps to facilitate this transfer of (or at the very least, sharing of) ownership of Dialogflow with product designers by allowing the display of response options to be configured via the intent itself. By appending a display type to the end of an action value for an intent, it would override the display type set for the response on the backend.

For example, any intent with action CONFIRM_SELECTION?MENU would be showing ‘Yes’/’No’ options as part of the response to Messenger users as menu (vertical) buttons. Any intent with action CONFIRM_SELECTION?FIXED would show the same options as quick replies.

Conclusion

Of course, this is the beginning of what we’re exploring as possible avenues for managing the bot flexibly via Dialogflow.

For possible suggestions on how Dialogflow can be used more effectively (given what you’ve read so far), feel free to reach out to anyone on our team — myself, Amir Ariffin or Mani Kuramboyina. Admittedly, what we’ve touched on do not come close to making use of the full set of features available on Dialogflow, and this initial integration will continue to be revised and improved upon as we expand and introduce more features to the bot.

I would also like to thank the following individuals for their contribution to this document, our journey with Dialogflow, and of course providing general mentorship and guidance: Amir, Mani, Shaun, Edison Mok (Google) and Han Wen Kam (Google).

Thanks for reading!