Chatbot Cache System

Most chatbot systems (including IBM’s Watson Assistant) run on a straight forward API call based interface; you POST a message text and a response is returned. Chatbots are great because they can understand various user utterances to mean the same thing — both “hi” and “hello” can be understood as a GREETING for example. Often enough, for large scale implementations at least, the same message gets sent to the chatbot over and over.

For repeat utterances, why don’t we cache the responses on the application and replay that to the user instead of making additional calls to the chatbot. This makes our application more efficient, faster, and saves us money. Note, most of the time, the API call volume is how chatbots are priced; each additional call may costs some fraction of a penny (but it all adds up, right!). Here’s a quick diagram:

Chatbot Cache System Diagram

We could also try to make this chatbot cache system somewhat smarter. First, chatbots often rely on something called context which helps it understand where in the conversation it is with a user. The cache system would need to ensure that the context variables are the same to serve a cached response instead of going to the chatbot service. Another thing to consider is that chatbots often get updated and the responses change. Remember the famous quote:

There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors. —

To ensure that stale messages (invalid cache) don’t get returned, I think there are a couple things we could do:

1. Direct our cache system to send a real message to the chatbot every X number of times and automatically update if it is off. This won’t ensure that users NEVER get a stale message, but it will slowly roll out updates when they do occur.
2. We could set up a polling system to the chatbot service (depending on which one we use and if it allows it) to check when it was last updated. If a cached message is older than the latest update date, we can force a new call to the chatbot.

Definitely a lot to optimize here and there are a lot of potential issues, but think the general concept makes sense.

Note — the opinions and ideas expressed are my own and not necessarily those of my employer.