Where do I begin, lets start with some numbers. Since my last article in December, there has been 114 commits / 9,912 ++ / 6,078. There has been a lot of activity, and I want to highlight some of the key changes between version 0.2.0 and 0.6.0.
New Topic System
Over the Christmas holiday I set out to see what the key differentiator at 2014 Loebner contest was, and what set AIML chatterbots apart from ChatScript chatterbots. All signs pointed to how the conversation flowed within the topic system, and how the bot was able to recover when the conversation drifted off topic.
Prior to 0.2.0 SuperScript would only stay in the assigned topic until it was explicitly told to change — like AIML. This allowed for a very ridged conversation flow and not much room for organic exchanges, furthermore, once SuperScript found a match of any kind, it would return back with the results.
I’m excited to share that SuperScript has a brand new topic system. It will flow from topic to topic based on the input provided, the previous matched topic AND fuzzy keyword matching using TF-IDF search algorithm. Along with this change, we will continue to process matches until there is actually something to return back to the user. This means we can have gambits that match and process input in different ways — more on this later.
Conditionals and Filters
One of the key power features in SuperScript are the plugins. Plugins are JavaScript functions written in Node.js and are asynchronous by default allowing for more extended behaviour like calling remote API’s, reading or writing to Databases or doing additional processing.
Plugins are called out from the reply line like this:
+ when i see this
- call this ^customFunction()This would lookup customFunction and process the results, then inline them back into the reply. However, what if we want the entire reply to be considered based on a condition. Here is were filter functions shine.
+ when I see this
- {^someTruthFunction()} Say this
- {^someTruthFunction()} Say thatIn this example the someTruthFunction is evaluated when we are making the decision to pick a reply. Lets see a real example based on one of the Loebner questions.
+ my name is <name>
- {^hasName(false)} ^save(name,<cap1>) Nice to meet you, <cap1>.
- {^hasName(true)} I know, you already told me your name.hasName checks to see if we know the users name, and if we don’t, (false) then we say “Nice to meet you <name>” and we save it using the save function. Then when we hit this trigger again, hasName(true) will be filtered in, and reply with a cheeky remark. “I know, you already told me your name.”
Filter functions are not just for replies, and they can be applied to triggers too, and I see extending this to topics as well, but we can save that for another release.
+ {^not(fish)} I like *
- Thats great.The syntax is identical to filters in replies, this function not(…) will return false if the message contains the word fish.
Persisted User State
SuperScript supports having multiple conversations with multiple people at once, however it didn’t fully persist the conversation to disk after the person logged off, or when the client was not running. This was added in 0.4.0, and was great, conversions were finally able to continue where they left off without the bot having to re-learn who I was.
This was a key change to the system but came with an added dependancy of MongoDB for persisting user data, which would be later exploited further in 0.6.0.
Out of bound properties
In 0.5.0 we introduced extra metadata in the reply. Up into this point SuperScript would always return a string to the user (or client). However this seemed limiting, and was one of the last things carried over from RiveScript.
Lets say you wanted a bot to be able to communicate with a user by slack and SMS. Sure you could simply reply on the medium you received the message on, but what if the message itself contained a clue to change the communication medium.
+ can you (txt|sms) me the address
- {^hasNumber(true)} ^addMessageProp(medium,txt) 34345 Old Yale Rd.
- {^hasNumber(false)} I need your number first.The resulting reply object look something like this:
{
string: "34345 Old Yale Rd.",
medium: "txt",
createdAt: "..."
}Improved Authoring
My plan was to get an example bot in the wild for people to play with and really showcase what exactly SuperScript is capable of. However as my example bot kept getting bigger, the resulting workflow to test gambits and triggers was painfully slow.
A bot with 1,200 triggers and 3,000 replies took 2 minutes to compile the data file. This process was deliberately ripped out if the core application to keep the startup time fast however it took a toll on the overall authoring experience. I was able to add checksums checking to make sure we didn’t need to re-compile files that were not changed, but ultimately this was only a half-measure.
The Topic System was still very much read-only and authoring experience felt very disconnected.
In order to create a unique authoring experience we needed to re-think everything.
I wanted to tweak the conversation in real time, pull the marionette strings and see the bot dance and spring to life. This change required throwing out the parser and the concept of editing files entirely, and was a radical change from how other systems work.
Thanks to the introduction of MongoDB, I was able to re-write the Topic System (again) to make use of persisting the topics, replies and triggers. This meant, I was able to expose a Topics API and build a web interface to directly manipulate it.
I can now jump back and forth between what was said to the user and understand why.
The resulting web interface should provide authors with the tools they need to fully visualize and test conversions, as they happen and tweak them along the way.
Another side effect of this change is that topics can be authored by people OR the bot itself. I will let you think about that for a moment.
The editor is available at https://github.com/silentrob/superscript-editor and comes with SuperScript baked in, so there is no need to download both.
Road to 1.0
My plan is to continue to support both the editor and dialogue engine as well as expose the remaining SuperScript features. This includes the graph knowledge base, fact system, and bot facts as well as user and message objects, and robust plugin system.
The remaining work will be providing example bots and better documentation as well as explore commercial support.
For more information check out superscriptjs.com and follow me on twitter @rob_ellis and @superscriptjs
