What’s in your bot stack? Survey results of people building bots today

There have been continuous releases of various APIs, frameworks, and other tools to help developers build and scale bots over the past two years. From Facebook acquiring Wit.ai in January 2015 and the Messenger Platform announcement in April 2016, to Microsoft announcing Project Oxford and their Cognitive Services APIs in May 2015 and their Bot Framework in March 2016, to smaller companies launching frameworks and language tools—like Howdy, Smooch, and Init.ai — the possible components of the ‘bot stack’ are ever increasing.

As the space becomes popularized with new options for developers, the on-going questions I’m hearing are:

What should I use to build?

What’s best for my use-case?

What’s NOT working right now?

Questions like these are being asked and answered by the early (and amazingly supportive) bot community of developers, designers, and founders.

I’m lucky to know a lot of smart people working on interesting bots, and wanted to get an understanding of the decisions that went into how they chose the pieces of their bot’s stack. In keeping with the norms of openness and transparency that keep Bay Area tech humming along, I decided to reach out a bit more broadly, with the intention of sharing the results publicly. This post does just that — a summary of how people are building bots today, and what went into those decisions.

The survey and who answered it

I first focused on funded bot startups. Sar Haribhakti helped by sharing the survey with the companies in the betaworks Botcamp program (thanks, Sar!). Meanwhile, I reached out to people I know that are founders of bot startups.

Since the space is still so young and there aren’t many funded companies, there were only 13 respondents :( I then shared the survey in the Chatbot Facebook group and the Botmakers Slack group. That led to a total of 40 responses, collected between August 19th to August 29th. Each of the questions allowed for multiple responses, as I’d seen that many people were using multiple tools or frameworks that had some overlap.

Survey results

Frameworks

Similar to mobile and web apps, bot frameworks help developers build out some of the core pieces of their bot without reinventing the wheel. Bot frameworks can do a lot — they can receive/interpret/reply to messages, format responses for various platforms, integrate cards or other UI elements into your chat, connect to NLP/NLU or other language services/APIs, manage testing/debugging, track analytics, and have other built-in tools to cover many common requirements that bots and their builders need.

The vast majority of responses pointed to people not using any of the existing frameworks, and going custom instead. Microsoft’s Bot Framework was the second most popular choice, with people citing a more fully serviced option as why they chose it.

An interesting narrative emerged in the follow-up question of “Why did you choose that framework solution?”. A few of the respondents who built a custom solution mentioned that when they were first getting started, none of these frameworks existed — and so they had to build their own. Other reasons cited for building custom included flexibility, control, speed, and unreliability of the current options.

Language tools

Since bots interact via language, using NLP/NLU and AI tools is a required piece in the vast majority of bots being built today. Some of the simpler bot are using basic scripts or regular expression matching, but many products are taking advantage of the complex language tools that bigger tech companies have made accessible.

Facebook’s Wit.ai (via a 2015 acquisition) was the most popular, with 11 respondents using it in their current product. The second runner-up was custom, with many bot builders choosing to build out their own language processing tools. When choosing to build in-house, respondents replied that they didn’t feel the existing solutions were good enough, and wanted more flexibility. Overall — ease of integration, flexibility, and completeness were cited as reasons why these various language tools were chosen.

Missing from the responses was IBM’s Watson, released in the fall of 2015 (which I found surprising!). This piece of the bot stack also has more representation from small open-source projects (i.e. NLP Compromise, spaCy).

Platforms

Another question in the survey asked which platforms bots are being deployed on.

The initial funded audience that completed the survey is majority US-based, while the location of the broader group isn’t as clear. However, notoriously US-centric Messenger was the most popular platform amongst respondents, with Slack coming in a fairly close second place. As for the ‘Other’ category, they included Spark, Email, Line, Hangouts, Unity Game Engine, Twitter, and iMessage.

Suggestions — What else works (and doesn’t)

While Wit.ai was the most popular response NLP/AI used, Wit.ai also received the most responses to the “Did you test out any tools, but they were so bad you abandoned them?” question. Respondents felt it was simply too difficult to use, awkward, and hard to build on top of.

In terms of other problems, respondents cited having trouble with AWS Lambda, LUIS, and TensorFlow, commenting that the services were too complicated and/or too difficult to debug. Two respondents said they’d used Howdy’s Botkit, but were moving away from it — one due to scaling issues, the other because it didn’t fit in their new architecture.

Crafting the actual responses that a bot sends and modeling conversation flows was another activity where there seems to be a lack of good tools. Twine was mentioned, as was AIML — but neither of those respondents were happy with the experience of either product.

So…where does that leave us?

The results of the survey were far less definitive than I’d expected. The answers to the various forms of “I want to build Bot Type X, what should I use” are very much TBD at the moment. There’s also a fair amount of bias in the responses collected at this point, since so many of the tools simply didn’t exist when these companies were first started and were making these decisions. Some of those people believed that if they had existed when their company had first started, they may not have chosen to build everything custom.

The survey results also show that there’s a lot of flexibility if you’re currently getting started building bots. The frameworks and language tools accessible to anyone with an interest in spinning up a bot on one (or many) platforms is incredible, considering that most of the major advances and releases have been in the past 18mo. I’m personally looking forward to what the next 18mo hold for the development of the tools and services available to bot startups, and how the lessons from these early entrants will inform both the existing players’ products and new entrants too.