Enterprise Bot Experimentation: How Talla Thought About Building Out Interactive Messaging in Chat

Since founding Talla, our B2B AI assistant, in 2015, we’ve learned a huge amount about taking an AI-powered product to market. We sit at the intersection of a few new spaces: artificial intelligence platforms for enterprises, conversational interfaces, and more specifically, chat platform integrations, like Slack and Microsoft Teams.

Along the way, we’ve raised over $12 million, making us one of the best funded B2B bot companies. We’ve been really fortunate, to have the time and money to invest in lots of marketing and sales experiments without sacrificing on the tech side. Often, young companies in our field have to either hold back on building a robust product, or go light on the go-to-market business side of things. But, while we were creating the technology that powers Talla, including enterprise-ready machine learning, security, and architecture, we got to arrive in the market with a good understanding of how to communicate with the people we think will benefit most from Talla.

We launched our primary product in April of 2017, after more than a year of testing more simple, experimental tools. Think of this main product, our Service Assistant, as an expert on company-specific data. Today, HR, IT and other internally-focused teams use Talla as an extension of themselves. Talla sits in chat, and, based both on the data sources it’s connected to and the information it’s trained on, can automate support queries, manage communication workflows around new or unique user issues, and proactively educate employees and new hires, on information they need to be successful.

Conversational interfaces are new, so there has been (and continues to be) a lot to learn when messaging to people through the bot. It’s a direct line to our users, and there’s a lot of information we need to gather and disseminate through the Talla assistant. Though I’m in marketing, and this might not seem a typical marketing activity, bots and AI have created new problems to solve, from data annotation to bot personality designers, and more. So, I’ve taken this on at Talla.

In this post I’ll talk about what our goals were, how we thought about building out the tooling necessary to achieve them and how some of the results turned out. I’ll also talk about how this internal tool turned into so much more than we intended.

Why We Focused on Direct Messaging in Bots

Our initial aim was to create a way to message our users through a free trial campaign via the bot. With traditional SaaS, you’d get an email drip campaign that walks you through your trial. The goal is to train and convert the user. Because our software is conversational, we wanted to run this marketing automation process through the bot itself.

Today, there are more tools emerging to help developers build and deploy bots and provide analytics and insights. But when we started talking about this, the Slack App Store wasn’t yet live and the existence of Microsoft Teams wasn’t even a rumor. So, we explored creating it ourselves, internally. It’s a lot of team time to invest with so much else to get done (such as building an entire sellable product), so we began by testing with the bare minimum. We started with a simple interface where I was able to send messages to people one by one. I would copy and paste a message to different users, literally hundreds of times. Then, I’d read through the data of the interactions each person had with the bot after they received a message, and compare them with the period before to determine the results of a certain campaign. Some of the analysis was obviously subjective, like the sentiment of the response, but we could also measure which product features people then used, or if an increase in overall usage was detectable.

Before the first tests, it was unclear whether running these processes through our bot might seem invasive, or if we’d be immediately removed from our users’ Slack accounts after interrupting their workflows. Some people we ran our idea by were strongly opposed to it—Slack was their sacred workspace, and it was already too noisy with messages from humans they worked with.

However, results showed that people were intrigued—even pleasantly amused by cold outreach from Talla. Feeling (personally) pretty smug having so far proved the naysayers wrong, and with much more to test, we continued building a campaign manager that allowed us to automate the sending of messages to certain groups based on parameters of our choosing. The next iteration allowed me to create an audience, by install date or company size, for instance, and send that batch of people a message. We also built out a more flexible campaign messaging tool, so interactive content, with buttons, etc. could be delivered, thus making result measuring much easier too.

Though our initial goal was to develop and test for that free trial campaign, we realized right away there was much more to learn before sending a complex series of messages.

What We Have Learned From DM Experiments

We’ve since sent tens of thousands of messages through our bot over many different experiments. Beyond guiding your users through an onboarding process and free trial flow, if you’re thinking about doing the same, I recommend you consider the following, as we did:

  1. Upsell from a freemium to paid version of your product
  2. Product announcements and ongoing user training
  3. Feedback collection to inform product strategy or market direction
  4. Internal lead generation and in-Slack discovery
  5. Messaging and positioning testing

Some interesting results we’ve seen are below:

No, that 3% isn’t a typo.

Key Takeaways:

  • While it looks like we had great success with the cold outreach message in the last row of the table above, and no one really hated us for it, we didn’t see an actual spike in product usage other than bemused replies.
  • One of the least successful messages ever at ~3% response rate was from trying to secure an introduction to a buyer via original Talla installers who were not the ideal admins — even with a bit of social proof provided. I’m still surprised (and a little dismayed) about how poorly that performed, but while Slack has flourished in terms of adoption from a bottoms-up model, it’s closer to a social network than a targeted SaaS application for a specific buyer, so I think few bots can expect the same type of success.
  • Our most successful messaging is ongoing training on how to use the product.

I can’t stress how important building and reinforcing a mental model of what your bot does, and how it fits into users’ workflows is, for two reasons. The first is that discoverability (and re-discoverability) is a challenge for bots. Many installers get it and forget it if you don’t figure out the right way to have them think of your bot whenever they’d normally do a bit of work themselves. Training the installer of your bot to evangelize it across the company, and use it in visible ways is a crucial part of adoption for many bots. Particularly if the main value add of the assistant is a more reactive than proactive workflow, you’ve got to think about this.

The second reason is one of the most interesting things that we saw in the very first prototype of Talla we tested. This was almost 2 years ago, and the bot bares (almost) no resemblance to Talla today. It could: tell you the weather, schedule a meeting for you, search your Google drive, and if you sent it a link to any online article or blog post, it could write a tweet for you based on a summary it created of the webpage content. People were incredibly excited about this bot. They couldn’t wait to install it. Then, they’d use it for just 1 thing, because they couldn’t remember what else it did. And, eventually, they’d drop off using it all together because actually none of those things powerfully changed their workday to last through the novelty of chatting about it. In fact, internally, we couldn’t even remember all the things Talla could do, because there’s no mental model for a job function like the things we’d built. They were broad and shallow, and therefore, too random.

So, from both a product functionality perspective, and a messaging standpoint, we needed (and then built) something that helped people in a deeply useful way, and they could remember to go to it with questions for IT or HR. Continuous messaging, through the bot, presented as product trainings, has helped reinforce that and drive adoption.

As for exact messaging to send, it’s truly brand and product specific. If you are building an enterprise bot, it’ll be just the same journey as figuring out your main value prop, website messaging, and content strategy. I will say that as bots have become more prevalent, the overall average response rates have gone down as the tolerance for receiving random messages wanes. Today, it’s all the more important to add value, build and maintain trust.

For us, this tool that was initially meant to be for internal use (internal in this case = just me, messaging people who had installed Talla) only later turned into a core piece of the product that’s exposed to end users. It allows them to build and send interactive information campaigns to employees at their own companies. It’s a large piece of why people buy Talla today, and I really believe that intelligent augmentation like this (for both internal and external use cases) is the future of work. If our product sounds like something your company’s HR or IT department wants to be in on, check it out.

And, if solving problems like the ones we’re thinking about gets you fired up, we’re hiring, so please reach out.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.