The Future of Conversational AI

The impact of Generative AI on chatbots and voice assistants

Arte Merritt
ConversationalAI
18 min readDec 4, 2023

--

This year has been incredibly exciting for the Conversational AI industry. The emergence of Generative AI infused new enthusiasm in the space and changed the way we approach Conversational AI development and the resulting user experiences.

As 2023 nears an end, I had the opportunity to ask leaders in the conversational AI space about what they saw as the biggest trends in 2023 as well as their predictions for 2024. Read more about their thoughts below.

The experts include:

The biggest trends of 2023

The emergence of Generative AI and Large Language Models

The biggest trend by far our experts agreed on was the emergence of Generative AI (“Gen AI”) and Large Language Models (“LLMs”). The industry was taken over, seemingly overnight by the buzz of Generative AI. As Philipp Heltewig of Cognigy states, we are finally at a stage where Conversational AI can deliver on the promise of truly human-like understanding and conversation.

Everyone is talking about Gen AI.

Gen AI has had a significant impact on the Conversational AI (“CAI”) space. It has brought about increased investment in CAI solutions, changes in tools, methodologies, and processes, and new concerns about safety implications of AI. Enterprises are not only experimenting with Gen AI in Conversational AI solutions, but exploring how to leverage it in other workflows too.

The impact of Gen AI was not only felt at the enterprise, but at the consumer level too. As Yuko Araki of AWS points out, services like ChatGPT brought consumers closer to the technology, and more curious about what AI is, or does. Consumers were able to see how useful a chatbot like ChatGPT could be, versus a “salesy” pop-up chatbot on a website. They gained a new appreciation and understanding, whereas before they may not have been aware of how chatbots worked with natural language understanding (NLU). JD Ackley of Black Box adds, ChatGPT grabbed the spotlight and became a household name. This year AI went more mainstream, as the technology was embraced in real-world applications.

While there was definitely a lot of excitement about the use of Gen AI in CAI, some of our experts also saw enterprises tempering expectations. As Hans van Dam of Conversational Design Institute puts it, everyone was looking to implement Gen AI, but at the same time realized there is no magic bullet, and it was still hard. Andrei Papancea of NLX further explains, expectations were tempered after enterprises gained a better understanding on what is currently practical.

This tempering resulted in hybrid solutions using both NLU and Gen AI/LLMs. As Dexter Zavalza of Deloitte points out, leading CAI platforms incorporated LLMs to provide flexibility and versatility between traditional NLU and Gen AI approaches. This is a similar trend Gordon Chu of Cathay Pacific saw as well.

The hype of Gen AI may also have muddied existing customer experience (CX) projects that used traditional NLU approaches, as folks wanted to overlook impending or inflight deployments, as JD indicates.

Increased investment

The excitement of Gen AI in the Conversational AI space, led to increased interest and investment in offerings. As Yan Zhang of PolyAI states, there were increased budget allocations for CAI given the increased interest in AI at the C-suite level. JD echoes this sentiment in that ChatGPT helped raise the question, “What are we doing with AI?” to the top of the C-suite’s mind.

Priorities started to shift and new business initiatives appeared, as enterprises tried to figure out what Gen AI meant to their businesses, explains Surbhi Rathore of Symbl.ai. She saw a significant increase in capital and resource investment. This led to more agile product roadmaps, as enterprises experimented with LLMs to unlock new features or improve existing workflows.

Changes in methodologies

Gen AI had a significant impact on the approach to Conversational AI design and development. As Dexter indicates, being fluent in Gen AI emerged as a prerequisite for CAI practitioners.

One of the biggest questions was the role of conversation designers in a new Gen AI world.

As Yan states, the role of conversation design was “thrown into limbo” by Generative AI. Dexter further adds, conversation design got inextricably tied to “prompt engineering.”

However, there is a difference in prompt engineering for the enterprise use case, versus a consumer asking ChatGPT to craft an email, as Greg Bennett of Salesforce points out. He adds that teams may have initially over-indexed on prompt-engineering. These teams eventually realized the operating costs of prompt-engineering were nearly as high as manual chatbot design in terms of interaction design labor. Now teams are developing more realistic cost analysis and delivery times around Gen AI / CAI solutions.

We also saw the emergence of new processes, like Retrieval Automation Generation (RAG), in conversational AI solutions. As Greg points out, RAG is a fundamental aspect to “hydrating” prompts to optimize model outputs.

New AI safety concerns

Generative AI led to new discussions around the safety of AI. In particular, there were concerns about sharing Gen AI responses with end consumers — for example, if the generative response is off-brand, is in-appropriate, contains personally identifiable info (PII), or is inaccurate (“hallucinations”).

These concerns have led to customized LLM models, guardrail solutions, and hybrid Gen AI / NLU approaches to reduce risks.

This is an area for opportunity and improvement in the future. As Yan adds, the discourse on AI safety has been conflated with AI control — not having the chatbot say something incorrect is different from being able to pinpoint how the mistake occurred.

Predictions for 2024

Generative AI will be more mainstream in production

Our panel of experts predict Generative AI will continue to mature and become more mainstream in end-user, Conversational AI applications, as enterprises shift from experimentation to implementation. As Hans states, 2023 was the year for experimentation and discovery, and 2024 will be about implementation and adding value.

Improvements in Gen AI and LLMs will lead to improved customer experiences. As Yuko predicts, further adoption of LLMs in the contact center space can lead to a better user experience, and help achieve concrete business outcomes. JD further predicts that 2024 will be the year of “Bot 2.0,” wherein enterprises recognize the positive impact advancements in Gen AI, NLU, and Automatic Speech Recognition (ASR) can have on the customer experience.

There will be an increased usage of Gen AI and LLMs in Conversational AI experiences. Surbhi predicts an increased reliability in LLMs through RAG, fine-tuning, or custom models, and increased scaling for end-to-end workflows. Similarly, Andrei also sees an increased focus on purpose built LLMs for specific verticals and use cases. These fine-tuned, custom, and specific models can improve reliability and accuracy of responses. As Gordon predicts, more enterprises will be able to deploy solutions while minimizing hallucination risks.

As the models become cheaper and more optimized, overall costs for implementing Gen AI will continue to go down. Greg predicts these factors, as well as industry pressures to move faster, will help drive adoption of Gen AI.

Even for those enterprises not yet exposing Generative AI to end customers, the technology will have a huge impact on improving the customer experience. As Jonathan Barouch of Local Measure predicts, Generative AI will have a big impact in chatbot design and testing conversational flows and experiences.

Beyond the enterprise, Yan believes there will be a wider adoption of Gen AI solutions in the small and medium-sized business (SMB) sector, where concerns of AI safety and hallucinations may matter less.

Improvements in Generative AI will lead to new functionality as well. Philipp predicts non-textual Gen AI, including human-like voices and video generation in real-time, will play a large role in 2024.

New tools and methodologies will emerge

As the underlying technologies continue to advance, new tools and methodologies will emerge.

As JD predicts, the industry will demand new tools that help validate and provide insights into how Gen AI is performing. For example, he would like to see more tools that enable guardrails on Gen AI to give enterprises confidence in their solutions. Surbhi also predicts better evaluation frameworks specific to data, use cases, and industries. Dexter envisions new tools and workflows will evolve, in favor of expediting design and delivery.

The CAI design methodologies will continue to change as well. This is especially true for conversation designers. Dexter predicts there will be a redefinition and standardization of best practices for CAI design to find the right balance of NLU and Gen AI. Greg sees a potential erosion, or questioning, of the role of conversation design as Gen AI solutions become more cost friendly over manual design. Greg would also like to see a standardization for prompt engineering. He believes setting a standard for what is considered ideal for a prompt in an enterprise use case is paramount to scaling.

Increased education on AI safety

As AI becomes more prevalent in our daily lives, safety and education will become more important topics.

Concerns on AI safety will continue in 2024. As Yan predicts, safety will be a key topic, but may not be solved. He adds, the controllability of conversational AI will be important. Hans adds to this in predicting there will be a shift to focus on risk mitigation and quality assurance. Greg predicts there will be a greater examination of how to use Gen AI in an ethical and compliant manner. He believes the European Union will most likely lead the way, given they previously took charge with GDPR for data privacy.

There is an opportunity to educate consumers about AI to help alleviate concerns. Given chatbots and conversational AI solutions are one of the most common ways consumers interact with Gen AI, Yuko believes those working in this space should play a key role in educating users and evangelizing the practical benefits.

Conclusion

The emergence and impact of Generative AI was definitely the biggest trend of 2023 and the source of excitement for 2024.

Generative AI brings the prospect of more human-like interactions in conversational AI that truly satisfy the customer with better user experiences and results — and are hopefully easier to build and maintain as well. We will continue to see advancements in the underlying technologies and new tools, roles, and methodologies emerge. With all of this comes a responsibility to educate users and help ensure trustworthy, compliant solutions.

We are truly living in exciting times. I look forward to seeing the advancements continue and what is yet to come in 2024.

In their own words

Below are the responses from the panel of experts

Andrei Papancea, CEO / Co-founder, NLX

What were the biggest trends in Conversational AI in 2023?

The biggest trend by far was going from extreme hype around Generative AI (text) for enterprise use cases to companies having a deeper understanding of the space, resulting in more temperate expectations of what’s practical for enterprises looking to deploy such technology for customer-facing use cases.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

I believe we’ll see consolidation in the enterprise conversational AI players. Furthermore, I believe we’ll see enterprises increasingly looking to centralize all their conversational AI management, across channels (text, voice, multimodal) and use cases (internal and external). Lastly, I expect the interest in generative AI (text) to continue to refine and mature, though with a focus on purpose-built LLMs for specific industries and verticals.

Dexter Zavalza, Conversation Design Lead, Deloitte Consulting

What were the biggest trends in Conversational AI in 2023?

  • CAI teams excelled at designing and delivering solutions, in part because of increased experience in the space and in part because of more powerful and intuitive tooling across leading platforms
  • Leading CAI platforms found ways to integrate LLMs into their workflows and give delivery teams versatility to flex between intents and GAI
  • Conversation Design became inextricably tied to Prompt Engineering as a discipline.
  • Generative AI tools became commonplace for accelerating and streamlining Conversation Design workflows
  • Generative AI fluency emerged as a prerequisite for CAI practitioners in technical and functional roles

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

  • 2024 will be characterized by the continued integration of LLMs and custom GPTs into CAI frameworks, including continued redefinition and standardization of design and development best practices to find the right balance of intent-driven development and GAI handling of user queries.
  • We’ll see a rise in enterprise production solutions that provide the appropriate balance of precision, security, and cost while putting user experience first through the continued integration of CAI and LLMs
  • The industry will become more fluent and efficient in delivering CAI + GAI conversational solutions as enterprise leaders cut through the hype and become more well-versed in the latest technologies to facilitate more mature strategic conversations
  • GAI tooling will continue to emerge and CAI workflows will further evolve in favor of expedited design and delivery

Gordon Chu, Digital Innovation Manager, Cathay Pacific

What were the biggest trends in Conversational AI in 2023?

Hybrid LLM-NLP platforms

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

Corporate creating GPT like conversational experience with minimized hallucination risk

Greg Bennett, Director of Conversation Design/Einstein GPT, Salesforce

What were the biggest trends in Conversational AI in 2023?

In H1, we saw a huge (over) indexing on prompt engineering. Indeed, the focus was pertinent at the onset of generative AI implementations across organizations, though teams quickly realized the operating cost of prompt engineering was nearly as high as that of a manual chatbot in terms of interaction design labor. Notably, for enterprise and B2B use cases, it quickly became evident that a distinction was needed between a user query (e.g., “write me an intro email to my new client”) and a grounded prompt that contains relevant variables needed to personalize generated content for a user.

This led to the rise of interest in H2 around retrieval automation generation (RAG), a fundamental aspect to hydrating prompts before a model call in order to optimize for a personalized output. Teams are starting to develop more realistic cost analyses and delivery timelines around generative conversational AI solutions that can deliver business value and user-personalized utility. As model providers such as OpenAI continue optimizing their service offerings as we saw with the more affordable GPT-4-Turbo, overall cost to serve for businesses implementing generative conversational AI solutions continue to go down, driving lift as we head into 2024.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

What I’d like to see happen: a more rigorous set of standards for prompt engineering to crystallize across the field. Currently, we see methods haphazardly applied on a use case by use case basis, which doesn’t scale. Setting a standard for what is considered ideal for a prompt to satisfy B2B and enterprise use cases will be paramount. Additionally, I’d like to see more focus on how to balance more optimized system architectural design and functional cost to serve against overall experience goals; sometimes, a model call isn’t the effective means of achieving a user goal, especially when considering heavily regulated industries such as finserv or HLS.

What I predict will happen: increased industry pressure to move faster and lower the operational cost of implementing generative conversational AI solutions writ large. We should expect to see an erosion — or at least a questioning — of the need for conversation design when generative models are executing conversational usability at a more cost-friendly scale than manual conversation design, and instead, a huge swing in the direction of automating everything using local LLMs as their operating cost continues to lower. I presume we’ll also see a greater examination of how to execute these solutions in an ethical and compliant manner, with the European Union’s GDPR leading the fray as usual.

Hans van Dam, CEO, Conversation Design Institute

What were the biggest trends in Conversational AI in 2023?

Everybody is looking to implement generative AI and create LLM powered chatbots. But at the same time, large enterprises also realize that there is no magic solution and that it’s still hard.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

I think we’ll see lots of LLM experiments and quite a bit of churn on a lot of these projects. I think you’re going to see more focus on risk mitigation and quality assurance. This year was about play and discovery, next year will be about implementation and value.

JD Ackley, Global CX Business Development Manager, Black Box

What were the biggest trends in Conversational AI in 2023?

Conversational AI adoption was growing steadily in the beginning of 2023 until ChatGPT grabbed the AI spotlight and became a household name. Seemingly overnight, “what are we doing with AI” was on the tip of the C suite’s tongue. This overwhelming crush of Generative AI attention mired CX projects deploying mature Conversational AI technologies as newly minted AI experts wanted to look over impending and in-flight deployments. Overall, 2023 will go down as the year AI went mainstream. The technology made so many advances and finally got embraced in real-world applications.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

I think 2024 will be the year of “Bot 2.0”. Enterprises that have had poor to marginally good Conversational AI experiences will come to see the advances in ASR, NLU and NLG will drive significantly improved experiences. I also see the industry will demand tools that validate and provide insights into what these new bots are saying to their customers on behalf of their brand-especially with companies experimenting with incorporating NLG from various LLM’s directly into the experience. I would like to see tools emerge that put clear guardrails on NLG responses and that can transcribe bot interactions to give enterprise businesses the confidence to continue to adopt Conversational AI technologies.

Jonathan Barouch, CEO / Founder, Local Measure

What were the biggest trends in Conversational AI in 2023?

In 2023 we finally saw more businesses go omni-channel with their conversational AI and bots. Historically we’ve seen brands dabble with conversational bots on their main website bot or in app bot limiting their engagement to first party channels. In 2023 we saw our brand clients take the leap to add more conversation customer service and conversational commerce experiences to the other digital channels they support most notably WhatsApp, Facebook and Instagram. We saw clients who created sophisticated marketing campaigns with entry points from online ad units into Facebook Messenger or WhatsApp with personalised conversational AI to help sell, serve and triage customers.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

In 2023 brands were still reluctant to open up any GenerativeAI use cases to end customer for fear of getting it wrong. Now a year later the technology is better understood and I envisage some brands will start experimenting opening GenAI to end customers for very limited and low risk use cases. Even without opening GenAI to end customers the promise for this technology to drastically improve customer experience is huge. From helping with bot design, to having GenAI test conversational flows or even end customer experiences this technology is going to have a much bigger impact on the conversational AI space in 2024.

Philipp Heltewig, CEO/Co-founder, Cognigy

What were the biggest trends in Conversational AI in 2023?

The biggest trend in 2023 was clearly the emergence and integration of highly capable large language models. Finally we’re at a stage where conversational AI is delivering on the promise of truly human-like understanding and conversation capabilities.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

I predict that non-textual generative AI, as in truly human-like voices, videos and real-time generated avatars will play a large role in 2024.

Surbhi Rathore, CEO/Co-founder, Symbl.ai

What were the biggest trends in Conversational AI in 2023?

New Business Initiatives: I think we all have seen business priorities quickly shifting to find out what generative AI applications and infrastructure means for a business — product, team, messaging — touching every part of the business.

Product roadmaps became more agile: Significant increase in investment (both capital and resources) in research and exploration of PoCs OR using open source language models to experiment with use cases that redefined existing product roadmap. Focus on figuring out how creative generation tasks can unlock a new feature or massively improved an existing workflow.

Architecting the abstraction layer for LLMs: For more tech-forward and digital native businesses, efforts were put towards designing their own LLM-infra layer and how they would make use of the right model for the right workflow bringing in their unique data or user experience on top of the model for differentiation.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

Simplifying business-specificity in LLMs with Ease, reliability and cost of fine-tuning, RAG or custom models. Making the use of LLMs personalized to use cases and businesses as a whole.

Scaling LLMs and supporting other components for end to end workflow keeping in mind non functional metrics for building a better user experience.

Better evaluation frameworks will evolve pushing the boundaries on current evaluation methodologies specific to data, use cases or industries.

Yan Zhang, COO, PolyAI

What were the biggest trends in Conversational AI in 2023?

The excitement around generative AI means that we’re seeing far more inbound requests in 2023 than before.

Perhaps because a lot of innovation in AI has come outside of the cloud giants, startups and scale-ups are seeing more enterprise AI budget allocation than before.

The role of conversational design has been thrown in limbo by GenAI, but imo designed conversational experiences will still be the mainstream.

There is a lot of excitement about AI at the C-suite and the board level. So innovation mandates and budgets are relatively generous. In general, enterprises have not yet figured out which problems would be best addressed by AI.

The discourse around AI safety is still conflated with AI control. Not having your bot say something incorrect is not the same as being able to pinpoint how a mistake was made. These are still two different problems.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

In 2024, safety will still be a topic in AI (as it probably won’t ever get solved). But controllability (the ability to isolate behavior to a certain part of the conversational design) will become a valued feature in ConvAI.

The new paradigm of conversational design will move beyond “”nodes and transitions”” to orchestrating “”jobs to be done”” in a conversation.

Wider adoption of customer-facing, fully GenAI bots from small and medium businesses. Safety and hallucinations will matter less for that segment.

Non-AI features in ConvAI, like ASR correction and latency minimization will become more differentiating as core models become commoditized.

Yuko Araki, Generative AI Specialist, AWS

What were the biggest trends in Conversational AI in 2023?

Probably the biggest impact the market had in the conversational AI space was bringing awareness of large language models to common users and consumers via OpenAI’s ChatGPT. ChatGPT gave people an opportunity to experience what a chatbot can do beyond what people experienced using Alexa or a support-bot that pops up on a shopping site. Being able to directly initiate a chat session using the OpenAI page made many people feel differently about AI, realizing how helpful the tool can be in knowledge retrieval or personal content editing, rather than an unwanted salesy popup that shows up on the right hand corner of your computer screen. It’s likely that most people who engaged with customer support bots in the past were unaware of NLP or ML models that were powering the interaction in the backend. But with the PR that OpenAI received in 2023, and with the consumer interest, I people (and market in general) now feel closer to AI as a technology tool. People are curious, and are learning about what AI really does or means. Businesses began integrating LLM modes into messaging apps by the end of 2023. WhatsApp is a great example of that, providing a conversational channel impersonating SnoopDog to attract consumer attention.

What do you predict will happen, or would you like to see happen, in the Conversational AI space in 2024?

With technology becoming more prevalent in our daily lives, in consumer applications or in business, we need to educate the masses about AI to reduce fear of AI. In some people’s mind, AI will replace humanity and take over the world. Conversational AI space in the form of chatbot or voice bot is the easiest and most visible way for common people to interact with LLM but LLM enables much more. So, in a way, those who work in the conversational AI space must represent the tech space, spearheading the market in playing the role of educating common users about what AI or LLM are so that we can collectively reduce unnecessary fear of AI. We need to evangelize the practical benefit AI brings with measurable business outcomes. Improving enterprise productivity across industries contributes to GDP growth at a macro level. In fact, working at AWS, I learned that protein studies, and DNA related work requires the most amount of compute, using large scale machine learning models. The technology will bring new drug discovery and diagnostics helping humans progress in the most meaningful way. Writing and debugging software code with the help of AI significantly reduces development time. By reducing time to develop, we can collectively achieve our purpose at a faster pace.

Specifically in the conversational AI space, I would like to see further deployment of automation bots in the contact center space. Today, we already benefit from some of the industry leaders providing NLP backed voice bots in the call center, but the interaction can be further improved using LLM, and integrated throughout wherever end users are in omnichannel messaging channels in the most interactive, connected way. And I would love to start to see concrete business outcomes that derive from these technologies in the workflow automation and contact center space.

Arte Merritt is the founder of Reconify, an analytics and optimization platform for Generative AI. Previously, he led the Global Conversational AI partner initiative at AWS. He was the founder and CEO of the leading analytics platform for Conversational AI, leading the company to 20,000 customers, 90B messages processed, and multiple acquisition offers. He is a frequent author and speaker on Conversational AI and data insights. Arte is an MIT alum.

--

--

Arte Merritt
ConversationalAI

Conversational AI & Generative AI Entrepreneur; Founder of Reconify; Former Conversational AI partnerships at AWS; Former CEO/Co-founder Dashbot