CAI Challenges and Solutions
This story is a rough-draft. Check back later for the fully-polished story or post a comment telling me what you’d like me to research and write for you.
AI is not lacking challenges. And those who work on these challenges are looking for and publishing lots of solutions.
Here I list the challenges that designers, researchers, and developers have faced as they try to improve the state of the art in conversational artificial intelligence (CAI) or have tried to implement it in business. I include solutions and references where available, such as where I learned what is challenging for others in this field. I have sorted them in order, from easiest to hardest, according to my own subjective intuition and my mood at the time of writing. The easiest challenges are business challenges that should be straightforward to implement, given the right skill and knowledge, while the hardest are research challenges that may not be solvable by anyone yet.
Skilled Talent
Lack of skilled talent to develop and work with bots. Most businesses are not equipped to teach untrained virtual agents how to serve their customers.
Solutions:
- Subject matter experts (e.g. customer support employees) should be used to train the bot.
Sources (challenge):
- Accenture Survey.
Developer Education
One challenge that the chatbot industry faces is ignorance. This is an industry-, market-, or world-wide challenge. “We have to keep educating people about the technology and its benefits. Like how can the technology help them to streamline operations, in cost optimization, data collection and management to inventory and supply chain management,” says Vaish.
Solutions:
- Education
- Credible Marketing
Sources (challenge):
Benchmarks to Drive Advancements
Sources (challenge):
Hidden Chatbot Affordances
Related to education as a market-wide concern is this concern: each individual chatbot must “educate” each of its users about what it is capable of doing. Customers and prospective customers don’t immediately understand what the virtual chatbot can do or why it can be helpful to them. Users won’t know what the bot can do without the bot telling them, at least until the industry becomes standardized and familiar to most users.
Solutions:
- A bot must (1) tell a user what it can do, what its scope is and then (2) perform well within that defined scope.
Sources (challenge):
- Casey Kennington, Elafris blog
Learn more:
Ease of Upgrade or Maintenance
Sources (challenge):
- Accenture Survey
Customizability
It can sometime be hard for a business to buy a CAI solution and then customize it enough to fit their use cases and data.
Sources (challenge):
- Anonymous sales team
- Possible solution: Reframe the problem to efficiently acquire data
Picking the Right Tech Stack
“You need to undertake proper due diligence of the technology but, unfortunately, this is not always done. There is so much technology available in the market but most of it is not appropriate for enterprise, due to lack of scalability, bad language understanding or lack of transactional capabilities.”
Learn more:
Alignment between Natural Language and Formal Programs
Sources (challenge):
Discoverability
It can be hard to help your chatbot users even discover that your chatbot exists. Many mechanisms exist to promote discoverability, with some being poorer than others.
Sources (challenge):
- Accenture Survey
Solutions:
- Social media and blog post mentions
- Make your chatbot name or brand memorable so it is easy to search for it in Facebook or Google
- Make your chatbot “searchable” in Facebook Messenger by adding keywords, description, etc.
Trustworthiness and Interpretability,
Sources (challenge):
Determining Success Criteria
If CAI objectives are poorly defined or your tools don’t meet your objectives, you will not succeed with CAI. “Determine your success criteria and goals as early as possible. If you want to automate 80% of your customer interactions or increase your CSAT score, this is something you need to hold your vendor accountable to and for. We often see many projects drifting without higher purpose and low ROI.”
Learn more:
Balancing Short- and Long-Term Development Goals
“Think both short term and long term. Make sure that you get early success to gain internal approval and buy-in, but also make sure you have resources in the long term to develop a virtual agent. It is a continuous journey — like a human appointment to your team.”
Learn more:
Structured Data
Business data and other kinds of data needs to be structured in a way that supports CAI use cases.
Solutions:
- Knowledge graphs
- Other structured data
Shortage of Skilled Developers
Sources (challenge):
- Accenture Survey
Knowledge Acquisition
Creating an advanced AI based knowledge extraction and ingestion feature set to take structured and unstructured content to produce bot knowledge.
Sources (challenge):
- Kore.ai
Learn more:
Efficiently Acquiring Data
Includes word knowledge and grammar data. Related to customization challenge.
Sources (challenge):
- Casey Kennington
Reward Data
Feedback or supervision data for machine learning and especially reinforcement learning based knowledge acquisition. Reward is the evaluation metric as well as the optimization criterion. Only 1% of users will likely provide explicit feedback if asked. Sentiment analysis provides positive and negative cues to the system of its performance and the user’s acceptance. Customer care center deployment can provide reward each time a call is not transferred to a human user. Receiving additional knowledge from the user as a reward signal.
Sources (challenge):
- Milica Gasic
Learn more
Correct Reward Data
Data that correctly guides reinforcement learning to the correct policies.
Sources (challenge):
- Milica Gasic
Sentiment- and Emotion-Awareness
Sources (challenge):
- Milica Gasic
Flexible or “Non-Linear” Dialog Management
Developing an artificial agent that you can talk to and that is flexible in how it handles the dialog management. DM is state-full. NLU and NLG are stateless. Usual ML works well for NLU.
Solutions:
- I have a hypothesis for making chatbot dialog management much more flexible while still allowing the chatbot to precisely satisfy customer. This is very different from the Rasa approach which uses machine learning to try to nudge the chatbot toward an effective sequence of reactions to users’ inputs — which I believe is not very effective. My proposal involves a more traditional AI technique you might call goal-directed state-space search using knowledge graphs. I hope to write about it and try it out in the future. I’ll let you know if I get it to work.
Cost
In a business setting, the cost to acquire or deploy.
Sources (challenge):
- Anonymous sales team, Accenture Survey, TMR Research
Empathy
Sources (challenge):
- Online discussions
State Space (in RL)
The state space for reinforcement learning can become too big.
Solutions:
- Hand-craft constraints to prune the available actions. Use reinforcement learning in an interactive environment to learn policies. Static environment/data is not sufficient because then it turns into supervised ML, not RL. Simulated users from recorded dialogs are okay to start with. Real users are essential eventually.
Long-Term Interactions
Being able to carry on a long, open-ended conversation.
Sources (challenge):
- Amazon Alexa Prize, Milica Gasic
Poor Communication Quality or Performance
Current and prospective customers have so many different needs and requests that all but the best chatbots for a domain are too simplistic to really give an adequate return on investment. Inability to incorporate history/context for personalized experiences, failure to adequately understand human input.
Solutions:
- Reduce to grammar engineering or supervised ML training. Get hands dirty and test the accuracy of candidate technologies.
Sources (challenge):
- Accenture Survey, TMR Research
Platform Fragmentation
Sources (challenge):
- Accenture Survey
Voice Conversations
“Voice conversations tend to be more nuanced, and specific capabilities need to be built into the interfaces to understand interruptions, cues and tone-of-voice signaling, which is currently beyond the scope of most implementations.”
Learnable Knowledge Base
Sources (challenge):
- Milica Gasic
Self-Learning/Self-Awareness
Sources (challenge):
- Accenture Survey
Other Information about Solving Challenges
Rasa’s predictions about which challenges will be solved in what order: https://blog.rasa.com/conversational-ai-your-guide-to-five-levels-of-ai-assistants-inenterprise/
What other challenges or solutions do you know about? Add them to the comments.
