Designing trustworthy AI applications
Today, designing AIs you and your customers can trust is more than a competitive advantage, it’s a necessity. Let’s look at why and how companies can deliver on that promise. This post was originally published on the Deepflow blog.
How would you feel if one day you woke up only to find yourself declared a criminal by an algorithm?
Well, that’s what happened to a bunch a people in Michigan. In 2014, an automated system designed to detect and report unemployment fraud started to charge people with misrepresentation and demand repayment. With no human in the driver seat 40,000 people were billed about five times their original benefits. This whole ordeal lasted for two years. The agency in charge later admitted than 93 percent of the charges had been erroneous, but only after completing a partial audit of the system. To this day, most of the cases aren’t resolved and millions of dollars have been incorrectly collected. It would be easy to criticize this issue as a “bad apple” that fell from the AI tree but as automated systems are poised to become the spearhead of modernization initiatives, cases like this one cannot be classified as isolated incidents.
Even if that case seems a bit extreme, it should act as a cautionary tale. As more and more organizations rely on AI to modernize critical systems, a new inevitable challenge arises: How can we trust such systems? Companies are starting to grasp the reality of relying on predictive applications for their business, it is only truly valuable when every stakeholder understands how decisions are made.
Introducing Explainable AI (XAI)
When thinking of automation, the best way to avoid bad surprises is to approach it from a risk management perspective. In other words, we need to think how to design AI processes with human supervision in the loop. Let’s go through an example. Imagine you own a bank (dream big). In order to stay competitive you decide to implement an automated system that grants or denies loans based on your applicants’ data. On paper, it’s win-win scenario. Customers get an answer in minutes when it used to take days. Time and resources allocated to the task are reduced significantly and your bottom line is positively impacted. Congratulations? … Not so fast! It is important to remember that AI systems are not standalone applications. In production environments, they are part of an ecosystem and need to cooperate with the other actors. The difference with AI is that you are introducing a system that creates its own rules in an ecosystem that traditionally follows a pre-defined playbook. If you cannot clearly understand the rules by which your models play, aka a black-box AI, clashes will be inevitable at every level of your organizations. No matter how well your AI system is performing, risks are unmanageable by design if it is a black-box for the people whose job is to “cooperate” with it.
In our “banking” scenario, a black-box AI can create problems ranging from a frustrating user experience to company-wide compliance issues. Let’s illustrate it with examples on how the different stakeholders can react to your system:
Customer: why is my loan application denied?
Customer support: how do I answer this customer complaint?
IT operations: How do I monitor and debug this model?
Data Scientists: Is this the best model I can build?
Internal Audit / Regulators: Are these credit scoring models fair?
Your ability to give clear and transparent answers at every stage of the process will determine the usefulness of your models. For this particular reason the field of Explainable AI (XAI) aims to provide a comprehensive framework for AI practioners to future-proof their models. We will keep the technical recommendation for another article and will focus here on the things to consider to democratize explainable AI in your organization.
XAI and data teams
As models grow more complex, interpretability becomes a challenge. Depending on what algorithms are used, it is possible that no one, including the algorithm’s creators, can easily explain why the model generated the results that it did. Choosing the right model is about navigating the trade-offs between accuracy and explainability. For example, decision trees are easy to interpret by design, their predictions resulting from a chain of easily traceable sequential steps. But on the other end of the spectrum you have neural networks which provide a high predictive accuracy in cases such as computer vision (great for unstructured data types like images or documents), but at the expense of interpretability.
Today, the choice of algorithms is also constrained by the regulatory frameworks in place. For example, the European GDPR regulation states that “the existence of automated decision-making, should carry meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject”. A challenge that covers more than just selecting the right models. Bias in datasets and features are model agnostic but their impact on your model cannot be overlooked. Your AI systems will only be as good as the examples they can learn from. If your lending app regularly discards loan applications from minorities (a known issue) you need to be able to fix it quickly.
This type of problem are usually the result of data teams living in a bubble. Most organizations structures their data science operations in silos. Separation from the domain expert teams deprives them from a valuable feedback loop. Time for a little introspection, who owns AI initiatives in your company?
Enterprise XAI: beyond the data team
Companies with AI experience were the first to understand that a full stack data scientist does not exist, therefore collaboration with other parts of the business is what will make or break any AI strategy.
The people who raised the ethics issues with banks — the original ones — were the legal and compliance team, not the technologists. The technologists want to push the boundaries; they want to do what they’re really, really good at. But they don’t always think of the inadvertent consequences of what they’re doing.” — Paul Moxon, SVP Data Architecture at Denodo Technologies — Nextgov
But collaboration with members outside of the data team is the struggle today. The lack of a shared language between both sides is to blame. 87% of AI projects never make it to production and 80% resemble alchemy, run by wizards whose talents will not scale within their organization. As the AI hype is fading (no, robots won’t replace you) companies are realizing that throwing money at the problem and hiring more data scientists won’t magically create value, especially in global context of AI skills shortage. The focus is now shifting to the training of domain experts and how they can gain AI skills. Instead of PhDs, what about training domain experts on how to utilize existing AI tools? No need to reinvent the wheel, we need better drivers.
In The State of Enterprise AI Adoption (Cray — 2019), 95% of respondents stated that AI will become critical to their business capability sometime over the next 3 years and the lack of technical expertise was the biggest challenge ahead, second to the cost of deploying AI systems. While most companies understand the opportunities lying ahead, the market is still in the “education phase” of the AI adoption lifecycle, trying to pin down the use cases that will create most value. The process is slow because educating the workforce on AI is a challenge in itself. The learning curve is very steep and most are beginners with little to no background to follow theory/code-based courses. AI training content for domain experts is still lacking but the need is real. In the same survey, 72% of respondents said they have participated in one or more activities to educate themselves on AI in the past year.
It’s safe to say that the future of AI will follow the traditional technology adoption curve: it’s only when non-experts can create value with it that AI will truly become a commodity!
(As opposed to a Data Scientist) an informed business owner understands the domain, the context and the consequences — the true consequences — of the wrong answer. — Richard Eng, Chief Engineer at The MITRE Corporation — Nextgov
