TrojAI
Published in

TrojAI

Achieving Trustworthy AI Depends on the Consolidation of These 3 Pillars

The challenge is that each of these pillars is an entirely separate discipline.

Most breakthrough technologies are introduced by first-mover startups that prove out market need before the emerging best-of-breeds are scooped up by larger companies in the inevitable market consolidation that follows. Across many sectors in AI this has already occurred and the innovations are already impacting our daily lives in mostly positive ways, give or take. The difference with AI, however, are the lesser known dangers that AI introduces. Companies need to move towards addressing these inherent dangers to protect the very pace of AI innovation that we have come to enjoy.

Trustworthy AI includes policies and practices around things like MLOps, governance, model versioning, and monitoring of environmental changes that can lead to data drift. These items are well understood and increasingly becoming table stakes for AI developers. To truly achieve Trustworthy AI, however, involves addressing 3 pillars, each of which is an entirely separate discipline requiring specialized expertise. Since all of these pillars are blue ocean waters, consolidation of a single commercial solution that addresses all 3 is several years off. In the meantime, we offer some recommendations on how your organization can move towards Trustworthy AI today.

(1) Interpretability and Explainability

It is no longer sufficient to deploy models without understanding why they make the decisions they make. In the words of one of our favourite explainability CEO’s, “If we don’t know how it works, we can’t know when it will fail.” While there are several open-source explainability libraries like SHAP and LIME, these tools only scratch the surface towards providing insight into why a model makes its decisions. Conversely, commercial tools built by dedicated explainability experts can help identify why a model is making incorrect decisions, or when it’s making the right decisions but for the wrong reasons. They can also highlight new opportunities of what human-in-the-loop experts should be observing when they make their own decisions. While it is possible to build explainability in-house, understand that this is a specialized area that may be better outsourced to dedicated experts so your team can focus on delivering the AI innovations driving your business.

(2) Fairness and Bias

We all know the awful stories and, suffice it to say, that it is no longer acceptable to deploy models without understanding the inherent biases that may be contributing to discriminatory model behaviours. From a data science perspective, this is another highly specialized area. It is not enough to simply balance data or scrub datasets of the features that may possibly lead to marginalization of particular social groups. Bias can leak its way into models from unexpected places and AI innovators have an obligation to protect everyone’s right to equal opportunity, and the penalties for ignoring this responsibly are getting real. The European Commission just published the first-ever legal framework on Trustworthy AI and the fines for neglecting model bias are significant enough to impact a company’s stability.

(3) Robustness and Security

An emerging and equally important pillar for Trustworthy AI is robustness and security. Newsflash. All of these world-changing AI innovations are hackable — yes, even the ones that your CTO says are safe. Malicious actors can poison training data to modify model behaviours or even embed Trojan attacks that can be enacted at a later time. They can also leverage naturally occurring noise in the data to perpetrate model evasion attacks to force a desired misclassification. These attacks are possible because the inputs to both ends of the model hang outside of traditional cybersecurity perimeters. Accuracy metrics, even when done correctly, do not account for how brittle a model can be when moving from the lab to the real world and thus robustness metrics are required to test and quantify how models handle environmental edge cases. It’s these same edge cases that malicious actors can exploit to harm us. Similar to explainability, open-source adversarial AI libraries exist like ART and CleverHans but building in-house solutions requires specialized expertise and dedicated resources to keep up with emerging attacks, again possibly taking time away from your core innovations.

So what should your organization do to deploy Trustworthy AI today?

These 3 pillars require completely different solutions that we are only just beginning to understand. Consequently, market consolidation into a single solution will take time — time that we don’t have if we wish to maintain the current pace of AI innovation.

At minimum, we need to think about these 3 distinct pillars and how they combine to provide Trustworthy AI. We need to move towards incrementally addressing each as early in the development cycle as possible, and immediately addressing the vulnerabilities of those models that are already deployed.

If you need help with Robustness and Security, we can help. If you need help with the other pillars, we know some great companies. Finally, be cautious if a vendor tells you they are experts in everything.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store