Using the FEAT Approach to Avoid Biased AI
By Usha Jagannathan — Principal Engineer, McKinsey & Company
Today’s AI tools can produce highly accurate results but can be difficult to understand — even for experts. Explainable AI (XAI), in which “black box” machine learning (ML) models that make predictions are investigated to see how they reach these predictions, has become both an ethical and legal obligation. At a time when an increasing number of organizations are harnessing AI as a strategic capability, policymakers and researchers are asking how results can be trusted without transparency in how they were achieved — and how algorithmic accuracy should be balanced against other factors when developing ML models.
To answer these questions, many organizations are now turning to the FEAT (Fairness, Ethics, Accountability, and Transparency) principles to shape governance frameworks that develop responsible AI. This article explore each element of the FEAT acronym and outlines how it offers a framework to support the engineers and data scientists designing, developing, deploying and maintaining systems that learn from data.
Fairness
Unfairness often appears in training data, so mitigating data bias is an ethical necessity to gain user trust. ML developers and data scientists should consider both legal and ethical obligations to people interacting with their systems and examine components across accuracy, bias, model agnostics and justice when determining fairness in AI.
To determine if an AI system is fair, we must first define fairness. There are two paradigms of fairness: group (statistical) and individual. Under the statistical definition of fairness, minority groups should be treated the same as majority groups overall. The key to preserving statistical fairness is the parity of a selected statistical measure across these groups. However, different statistical measures may give contradictory results, making them difficult to optimize. On the other hand, individual fairness compares each pair of individuals by emphasizing that similar individuals should be treated similarly.
With this defined, we can progress to examining how development teams can incorporate key components of the Fairness principle into their AI framework.
Accuracy-based fairness optimizes for equality and ensures that no group in the dataset receives preferential treatment. A consistent focus on fairness across a model’s development is required to ensure predictions are as free from bias as they can be and there are a variety of ways for tech teams to maintain this vigilance. Stakeholder feedback should be sought when defining business requirements, training data should be as representative as possible and inclusive labels should be deployed liberally during data preparation. Techniques such as causal inference should be harnessed to discern actual cause and effect and intersectionality should be incorporated into the model evaluation phase.
Model-agnostic interpretability is another crucial component when developing trustworthy and fair ML. The model-agnostic explanation provides a generic framework for interpretability that allows for flexibility in the choice of ML models, data representations and user expertise.
Finally, when determining a model’s fairness, teams should consider not just the outcome but the process by which the outcome was reached. Procedural justice is the principle of ensuring that systems are transparent to end users, enabling individuals to observe and understand how decisions are made. Maintaining an openness around the data and methodologies that models harness to make decisions helps to enhance the perceived fairness of that model’s decision-making process — however, teams will naturally need to balance this transparency with sensitivities around any proprietary technology.
Ethics
Any AI/ML initiatives adopted within an organization should aim to do no harm by not subjecting specific populations to bias. As an ethical principle, to keep AI transparent, business stakeholders must ensure that transparency extends to the choices made when designing ML systems. They must also take precautions to avoid introducing bias into the data during training, testing and production.
To uphold ethics in AI models, the development team must ensure user privacy for explainability needs and promote the well-being of the people that interact with the AI-based applications. By avoiding unwanted biases that can creep into the algorithms, engineers can help users have a more engaged and consensual relationship with AI-developed products.
Accountability
Effective public and private governance requires the creation of accountable AI systems. Accountability is the ability to determine whether a decision was made in accordance with procedural standards and regulatory compliance — if these standards are not met, effective accountability enables organizations to hold the relevant person responsible. But at what point can we hold someone responsible for something beyond a reasonable level of risk analysis? Can the organization determine whether the AI/ML engineer or the wider team is responsible? Could any and all issues relating to the system be described as ‘engineering problems’?
These details can be determined by asking counterfactual questions of the decision-maker. For example, we may want to know what effect changes to an input has on the outcome, or conversely, what changes must be made to the input to change the output in a specific way. This helps us understand why two similar input data sets result in varied outputs. Counterfactual explanations promote accountability and trustworthiness, allowing organizations to check for consistency when decisions are made. It is critical that accountability and transparency be introduced in the early design and development of technological systems, rather than added later during after-the-fact regulation and review phases.
Transparency
There are several challenges regarding transparency for risk-assessment tools and automated decision-making systems. For instance, it is considered high risk if an organization doesn’t reveal which specific ‘features’ (data points) are relevant in automated calculations, or what weight those features are given. AI models should be ethically permissible and worthy of public trust. The goal should be to expose why certain standards for the application are met or not met.
Today, we rely on AI to make critical decisions such as those associated with automated health appointment schedulers, claims processing, loan approvals, and more. AI systems that are unexplainable should not be acceptable, and XAI needs to be part of the equation when building AI systems we can trust.
Some practitioners may assume that training data is always clean and represents society at-large, but results have proved this is not the case. Search engines have been known to encounter issues in which image recognition systems wrongly classified images of marginalized groups — searches for ‘white teenagers’ produced wholesome photos while ‘black teenagers’ supplied users with criminal mugshots. Chatbots have also revealed weaknesses in training AI with existing data, with notable examples of bots that have mined the internet for social media conversations and ended up replicating the offensive languages they observed.
Cultural stories do shape how we interact and humanize AI development, and addressing societal bias is critical in improving the design of conversational AIs. Intense testing is just as important as training the data and performing extensive stress tests for both good and worst scenarios can improve the quality of AI model prediction. This level of testing is only possible with transparency in the model’s decision-making process — even if the model fails, XAI methods can provide better explanations.
Existing and newer ML models must follow ethical guidelines during product development and focus on the four key principles (FEAT) outlined above. In this way, the process is made much clearer and XAI development can be incorporated from the beginning of the design phase, rather than assembled after the technology is created. Although implementing explanation earlier in the design does have some limitations, it will generally lead to better design practices. Wherever human data is involved, extreme care is needed to prevent the introduction of human bias in the data. Humans can’t trust an AI model if they don’t know how it works.
Although we are still scratching the surface of making truly explainable AI models, it is essential to ensure that we create a better, more equitable world by adopting the AI governance framework. By understanding how these models work, engineers can decide if AI models abides by local tech regulation and legislation. AI offers the opportunity to reshape not just how business but wider society functions, and data practitioners will be at the forefront of developing these world-changing solutions — but care must be taken to ensure ethical guidelines are followed to avoid AI replicating existing societal biases.
