Mitigating Bias: Integrating Generative AI, Foundation and Large Language Models in Enterprise Workflows
The business world is rapidly evolving due to the rise of generative artificial intelligence (AI) and advancements in natural language processing (NLP) technologies such as Foundation Models (e.g. ChatGPT, Bard) and Large Language Models (e.g. GPT-4, LLaMA). These technologies have the potential to foster new ways of communication and collaboration that can transform enterprise workflows, making them more efficient, assistive and user-friendly. But on the other hand, when left unchecked they have the potential to introduce biases in the enterprise workflows and cause significant damage to the society.
In this post, we will explore:
- The technologies: Generative AI, Foundation Models, and Large Language Models
- How these technologies are set to revolutionize enterprise software by enabling human language interaction that crosses multiple system boundaries
- What patterns may lead to biases when using these technologies and what techniques enterprises can employ to mitigate them
Generative AI, Foundation Models and Large Language Models
A new age of AI is imminent, we see many tech companies including OpenAI, Google and Microsoft pushing hard for AI initiatives and especially generative AI initiatives. Following are some of the different technologies that has been propelling the forthcoming of AI:
Generative AI: This term describes AI systems whose primary function is to generate content. The content can be generated in any medium such as textual, visual and aural. Some examples of Generative AI systems include AI chatbots that generate textual responses to user’s prompts (e.g. ChatGPT), image generators that generate visual responses (e.g. Midjourney, Stable Diffusion)
Large Language Models (LLM): Language models in the Natural Language Processing (NLP) field are models that can process, analyze and generate natural language text. To put it simply, a language model is a model that has an understanding of a language or languages. With a good understanding of languages, these models can be utilized to be successful at many different language related tasks such as summarization, translation, and answering questions. A large language model, basically, refers to a language model that is large in terms of the data, computational power and parameters it has been trained with. Some examples of LLMs include OpenAI’s GPT-4, Google’s PaLM and Meta’s LLaMA. Products like OpenAI’s ChatGPT and Google’s Bard utilize LLMs to understand and interact with language provided by users.
Foundation Models: This term refers to AI systems trained on large and comprehensive data that can provide broad capabilities and be adapted to complete a variety of different and specific tasks, unlike other types of AI systems that are specifically trained and used for a particular purpose. Basically, a foundation model provides a base on which other products and systems can build on for their own use case. LLMs fit this description and are considered as Foundation Models, since their understanding of language can be used by different products and systems to complete a variety of different and specific tasks as in the case of ChatGPT which uses an adapted version of an LLM (GPT-3.5) for its foundation. Another example for Foundation Models is Stable Diffusion, which can be adapted for a variety of different visual tasks.
In the rest of this post, we will use the term “AI technologies” to refer to the technologies described above and we will specifically focus on textual applications of AI technologies for enterprises.
AI Technologies Can Revolutionize Enterprises
Now that we have a better understanding of the AI technologies that is rapidly shaping the business world, let’s take a look at some of the different aspects they can revolutionize enterprise systems and workflows:
Enhanced User Experience
One of the most significant impacts of AI technologies will be improvements in user experience. Instead of relying on complex user interfaces and manual input, through AI technologies, enterprises will allow users to interact with their systems and workflows using simple and intuitive language. This will make the software more accessible, more dynamic and user-friendly, lowering barriers to entry for non-technical users and increasing overall productivity.
Improved Efficiency and Collaboration
AI technologies will enable more efficient communication and collaboration within enterprises. Employees will spend less time learning how to use each piece of software and spend their precious time actually utilizing the software for its use cases. This will lead to reduced training time and costs for enterprises. Furthermore, these technologies will help bridge the gap across departments/teams and provide a more seamless collaboration within enterprises, contributing to the efficiency.
Cross-Platform Integration
Similar to bridging the gap across departments/teams within enterprises, AI technologies will also bridge the gap between different enterprise software systems by making it easier for users to communicate with multiple systems and share data across systems. This will help enterprises improve efficiency and productivity.
Streamlined Decision-Making Processes
By leveraging the power of AI technologies, enterprises can streamline their decision-making processes. AI technologies can analyze vast amounts of data in a short amount of time and generate insights and recommendations to assist enterprise leaders in making informed decisions. This way, enterprise leaders can understand the history and potential implications of their choices much easier and faster than ever before.
Enhanced Security and Compliance
AI technologies can enhance security and compliance in organizations by monitoring and analyzing large amounts of textual data in real-time, detecting anomalous patterns, and automating security checks. For example, real-time analysis of textual communication data can help identify potential security threats like phishing, compromised accounts, sensitive information sharing and insider trading.
While AI technologies can advance enterprise workflows in various aspects, simply deploying AI models pre-trained on public data as they are may not be sufficient to truly revolutionize enterprise workflows. To unlock the full potential of AI technologies, enterprises must harness their own proprietary enterprise data and utilize it effectively for their use cases. However, caution must be exercised when enterprises build their workflows and systems on top of AI technologies, since during the process enterprises can introduce biases into the AI technologies that can inflict serious harm on society. Therefore, in the pursuit of revolutionizing their workflows, one of the foremost priorities of enterprises should be to mitigate bias by ensuring the data and interactions they use to enhance AI technologies are representative, diverse, and devoid of discriminatory elements.
Mitigating Bias in AI Technologies
Bias in AI technologies can disproportionately benefit or harm different groups (ChatGPT and Generative AI Bias). To address these potential harmful impacts, mitigation strategies to reduce bias are important. By implementing effective bias mitigation strategies, enterprises can create systems and workflows that foster a more inclusive and just society where AI technologies are leveraged to empower and uplift all individuals.
In this section with the help of some examples we will explore and address different kinds of biases that could be introduced into AI technologies (specifically textual Generative AI technologies like ChatGPT and Bard), and strategies enterprises can implement to mitigate them.
Example 1: Writing a Python function
Here is an example of potential bias from OpenAI’s ChatGPT and Google’s Bard caused by biased user interaction (inspired by Steven T. Piantadosi). In this example the user asks AI models to “Write a python function to check if someone would be a good scientist, based on the college they attended”, the AI models respond with a python function that considers a scientist good if and only if the scientist graduated from a list of 5–10 colleges deemed as top science colleges by the AI. In this example, the prompt itself is biased. The user has an expectation that based on the scientist’s educational background one can make a decision on whether the scientist is good or not. Both ChatGPT and Bard identify and flag the bias in the user’s prompt, but they still generate a python function that favors certain colleges.
How to Mitigate Bias:
In this example, the main issue is the bias in the prompt itself (biased user interaction). Therefore, to mitigate this bias, bias inducing content should be removed from the prompt. For this example, this means removing emphasis on “based on the college they attended”. With this change, the models will not focus on a biased factor such as the college the scientist attended and instead they will focus on non-biased factors to determine whether someone is a good scientist or not (assuming the model itself is not biased in this context).
In order to be able to remove bias inducing content from the prompt, first the bias inducing content should be identified, which is not always trivial. There are a couple different approaches that could be taken to overcome this challenge. We believe one of the best approaches is to make sure users (developers of enterprise workflows) are educated about ethical prompt engineering and bias in AI Technologies. An educated user will be able to identify potentially bias inducing content and refrain from using them in prompts. Another beneficial approach could be to utilize or develop automated tools that can identify bias inducing content in prompts and warn users to refrain from using them.
In addition to removing bias inducing content from prompt, the prompt can also be reinforced with explicit constraints to discourage models from generating biased responses (assuming the model has a good understanding of bias). For example, the following phrase can be appended to the prompt: “Please make sure your response is not biased and does not discriminate against certain groups of people”. Using such explicit constraints will help the models to be mindful of bias when generating the response. We used this method for the bias inducing prompts in the example and observed that neither ChatGPT nor Bard provided the function to check if someone would be a good scientist, based on the college they attended, stating that using college as a deciding factor in this context would not be a fair and accurate assessment. Focusing on the explicit constraint for bias, the models were able to identify the bias in the prompt and refrain from generating biased responses.
Example 2: Asking a Partial Question
Let’s take a look at another example of Bias in ChatGPT caused by biased user interaction. This example demonstrates that for prompts on the same topic (formulated as questions), ChatGPT can provide answers in different sentiments depending on the partiality of the prompt. Here are the prompts used to ask ChatGPT on whether socially assistive robots are an effective treatment for older adults with dementia or not:
- Explain why socially assistive robots are an effective treatment for older adults with dementia.
- Are socially assistive robots an effective treatment for older adults with dementia?
- What are the benefits and drawbacks of socially assistive robots as a treatment for older adults with dementia?
Similar to the prompt in the example before, the first prompt makes an assumption that socially assistive robots are an effective treatment for older adults with dementia. ChatGPT’s response to this first prompt is guided by this assumption and the sentiment towards socially assistive robots being an effective treatment for older adults with dementia is very positive. On the other hand, in the second prompt there is no assumption and the prompt is asked in a neutral tone. We see that ChatGPT’s response to the second prompt is still mostly positive in terms of sentiment, but ChatGPT is more hesitant to make a claim that socially assistive robots are an effective treatment for older adults with dementia. Whereas, in the third prompt in addition to using a neutral tone, both benefits and drawbacks are being asked to ChatGPT. Since this time ChatGPT focuses on both the benefits and drawbacks, the sentiment in its response is the most neutral.
This type of issue is not something we are unfamiliar with. Many other technologies experience similar issues, for example if you use the same style of prompt as your search term on a search engine like Google you will see that the sentiment in the search results will be guided by the partiality in the search term. This example clearly shows us that even though AI technologies are pretty far advanced in terms of their complexity and capabilities, they still have limitations. As users it is very important for us to be reminded of these limitations and shape how we interact with AI technologies in a responsible way to mitigate bias.
How to Mitigate Bias:
Similar to the previous example, one of the most important mitigation strategies for this example is to remove the bias inducing content in the prompt. In this case we focus on removing the partiality in the prompt. This can be done by educating users to adopt the following best practices when creating prompts:
Avoid making assumptions: Assumptions can guide models towards a partial response. For this example, in the first prompt, the user makes an assumption that assistive robots are an effective treatment for older adults with dementia, causing the model’s response to be partial (positive sentiment in this example). As we have seen in the second prompt, avoiding this assumption helps the model provide a less partial response.
Encourage Different Perspectives: When models are asked to consider different perspectives such as the benefits (positive perspective) and drawbacks (negative perspective) in prompt 3 in the example, the model will be more likely to avoid generating partial responses.
Use Open-Ended Questions: Instead of using questions that suggest a specific answer or viewpoint in the prompt, use open-ended questions this way the models are encouraged to explore different aspects of the content in the question and provide a comprehensive analysis, without generating responses that fall to a particular bucket. For this example, in prompt 3 the user asks an open-ended question focusing on the benefits and drawbacks without listing any individual benefit and drawback. The open-ended question helps the model to provide an impartial response.
Example 3: Generating Job Description with Restricting Educational Qualifications
Our final example focuses on biased educational qualifications Google’s Bard provides when asked to write a job description for a position (a task showcased by Google itself for their Generative AI technologies). We tested Bard’s capabilities as a job description writer for various positions (e.g. software engineer, sales representative, construction worker) using the following format: Can you write a job description for [Position X]?
We observed that Bard was producing a pattern for the job descriptions and this pattern included an educational qualification for every single position we asked Bard to write a job description for. For some of the positions, the job description considered a bachelor’s degree as an educational qualification, whereas for other positions a high school diploma or equivalent. However, regardless of the position, in those samples, it always produced a minimum education qualification of high school diploma or equivalent, even for positions that do not expressly call for a high school diploma or equivalent such as construction worker and delivery driver. Job descriptions for these types of positions, sometimes do mention a high school diploma or equivalent as a required qualification, but sometimes only as a preferred qualification to encourage populations without a high school degree or equivalent to apply.
How to Mitigate Bias:
The mitigation strategies for the bias in this example, are slightly different than previous examples. Previous examples focused on issues caused by biased user interaction that could be resolved by removing bias inducing content from the prompt. However, this is not the case for this example. In this example, the model itself produces a response that can negatively impact a certain group of people (population without a high school diploma or equivalent). This example demonstrates that evaluating model responses is critical and that we should avoid using model responses as it is without any thorough evaluation.
After evaluating the model responses and identifying potentially biased content, effective post-processing mechanisms can be utilized or developed to address the biased content. These post-processing mechanisms can involve automated or manual processes that filter, modify or flag bias inducing content. For this example, a post-processing mechanism can be developed to replace the educational qualifications in job descriptions generated by models with more appropriate ones using a mapping of educational qualifications and positions gathered from various datasets and sources.
Another method to address the bias issues in model responses could be to reinforce the prompt with explicit constraints that discourages models from including bias inducing content in the response. For this example, we can append the following instruction to the prompt to prevent biased educational qualifications appearing in the job description generated: “Please do not include educational qualifications.”. We used this method for this example and observed that Bard no longer provided educational qualifications in the job description.
Bias Mitigation Strategies
So far, we described a few different examples of bias in AI Technologies and specific strategies to mitigate them. In this section, we want to share with you some strategies we adopted at Eightfold AI to mitigate bias in AI technologies, which can be helpful for other enterprises utilizing AI technologies to enhance their systems and workflows.
At Eightfold, we believe that AI systems should have the checks and balances to ensure that they are Fair and Equitable. In order to ensure that the implicit biases of the language models are not propagated to our applications we employ approaches from the following suite:
Prompt Engineering
For most applications incorporating AI technologies such as Large Language Models, prompts help improve the performance of the models in a variety of NLP Tasks. Coming up with the right prompt for your application is in itself a problem for which the solution space is extremely vast. While engineering for the prompt, we should not only focus on the accuracy of the response, we should also be mindful of the degree of inclusivity in the prompt.
Here are some guidelines around prompt engineering to mitigate bias when using large language models (LLMs):
- Use clear and specific instructions. The prompt should be clear and specific about what you are asking the LLM to do. This will help to avoid ambiguity and ensure that the LLM generates a relevant and unbiased response.
- Be mindful of your own biases in your assumptions and opinions, avoid using harmful stereotypes. The prompt should not contain any harmful stereotypes or biases. This could include anything that is racist, sexist, or otherwise discriminatory.
- Use a variety of prompts. Don’t just use the same prompt every time. This will help to ensure that the LLM is exposed to a variety of viewpoints and doesn’t become biased towards any one particular perspective.
- Use inclusive language. This means using language that does not discriminate against any particular group of people. For example, instead of saying “mankind,” you could say “humanity.”
- Test the LLM’s responses. Once you have generated a response from the LLM, test it to make sure that it is unbiased. This could involve asking a human to review the response or using a tool to detect bias.
- Use multiple LLMs. No single LLM is perfect, so it’s a good idea to use multiple LLMs to generate responses. This will help to ensure that you get a more balanced and unbiased view.
- Have humans in the loop. Once you have generated a response from an LLM, get feedback from multiple humans when testing the LLM’s responses.
PromptAI
PromptAI aims to improve the developers’ capabilities in searching for the right prompt. Currently the process is fairly manual and relies on heuristics. One could argue that we could use AI to improve the Prompt Engineering process itself. This is a fairly recent development in the field of Large Language Models. As stated above, the search space to find the right prompt might be an expansive one. Recent literature explores the idea of using gradient-descent based optimization methods to tune the prompts. This is an interesting approach as one can start with a seed prompt and tune the prompts to get the best possible response from a Large Language Model based on a specific set of metrics. Some of the approaches being explored include:
AUTOPROMPT
The problem with prompts is that they are often hand-crafted, which can be time-consuming and error-prone. Shin et. al. propose a method for automatically generating prompts for LLMs. Their method, called AUTOPROMPT, is based on a gradient-guided search algorithm.
The AUTOPROMPT algorithm works as follows,
- The algorithm starts with a set of seed prompts.
- The algorithm then iteratively generates new prompts by replacing tokens in the seed prompts with other tokens.
- The algorithm evaluates the new prompts by feeding them to a LLM and measuring the LLM’s performance on a downstream task.
- The algorithm repeats steps 2 and 3 until a set of optimal prompts is found.
Shin et. al. evaluated AUTOPROMPT on a variety of NLP tasks, including sentiment analysis, natural language inference, and fact retrieval. They showed that AUTOPROMPT can generate prompts that are as effective as hand-crafted prompts.
Prefix Tuning (P-tuning)
P-tuning is a technique for improving the performance of Large Language Models on natural language understanding tasks. The basic idea of P-tuning is to add a small, continuous vector to the beginning of the input sequence. This vector, called the prefix, is optimized to improve the performance of the PLM on the target NLU task. The prefix is typically initialized to a random vector, but it can also be initialized to a human-crafted prompt.
In the paper “GPT Understands, Too” by Liu, Zheng et. al., the authors compared P-tuning to fine-tuning on a variety of NLU tasks. They showed that P-tuning can achieve comparable performance to fine-tuning with significantly less labeled data. Additionally, P-tuning is more robust to noisy or incomplete data than fine-tuning.
Soft Prompts
The key idea is to use a mixture of soft prompts, which are continuous vectors that are not necessarily word type embeddings from the language model. The mixture of soft prompts is optimized to improve the performance of the LLM on the target task.
One of the key benefits of the proposed approach is that it is more flexible than traditional prompt-based methods. Traditional prompt-based methods typically use a single prompt, which can be limiting. The proposed approach, on the other hand, can use a mixture of soft prompts, which allows the LLM to be more creative and informative in its responses.
One could follow the following steps to utilize this approach,
- Choose a pre-trained language model (LLM). Any pre-trained language model can be used with mixture soft prompts. However, some models may work better than others for certain tasks.
- Generate a set of soft prompts. Soft prompts are continuous vectors that are not necessarily word type embeddings from the language model. They can be generated randomly or using a variety of methods, such as grammar rules or natural language processing techniques.
- Initialize the mixture weights. The mixture weights are used to determine the contribution of each soft prompt to the final output. They can be initialized randomly or using a variety of methods, such as grid search or Bayesian optimization.
- Train the mixture soft prompt model. The mixture soft prompt model is trained using a supervised learning algorithm. The input to the model is the mixture of soft prompts, and the output is the desired response from the LLM.
- Use the mixture soft prompt model to query the LLM. Once the mixture soft prompt model is trained, it can be used to query the LLM. To do this, simply provide the model with the mixture of soft prompts, and it will return the desired response from the LLM.
Output Processing
With output processing, responses generated by AI technologies undergo sets of checks and balances to minimize biased output and leakage of sensitive information. These checks may be in varying degrees of complexity: For example, a simple regex pattern that is used to identify and replace a set of information, to a language model that is trained to identify sensitive information, suffices in many cases.
Below is an example, in which a job description is processed with a simple regex pattern check to identify and replace exclusive phrases that can discourage people with differing backgrounds from applying to the position the job description is for. Such an output processing mechanism can be used to ensure exclusive phrases are not present in a job description created by AI technologies.
Developer Guidelines
In addition to the strategies mentioned so far, following the developer guidelines listed below would be helpful for enterprises to mitigate bias when applying AI technologies to enhance their systems and workflows.
Human Oversight for AI Generated Content:
Capabilities of generative AI technologies are vast, however they still have their limitations. By implementing robust and critical evaluation processes overseen by humans, we can identify potential biases in AI-generated content, and take necessary corrective actions to promote that the information and narratives generated by AI systems are fair and inclusive.
User Feedback and Iterative Improvement:
Engaging users and soliciting feedback is a valuable approach to address bias in AI technologies, especially for generative AI technologies that rely on interactions with users. Encouraging users to examine and report any biased output they encounter allows issues that may have been overlooked during the development phase to be identified and help developers refine systems and workflows in an iterative way to ensure they are more inclusive and unbiased.
Diverse Development Teams:
To mitigate biases, it is essential to foster diverse, inclusive and multidisciplinary development teams. By assembling teams that encompass a variety of backgrounds, experiences, and perspectives, we can foster a wider range of viewpoints to be considered during the development process and minimize the risk of inadvertent biases when applying AI technologies for enterprise systems and workflows.
Continuous Monitoring and Evaluation:
Bias is a dynamic and complex issue that can evolve over time. Therefore, bias mitigation should not be a one-time process; it should be an ongoing effort. Continuous monitoring and evaluation of enterprise systems and workflows using AI technologies is crucial to identify and rectify biases that may emerge over time.
Regular Auditing and External Review:
Independent audits and external reviews can provide valuable insights into fairness and potential biases of enterprise systems and workflows using AI technologies. Engaging external experts, academic institutions, or organizations specializing in AI ethics can help evaluate and challenge the assumptions and algorithms used. These reviews serve as a safeguard, ensuring that the deployed systems and workflows align with ethical standards.
