Aug 2, 2022
AI Ethics Playbook
Using Data for Good and Taking the Mystery Out of AI Decision-Making
What should AI and machine learning do for us today? With AI expected to become a $360 billion industry by 2028, technology leaders in virtually every industry are trying to figure out how to use AI and machine learning for everything from automation to robotics to decision-making.
The technology behind these advances, however, are often not well understood and there’s little federal regulation in how they’re developed and used. As a result, ethical AI practices need to be at the center of product development and business processes to ensure these technologies deliver on the promise of driving positive change for industries and society at large.
At RS21, we’ve created an AI Ethics Playbook that provides clear oversight and guidance that stakeholders involved in the development and use of AI and machine learning products can apply to navigate opportunities, limitations, and other considerations.
The goal is to take the mystery out of AI decision-making, by:
- Helping customers understand the value and limitations of AI
- Reflecting on and thinking through the impact of our work
- Establishing and implementing best policies and procedures for high-impact AI and machine learning solutions
- Being transparent about our AI processes and model results
By taking these steps, we can help ensure that AI and machine learning are used for the benefit of all and align with our values and use data for good.
Operationalizing Ethical AI
Our playbook came about from our AI Ethics Working Group, a team that wanted to go beyond writing an AI ethics manifesto, and instead focus on creating a clear and repeatable framework. We’re sharing how we created our playbook in hopes that it will help others operationalize ethical AI within their organizations.
“There are a lot of inherit unknowns and things we need to be mindful of in AI and machine learning — from understanding model limitations to reflecting on data and data drift,” said Dr. Michelle Archuleta, Director of Data Science and team lead of RS21’s AI Ethics Working Group. “Our group is really focused on taking a serious approach to ensuring the proper infrastructure and processes are in place for solutions that will minimize bias and be socially beneficial.”
So what’s in a playbook?
While the playbook continues to evolve, we prioritized developing working frameworks for four plays and have a roadmap to add more. Each play includes instructions on how to put it into action, as well as templates, tools, and resources.
RS21's four current plays are briefly described below.
1. Identifying Bias in Data Sets
A framework for evaluating bias in data sets that can creep in through things like sampling issues (when the data set does not accurately represent a population), selection bias (when certain data is favored and the data set does not represent a random selection), and measurement bias (when data is not measured accurately). Bias in data sets can consequently affect models.
2. Evaluating Bias in Model Prediction
This play focuses on the full machine learning lifecycle, so bias is addressed before models are fully deployed. There are many sources of bias that may arise when developing predictive models — from the data set to model selection. For a deeper look at bias in predictive models, including an example of how it affected Amazon’s AI hiring tool, check out this article in Towards Data Science.
3. Evaluating the Performance of Machine Learning Models
Documenting model performance and defining quantitative metrics helps us understand if a model is accurate, efficient, and robust. This is especially important for models that are used in decision-making, prediction, and other highly consequential applications. This play creates a framework for creating and refining models and making results transparent.
4. Client Communications for Explaining the Value, Limitations, and Proper Use Cases of a Model
The people and organizations that ultimately use AI and machine learning models need a clear understanding of what models can and cannot do. Our client communication framework helps teams navigate important conversations and activities from problem definition and determining if appropriate data sources are available through deployment in the end user’s environment.
Each play consists of six subsections to make it easy for any RS21 team to pick up and use any play. The general format of our playbook was inspired by Atlassian’s Team Playbook, in which each topic is broken down into step-by-step instructions. Here’s how we break down our plays:
Topic of Play
- About This Play + Why it Matters
- Who Should be Involved
- When to Run the Play
- What the Play will Produce
- How to Run the Play
- Appendix with supporting information, templates, and resources
We heavily emphasize deliverables for each play so there is thorough documentation and measurable outcomes that ultimately benefit customers by providing educational and transparent communications and products.
AI Ethics in Action
RS21 implements these plays across domains and from proposals and roadmaps to MLOps and final products. Our Working Group continues to develop supporting documentation and refine plays for continuous process improvement, and we plan to develop additional plays in the future.
“The AI Ethics Working Group provides a forum that allows us to address a variety of technical and domain-specific topics,” said Kameron Baumgardner, RS21 Chief Technology Officer. “We take what we learn from implementing these plays on projects and feed it back into the process to make us better next time.”
One of the major benefits we’ve seen of AI ethics in action is that all customers, including those very knowledgeable about AI and machine learning and those who are just getting started in their AI and data-driven journeys, benefit from clear processes and communication in the development of new technology.
For example, we’re able to scope and define products with upfront discussions, build in time to develop the right models with the right data, and help customers understand model results with explainable AI and machine learning.
While our plays are specific to our processes, we do want to share a template that you can adapt for your team and projects.
➡️ Get the template
We hope this conversation sparks ideas and would love to hear your feedback on what has worked, or hasn’t worked, for you. Please drop a note with questions or suggestions.
About RS21
RS21 is a rapidly growing data science company that uses artificial intelligence, design, and modern software development methods to empower organizations to make data-driven decisions that positively impact the world. Our innovative solutions are insightful, intuitive, inspiring, and intellectually honest.
