New Norwegian Guide to Prevent AI Discrimination Launched With Minister of Digitalisation
The Norwegian Equality and Anti-Discrimination Ombud Launches a Guide for Built-in Protection Against AI Discrimination
The Norwegian Equality and Anti-Discrimination Ombud (Likestillings- og diskrimineringsombudet, LDO) just launched a guide to uncovering and preventing discrimination in the development and use of artificial intelligence. In Norway citizens have strong legal protections to protect against discrimination and attempt to ensure equality. In this article I will attempt to briefly cover the launch and some key points from the guide. Please note that the original guide is released in Norwegian, however I will do my best to give you a summary in English. I will also translate part of the text and the illustrations.
The launch of the guide in Norway
The launch was attended by the Norwegian Minister of Digitalisation and Governance Karianne Tung.
The guide was launched on the 8th of November 2023.
The guide was to a large extent written by Kathinka Theodore Aakenes Vik, senior advisor at LDO.
From the press release it is mentioned that:
“The danger of discrimination is among the biggest risks when artificial intelligence is used to make decisions about individuals — for example who should receive social security benefits, health care or credit loans. At the same time that the use of AI is increasing, surveys show that there is low awareness and little competence about discrimination in both the public and private sectors. The Equality and Discrimination Ombudsman is therefore launching a guide on how development teams can assess the risk of discrimination in AI systems.”
Background for the guide
Throughout the last decade, the media have reported scandals where algorithms and machine learning models have led to discrimination, often because the technology has not been assessed thoroughly enough. Two examples of this are facial recognition that does not work adequately on people with darker skin tones, and algorithms used to detect fraud that falsely identify people with an immigrant background.
The examples show that it is not sufficient to keep the risk of discrimination in mind when developing and using this technology. Discrimination must be prevented and measured systematically. Measures to prevent discrimination and promote equality must be built into all the development phases of a machine learning system (ML system), from planning to use of the technology. We call this built-in protection against discrimination .
You can find the full guide (in Norwegian) here:
So who is this guide for?
Target group: those responsible for the development, purchase and use of machine learning systems where the use implies that the systems may have an impact on people’s rights and obligations.
What is it based on?
The guide is, among other things, inspired by “Nondiscrimination by design”, Fundamental Rights and Algorithm Impact Assessment (FRAIA) and “Promoting equality in the use of Artificial Intelligence — an assessment framework for non-discriminatory AI”.
Why is it crucial to consider?
Norwegian law prohibits discrimination in all areas of society. The prohibition is based on the principle of equality and non-discrimination, which is enshrined in Section 98 of the Constitution. This principle is also central to the European human rights convention (ECHR) and several other human rights conventions. Additionally there are a lot of overlapping possibilities for discrimination that may need careful navigation.
To assist people working with development, purchasing or use of machine learning systems the guide works to address several stages.
The guide is structured in phases with relevant discrimination challenges and case examples.
1. Planning
The first phase is about putting the spotlight on the problem that the system is supposed to solve, defining what is the goal, success criteria and which social and political consequences the use may have in an equality and discrimination perspective.
Relevant discrimination challenges
The intended use of the ML model can be decisive for the risk of discrimination. If the ML model is developed to calculate the starry sky and that of the planet localization, this guide is probably irrelevant. If the ML model is developed to predict which citizens who are to be granted welfare benefits are, on the other hand the question of discrimination applicable. Furthermore, within the models that can have an impact on people’s rights and duties, there is a difference between ML systems that are used for:
- Control purpose: The system is intended to control, and can be used to sanction people. The system can discriminate by checking certain parts of the population disproportionately scope or more in-depth than others.
- Allocation purpose: The system aims to provide citizens with better and correct services. Such systems can discriminate if they work less precisely for certain groups. This can, for example, happen within the health sector, in that certain groups of people do not receive equivalent public services that they have a claim to receive.
They raise a range of questions in the guide.
Purpose of the model:
- What is the system’s intended use?In what context should it operate?
- To what extent is the system autonomous?
- Which groups of people are differentiated and why?
- Have representatives from these groups of people been involved and heard during planning and design of the system?
The model’s effect:
- What significance can the system have for certain people?
- Is it used (as part of a process) to determine the legal status of individuals?
Control purpose: Control of persons? Like predicting possible future behaviour?
- Who is affected by the model?
- Will some groups of people be extra exposed to discrimination during inspections?
Award purpose: Improve (access to) services for people?
-Which groups of people will be affected by the model?
- What significance can the system have on a societal level?
- Can the system improve previous practices in terms of the occurrence of discrimination or other types of errors?
Success criteria: How should success be measured in terms of efficiency or increased precision?
What do the success criteria mean for different groups of people?
The guide also provides specific examples from a Norwegian context in this section.
2. Training data
This phase is about ensuring proper collection, processing and use of data. Training a machine learning model requires large amounts of data and it is on background of this data that the machine learns to recognise patterns. The data will be decisive for which connections the system detects, and which predictions the system provides
Current issues that should be discussed
- Need for data: Collection
-What data is needed to achieve the purpose with the model?
-Does the business have access to the data internally, or must it be obtained from external sources sources? - Data quality: Map the data basis
-Is the data base set up representatively against the model’s purpose? Are some groups over- or under-represented?
-What could be the consequence of failure representation?
-Can some groups of people have deviations data patterns in relation to what the model calculate? Take the grounds of discrimination as a starting point.
3. Model development
This phase is about how the model will operationalize the purpose of the model. The decisive thing in this phase is that what the model calculates corresponds to what you want to achieve.
Relevant discrimination challenges
A typical source of discrimination is that it the model calculates does not correspond to it real purpose. This can be referred to as measurement bias. The challenge may arise if the purpose of the system is not directly observable or measurable. The data and variables used in the system thus constitutes a simplification of the overall goal, and there may be a risk of more weaknesses in the system as a whole, including a risk of discrimination.
Current issues that should be discussed
- What should the model calculate?
- To what extent does the calculation correlate with the overall purpose
- Which variables is the model’s calculation based on and why are these relevant?
- If the variables can be linked to a grounds for discrimination — can factually purpose, necessity and proportionality be detected?
- Is one model sufficient or should you develop more models that can compared with respect to any biases and justice?
4. Testing the system
This phase is about how the system should be tested before it is implemented. As the testing must be limited in scope and time, it is essential that the system is tested for relevant risks.
Relevant discrimination challenges
Even if the system does not use personal data that constitutes a basis for discrimination, it is there an imminent danger that others, apparently neutral information can reveal connections which coincide with the grounds of discrimination. The reason for this is that machine learning models often are superior at uncovering relationships, but have limited ability to distinguish between causality vs. correlations.
Research shows that it can be useful to keep the personal data that is directly captured by the grounds for discrimination in order to be able to test the system and survey whether different groups of people fare worse than others.
A particular challenge that must be taken into account during testing of the system, the above phenomenon is compounded discrimination.
If the system is tested for discrimination at group level completely above (such as gender), discrimination can be more finely masked — then group level could be overlooked. The combinations of grounds for discrimination are, so to speak, endless (e.g. different age groups, gender, different variations of functional impairments, different ethnicities).
This demonstrates the importance of knowledge of social conditions and the assumptions of different groups in the context in which the system must function, in order to be able to test the system for probable weaknesses.
Current issues that should be discussed
Carry out testing of the system:
- How does the model perform against the success criteria defined in phase 1? As such:
-How does the system perform in terms of false positives/false negatives for different groups? — compare the results for the various groups.
-Is data to be able to check for discrimination for different groups available?
-Who is responsible for following up on the model’s performance on these points? - How are representatives for them involved affected in the testing phase?
Correlations or causality:
Document background for connections
- What are the underlying reasons for the predictions the system gives?
- Investigate whether linking of data can derive personal data that can be linked to the grounds of discrimination.
-If yes — what is the rationale for this?
-Consider factual purpose, necessity and proportionality.
5. Implementation
This phase is about how the system is taken into account use. The phase assumes that the testing of the system (phase 4) gives satisfactory results.
Relevant discrimination challenges
If the mapping of the questions in the preceding phases indicates that the system involves a risk of that it will work worse for certain groups, or treats certain groups more severely than others, and this cannot be adjusted for in the system itself, it should investigated whether there is an opportunity to offer one alternative treatment that gives the relevant groups a equal treatment. If that ML system is implemented it is crucial that there is supervision of how the system performs over time. This technology is dynamic and changes based on the experience systems make the time it is put into use. Therefore, a non-discriminatory model can develop discriminatory properties over time.
Current issues that should be discussed
The supplement to the model:
- Can discriminate calculations performed of the system is compensated for in the face of relevant groups of people?
Control:
- How to ensure real human verification of the model’s individual elements decisions?
- How is structural control ensured with the system? As such:
-Control mechanisms for the model provides equally accurate decisions for everyone groups, and
-Control mechanisms to ensure that the model does not systematically process certain groups stricter.
Request for information (innsyn) and communication:
- Who should have knowledge of the system and its application be open and available to? Map what information the various groups of stakeholders need to ensure trust society, and what guidelines it lays down how the information is made available.
- How are those affected by the system looked after their interests? As such:
-Necessary insight into how the model works?
-What discrimination assessments that is done?
-Are there real appeals?
Evaluation:
- Define an evaluation strategy (continuous or periodic), and preferably involve external experts, and interest organizations that can represent those affected.
- How would the system have worked with a alternative model, justice definition or algorithm?
- Based on the evaluation, should the system be used further, adjusted or terminated?
Special thanks in the development of the guide
The author gives thanks to Inga Strümke (NTNU), Helga Brøgger (DNV), Robindra Prabhu (NAV), Iris Bore (NAV), Rita Gyland (NAV), Jacob Sjødin (NAV), professor Dag Elgesem (UiB) and Vera Sofie Borgen Skjetne (BufDir), who has given input in the process of developing this guide.
Hope you found this useful, and it brings you up to speed with some of the developments that are ongoing in Norway related to AI policy and ethics.
This is also part of my personal project #1000daysofAI and you are reading article 519. I am writing one new article about or related to artificial intelligence for 1000 days. The first 500 days I wrote an article every day, and now from 500 to 1000 I write at a different pace.
