The data economy can be a Catch-22. It can succumb to corporate surveillance capitalism on the one hand and an authoritarian digital “welfare” state on the other. The European Union (EU) places itself as the alternative to both. Its strategy to regulate technology over the next decade is to set that precedence. Whether it is successful is up for interpretation. On 19th February 2020, the European Commission (executive branch of the European Union) published a 26-page whitepaper on Artificial Intelligence (AI). The paper titled a European Approach to Excellence and Trust states the EC’s intent to regulate and advance AI.
This blog will explore the reach, requirements, and reservations of the guidelines the whitepaper introduces.
Reach: A Risk Barometer Approach
The whitepaper will have consequences for those using and developing AI. To be specific, businesses that are participants of the data economy. It’s drafted to effectively regulate AI while not being dictatorial. Strict measures could create a disproportionate burden for SMEs.
The paper defines AI as
“Systems that display intelligent behavior by analyzing their environment and taking actions — with some degree of autonomy — to achieve specific goals.”
However, the proposed requirements will mainly affect AI which is deemed “high-risk”. This is enumerated by the EC as:
“… deployed in health care, transport, energy and parts of the public sector, or if it is used in the employment sphere (for recruitment purposes or in situations impacting workers’ rights), or for remote biometric identification and other intrusive surveillance technologies.”
Due to this definition and set scope, the suggestions would not apply to advertising technology or consumer privacy. The assumption here is that risk can be finitely calculated. This leaves many contentious issues outside of the purview of the guidelines. For example, data brokers that leverage AI to predict identities and hyper-targeted advertising.
It is anticipated that the new framework will have extraterritorial impact, like the GDPR.
Requirements: The Precursor to Compliance
The AI applications classified as high-risk would be regulated by the following key features. These center on safety, security, fairness and transparency:
- Training data
The paper reiterates that if there is no data, there is no AI. The decisions and performance of an AI are dependent on the data sets it has been fed and trained on. To ensure that the services or products that the AI system enables are safe, the requirements dictate that it must be trained on a broad enough data set. The training data must also be representation to avoid inadvertent coded discrimination. The data collected to adhere to privacy and data protection standards i.e. the GDPR. (Interested in reading more on the data protection regulations in place in the EU and India? Take a look at our article comparing the GDPR and PDP Bill) - Data and record-keeping
Considering the opacity and complexity of many AI systems, certain requirements are put forth to verify compliance. It could allow potentially problematic decisions or actions by the AI to be traced back. The regulatory framework proposes that the following records can be kept:
a. Records related to the programming of the algorithm
b. Data sets used to train and test the high-risk AI systems (when justified) along with a description of their main characteristic and the reason for their selection
c. Documentation on the algorithm and the training methodologies adopted to build, test, and validate the AI - Information to be provided
Apart from the above information, the AI system’s limitations and capabilities must be proactively provided. It should also mention the degree of accuracy to which the system can achieve a specific purpose. This information could be useful to those deploying the AI application. The whitepaper reiterates that citizens should be duly informed when they are interacting with an AI and not a real person. The details should be easy to understand, concise and objective. - Robustness and accuracy
Across the AI system’s life cycle, it must correctly reflect its own degree of accuracy. The whitepaper mentions that the outcomes should be reproducible. The AI system must be able to deal with errors and inconsistencies. It should endure overt attacks, and be resilient against manipulated data. - Human oversight
The AI system must be ethical and trustworthy. To not undermine human autonomy, the whitepaper insists on the AI being human-centric. This could manifest in different ways depending on the system’s purpose and functioning:
a. Output is reviewed and validated by a human before it becomes effective. For example, human intervention needed to approve a person’s KYC.
b. Human intervention post the output being effective. For example, reviewing why the AI rejected a credit application, after the decision was put into effect.
c. Monitoring the operation of the AI system. This is with the possibility to intervene and stop its functioning in real time. For example, a deactivate button in a driverless car.
d. Constraints integrated during the design phase. For example, a driverless car will stop when visibility is low. - Specific requirements (Example: For AI applications used for remote biometric identification)
The application of AI systems for functions such as facial recognition affects the fundamental rights of a citizen. For example the right to a private life and the protection of one’s personal data. Processing of biometric data is to uniquely identify a person. This can only be done in special circumstances with adequate safeguards. The whitepaper declares that the EC will begin a “broad European debate”- on what these circumstances are and their justification.
Reservations: Missing the Mark
The proposed guidelines address issues of personal data protection and pivacy rights, non-discrimination, and cybersecurity. But, it seems to miss the perils of “low-risk” technologies with weakened guidelines.
The whitepaper overlooks that the classification of low risk is not absolute. This could actually be very risky for some. The harms of technology are often amplified to disproportionately affect the marginalized.
A draft version of the whitepaper was leaked in January. Held against that, the new criteria are feeble attempts to regulate the possible adverse implementations of AI. Here, the draft proposed a prohibition or what is called a “moratorium” on facial recognition in public spaces for 5 years. But, the released guidelines are merely a call for a “broad European debate” on the facial recognition policy.
Stakeholders can give their insights on the whitepaper by 31st May 2020. The EC will start drafting legislation based on the proposal and feedback at the end of 2020.
About Signzy
Signzy is an AI powered RPA platform for financial services. No matter how complex your workflow or operational complexity, Signzy is able to completely automate your back-operations decision making process into a real-time API. This is possible due to a combination of Nebula — Our no code AI model builder and our Fintech API Marketplace of over 200+ APIs. Today we work with over 90+ FIs globally including the 4 largest banks in India and a Top 3 acquiring Bank in US. Globally we have a strong partnership with MasterCard and offices in New York and Dubai to serve our customers in the 2 geographies. Our Product team of 120+ people is building a global AI product out of Bangalore.Visit www.signzy.com for more information about us.
You can reach out to our team at reachout@signzy.com
References
Note: The references below were used for preliminary research for this article.
[1] https://ec.europa.eu/info/sites/info/files/commission-white-paper-artificial-intelligence-feb2020_en.pdf
[2] https://www.mondaq.com/unitedstates/Technology/904878/10-Things-You-Need-To-Know-About-The-EU-White-Paper-On-Artificial-Intelligence
[3] https://www.technologyreview.com/f/615255/european-union-artificial-intelligence-regulation-facial-recognition-privacy/
[4] https://www.forbes.com/sites/carmenniethammer/2020/03/02/ai-bias-could-put-womens-lives-at-riska-challenge-for-regulators/#24d9bee6534f
[5] https://www.lexology.com/library/detail.aspx?g=a6b8110c-3b70-4ff0-93b8-88c3640a960a
[6] https://tech.newstatesman.com/guest-opinion/eu-white-paper-on-artificial-intelligence-falls-short
