The ‘Black Box’ Problem of AI

Artificial intelligence (AI) now affects every aspect of our social lives. Without always being aware of it, we interact on a daily basis with intelligent systems which optimize our journeys, create our favorite playlists and protect our inboxes from spam: they are our invisible workforce. At least, this is the role we have assigned to them: improving our lives, one task at a time.

Recent progress in AI’s several fields (driverless cars, image recognition and virtual assistants) and its growing influence on our lives have placed it at the center of public debate. In recent years, many people have raised questions about AI’s actual capacity to work in the interests of our well-being and about the steps that need to be taken to ensure that this remains the case. This debate has principally taken the form of a broad discussion about the ethical issues involved in developing artificial intelligence technology and, more generally, in the use of algorithms. In different parts of the world, experts, regulators, academics, entrepreneurs and citizens are discussing and sharing information about the undesirable effects — current or potential — caused by their use and about ways to reduce them. Faced with the need to take respect for our values and social standards on board when addressing the potential offered by this technology, these discussions have logically drawn on the vocabulary of ethics. They occupy the available space between what has been made possible by AI and what is permitted by law, in order to discuss what is appropriate. However, ethics is clearly a branch of philosophy which devotes itself exclusively to the study of this space by attempting to distinguish good from evil, the ideals to which we aspire and the paths which take us away from them. Furthermore, aside from these purely speculative considerations concerning AI’s ‘existential threats’ to humanity, debates tend to crystallize around the ‘everyday’ algorithms which organize our news feeds, help us decide what to buy and determine our training routines.

We are not all equal before these algorithms and their partiality has a real impact on our lives. Every day, invisibly, they influence our access to information, to culture, to employment or alternatively to credit. Consequently, if we hope to see new AI technology emerge that fits in with our values and social standards, we need to act now by mobilizing the scientific community, the public authorities, industry, the entrepreneurs and the organizations of civil society. There are a few ways in which we can start building an ethical framework for the development of AI and keep this discussion going in our society. These are based on the following principles:

- In the first place, there needs to be greater transparency and auditability concerning autonomous systems. On the one hand we can achieve that by developing our Aside from these purely speculative considerations concerning AI’s ‘existential threats’ to humanity, debates tend to crystallize around the ‘everyday’ algorithms capacities to observe, understand and audit their performance and, on the other, through massive investment in research into their accountability.

- Next, the protection of our rights and freedoms needs to be adapted to accommodate the potential for abuse involved in the use of machine learning systems. Yet it appears that current legislation, which focuses on the protection of the individual, is not consistent with the logic introduced by these systems — i.e. the analysis of a considerable quantity of information for the purpose of identifying hidden trends and behavior — and their effect on groups of individuals. To bridge this gap, we need to create collective rights concerning data.

- Meanwhile, we need to ensure that organizations which deploy and utilize these systems remain legally responsible for any damages caused. However, legislation cannot solve everything, partly because it takes much more time to generate law and norms than it does to generate code. It is therefore vital that the ‘architects’ of our digital society — the researchers, engineers and developers who are designing and commercializing this technology — do their own fair share in this mission by acting responsibly. This means that they should be fully aware of the potentially negative effects of their technology on society and that they should make positive efforts to limit these.

- In addition, given the important nature of the ethical questions that confront future developments in AI, it would be prudent to create a genuinely diverse and inclusive social forum for discussion, to enable us to democratically determine which forms of AI are appropriate for our society.

- Finally, it becomes more crucial to politicize the issues linked to technology in general and AI in particular, in view of the important part it plays in our lives.

A large proportion of the ethical considerations raised by AI have to do with the obscure nature of this technology. In spite of its high performance in many domains, from translation to finance as well as the motor industry, it often proves extremely difficult to explain the decisions it makes in a way that the average person can understand. This is the notorious ‘black box problem’: it is possible to observe incoming data (input) and outgoing data (output) in algorithmic systems, but their internal operations are not very well understood (see inset). Nowadays, our ignorance is principally due to changes in the paradigm that is introduced by machine learning, in particular deep learning. In traditional computer programming, building an intelligent system consisted of writing out a deductive model by hand, i.e. the general rules from which conclusions are inferred in the processing of individual cases. Such models are by definition explainable, in as much as the rules which determine their decision-making are established in advance by a programmer, and it is possible to tell in each individual case which of the rules have been activated.

On the other hand, being one of the most efficient machine learning technique today, deep neural networks (Deep Learning), does not rely on rules established in advance. In the case of image recognition, if we wanted to develop an algorithm which automatically categorized photos of cats and dogs, the data being processed would consist of images in the form of an array of pixels and it is virtually impossible to write out a programme by hand that is sufficiently powerful to classify all the images accurately from the data, pixel by pixel. At this stage, the accountability of systems based on machine learning thus constitutes a real scientific challenge, which is creating tension between our need for explanations and our interests in efficiency. Yet, although certain models of machine learning are more easily explainable than others (systems based on rules, simple decision trees and Bayesian networks), nowadays their performance does not generally match up to that of deep learning algorithms.

Neural networks and deep learning techniques are routinely condemned by their users for seeming just like black boxes. This argument can be equally applied to a large number of other machine learning techniques, whether we are talking about Support Vector Machines or random forests (the operational versions of decision trees). The reason is not so much inherent in the nature of the model used, yet resides more in a failure to produce an intelligible description of the results produced in each case and, in particular, to highlight the most important features of the case in question that have led to these results. This failure is largely due to the dimensions of the spaces in which the data are evolving, which is particularly crucial in the case of deep learning. For example, for image recognition, a deep network inputs images described by thousands of pixels (4K) and typically memorizes hundreds of thousands, even millions, of parameters (network weights), which it then uses to classify unknown images. It is therefore almost impossible to follow the path of the classification algorithm, which involves these millions of parameters, to its final decision. Although in terms of one image, this accountability seems of relatively low importance, it is a lot more crucial in the granting of a loan, for example. In the long term, the accountability of this technology is one of the conditions of its social acceptability.

As a society, we cannot allow certain important decisions to be taken without explanation. In fact, without being able to explain decisions taken by autonomous systems, it is difficult to justify them: it would seem inconceivable to accept what cannot be justified in areas as crucial to the life of an individual as access to credit, employment, accommodation, justice and health.

Say: “My Lord has commanded justice…” (Qur’an, 7:29)