Days 4–5: Just What Is In That AI Application?

Because we missed day 1 (due to Omicron!) we made it up this week, so we got a little ahead of our readings. So we are combining the blog posts for days 4 and 5 into one post.

Part 1: Reflection

On day 5, we discussed these readings from day 3:

• Slota, Stephen C., et al. “Good systems, bad data?: Interpretations of AI hype and failures.” In Proceedings of the Association for Information Science and Technology 57.1 (2020): e275.
• Abdulkareem, Musa, and Steffen E. Petersen. “The promise of AI in detection, diagnosis, and epidemiology for combating COVID-19: Beyond the hype.” Frontiers in Artificial Intelligence 4 (2021): 53.

Some of the discussion we had:

• The legal framework around AI seems very behind the growing development of AI. I think that maybe AI applications should go through some sort of verification to check for unintended harm, similar to how experiments with humans require IRBs to review them. Some of us didn’t know what an IRB is, but one of us was able to explain. Indeed, in the USA, if your research receives funding from the federal government, and your research involves data from human participants, and no other exceptions apply, then an IRB may verify whether it is likely to cause harm. However, that’s a lot of “ifs”, and even then not many IRBs have people trained in AI who can understand the potential risks of big data. As a first step, the paper suggests that companies should have interoperable and shared databases for their AI applications and models, even if that data is private or proprietary. Someone said, To me this seems like a good idea but I also wonder how plausible this is as I am not too sure if certain companies would have to protect the data they have collected. Also people who have participated in contributing to the databases could feel uncomfortable. Yes, this might indeed conflict with standard data handling practices expected by ethics review boards like IRBs.
• If a clinician decided to use AI and there was a mistake, who gets sued for malpractice? The AI cannot be sued but does the clinician get sued for choosing to use it? We had quite a discussion on this topic. Someone said, there is definitely opportunity for people developing AI to avoid responsibility in the eyes of the general public between the idea here, that data determines consequences, and in the fact that AI is often seen as intelligence beyond a human, and therefore beyond whoever creates it. However, on days 1 and 2 we highlighted that every step along the way, the AI is checked or constructed by a human. There can be no AI without the human component.
• So, if IRBs don’t cover much AI research, and lawsuits can’t clearly assign responsibility, what laws and regulations are there around AI? We took a whirlwind tour of emerging regulation from the EU, the USA and China. We also looked at how these regulations are affected by the balance between values that different cultures hold.
• Then, we turned to whether and how AI researchers can bear these regulations in mind as they develop AI solutions. It is easy for us as critics of AI applications to have the benefit of hindsight and think that the researchers should have thought of all possible outcomes. But it seems difficult to think of all possible adversarial attacks. Someone suggested that there is a role here in ML process that is missing. Perhaps a job that can be created where someone is focused on the “complex view of their interactions”. That seems like a great idea; to have a role for creative, critical thinkers who are outside the software engineering part of AI, but inside the loop of AI research or product development, and who can think about possible benefits and risks.

We want to raise for the reader questions like these:

• What might be an example of an AI application that could be biased without having a negative impact?
• It seems to me that attempting to translate human values into AI systems is arguably a bigger limiting factor than technology itself?
• If AI is not inherently neutral, whose responsibility is it to ensure that systems do not have bad implications for certain people?

Part 2: AI Solutions Derive From Data

Our basic goal in days 4 and 5 was to walk back through the AI application to the model(s) inside it and the data used to train them. We aimed to answer questions like:

• What is an AI model? What is it made from and how does it get made?
• Where does the data come from for AI?
• Where does the truth come from for AI?

To answer these questions, first we did a data treasure hunt! We split into our groups and each group listed as many data sets and models as they could find that are relevant to that group’s selected application. Groups started with these possible model and data sources:

• Papers With Code
• Hugging Face Models and Data
• ML conference websites like ICMI, ICDAR and INTERSPEECH
• Official data repositories like the Linguistic Data Consortium

The winning team found eleven data sets in one hour!

Then, we talked about where the labels come from. For most AI solutions, you need at least some data that has been labeled with “truth”, if only to evaluate how well the solution works. We talked about two basic sources of labels:

1. Humans provide them. For example, the researchers themselves may label data, or they may collect data from crowdsourcing platforms like AMT or Appen. On these platforms, people do microtasks — small bits of work, often to label data for AI/ML. I have posted tasks on AMT, and two of us have signed up for tasks on AMT as workers. We discovered that being a crowd worker may require a lot of unpaid pre-work (e.g. to qualify), and then the actual work is not paid very well at all — for example, a task that takes a minute and pays $0.10. How is this possible? Quite often the workers doing the labels live in countries where wages are lower. On the one hand, this provides employment and education for people who might not otherwise have it. On the other hand, this may affect the accuracy of the labels — for example, if someone has to do a task about American football but lives in a country where American football is not important, or if someone is a college student just fooling around.
2. The labels come from the data itself. All of our selected applications use Transformer-based models. A neat trick used to train these models is “masking”. With masking, you take the input, and hide or “mask” part of it, and then ask the model to predict the hidden part. For example, with language tasks masking looks a lot like mad libs! This trick is so neat because it means you don’t have to collect labels from humans to train a ML model; but you can’t use this trick for all tasks. For example, you can’t use this trick for object detection models because pictures don’t usually contain lists of all the objects in them.

Finally, we talked about model cards as means of communicating the nature and content of ML models. We discussed why they were originally proposed, and how they have been used by companies and researchers, and how they can be made accessible to “regular people”. Model cards are one way in which AI research can be made more open; open source (on the development side) and open science (on the research side) are also used by AI practitioners to make AI more open. So we had a general conversation about open science and open source practices, both in AI and in other fields.

Part 3: Discussion Questions

In our homework, we each wrote a short essay about the role of open science in the field of our major(s) vs in AI. Here are some of the comments from our essays:

What is open science about? The overarching goal … is to make it easier to publish and communicate scientific knowledge between practices.

What are some positive examples of open science?

• NASA says that all Webb data is available to the public one year after the data are first available to the observer which is similar to the policies for the Hubble Space Telescope. Astronomers worldwide can request data from the Webb archive through the Internet. The public will also be able to view many of Webb’s pictures through press releases and image release archives on the Internet as well. This is a great example of open science, something that the AI field could learn from and begin more freely sharing code between researchers.
• Another example … is the Polymath Projects, which began as one man posting an incomplete solution to a problem, and online viewers collaborating to solve it. There have been three solved Polymath Projects since the blog began.

What makes open science easier now than historically? Through the invention of the internet, the open science space exploded in popularity.

What is good about open science for researchers? With more researchers being able to look into different papers and research there would be another layer of peer review as these researchers could pick up on certain issues that the peer reviewers missed or disregarded. It would also surely speed up the time in which it would take a single group of researchers to come to a solution if instead multiple groups could focus on different aspects of the research. On top of this adding more perspectives could also help to prevent any unethical practices from emerging.

What is good about open science for “regular people”?

• Open Science is already a big part of CS. After just a quick internet search, I was able to access many libraries of infrastructures and services for specific applications of CS. I think that it is fantastic at how accessible CS is already because it makes it easier for more people to get into the field… Without the openness of the field already, it would be hard for more people to become the next great 10x programmers.
• Open science … would also increase public trust.

What are some risks of open science?

• While some of these companies may oppose open science because of the fact that it could make them less competitive, open science very well could benefit them. With more researchers and qualified individuals being able to contribute to certain solutions, these companies with the resources easily could optimize products that otherwise would have taken them much longer to complete.
• Some people worry that open AI could unleash unprecedented materials that could not all be processed correctly. There are just so many unknown possibilities with AI which makes it more difficult to open up the field entirely.

What makes open science hard?

• It is a commitment to take the necessary steps to make their software open source and reproducible. Documenting the analysis done to the data and the fine tuning of the models requires researchers to keep track of these things similar to how scientists keep a lab notebook. This is currently not common practice for engineers. Also, their code is often not commented to the level it needs to be for others to understand what is going on.
• In both Economics and AI, open science is preached but not often practiced, which I think traces back to a lack of incentives. I remember the reading this week talking about how the academic papers of researchers at universities are judged off how many citations they get. Citation statistics are what is valued, therefore the effort made to make something open source is not recognized.

Our after class readings for these two days:

• Sonnenburg, S., Braun, M. L., Ong, C. S., Bengio, S., Bottou, L., Holmes, G., … & Williamson, R. C. (2007). “The need for open source software in machine learning.” Journal of Machine Learning Research 8 (2007): 2443–2466.
• Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., … & Gebru, T. (2019, January). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency. (pp. 220–229).

Also, after today we were able to create first draft application audits for our applications!