Slack Chat: Ethics of AI

Machine learning refracts biases in society, just like Parmigianino’s Self-Portrait in a Convex Mirror (see also the late John Asbery’s poem)

yevgeni Welcome! There are many of us interested in the ethics of AI here at, and we wanted a format to explore and share our thoughts with a broader audience. So we decided to adopt the slack chat format used by the team at FiveThrityEight. This is our first chat, and we’ll be discussing ethics in AI.

tyler I’m ready!

kathryn Me too!

yevgeni Jumping the gun — I was about to introduce you. Today’s chatters are Kathryn Hume (our VP Product & Strategy) and Tyler Schnoebelen (our Principal Product Manager).

yevgeni Let’s start with something fairly easy. Privacy and ethics are topics that have been discussed for a while, and are already protected by law. Why do we need to reignite a debate on ethics now? Kathryn, why don’t you start?

kathryn Easy?!? Gosh, this is a huge question. My mind goes all the way back the Fourth Amendment of the US Constitution. At that time, privacy was closely affiliated with someone’s home being her castle. The executive branch of the government couldn’t enter the castle unless they had a viable reason to do so. As such, privacy was about space. Our home. Our property. A place where we could live freely without the infringement of the government.

tyler If we’re talking The Fourth Amendment — I just saw 12 hours of Taylor Mac performing a history of American Music, which starts in 1776.

Taylor Mac’s epic concert

The performance included a dramatic reading of Thomas Paine’s Common Sense (the most popular pamphlet ever published in English! Wait, what’s a pamphlet?). Some people think of government primarily as “this thing I give part of my property to (taxes) in order to protect the rest of my property”. That’s a very limited notion of what you want government to do and more to the point here, a limited notion of what is worth protecting.

kathryn Gotta be inspired by Thomas Hobbes…And on a different ethics and AI note, in Raising the Floor, Andy Stern points out Paine was an early theorizer of universal basic income (UBI). If I’m remembering correctly, Paine theorized that we all have a basic human right to property, and that taxation there upon should, normatively, generate a UBI. This was back when income was closely derived to land ownership; pre Karl Marx stuff.

tyler But I think we’re talking about more than just property when we talk about privacy. Like what about having police break into a bedroom and see (or THINK they see) prohibited sexual acts?

That is, Kathryn, I think your point is that the home is pretty different from when it was more of castle. We have all kinds of devices in our homes now and we therefore expose a lot more of our lives to a lot more people.

kathryn Absolutely, and this has been a long time coming. In the 1970s, a fellow named Katz made a phone call from a public telephone booth that was wiretapped. During the trial, a judge gave rise to the modern notion of “reasonable expectations” of privacy — which can hold outside one’s physical home. And then, well, what happens when we go from homes to phones to the internet? To all the digital data we leave traces of? And when it’s not just governments but companies, organizations using our data to market to us?

yevgeni That’s interesting. So is it the case that privacy moved from physical objects to ideas? And that’s not well protected by the outdated laws?

tyler Well, I think that part of the answer to your earlier question has to do with the fact that LAW always lags behind technology. So people in technology have special responsibilities. James Moor made this point.

kathryn For sure. It’s a big concern. There’s a lot of policy focus these days on explainable AI to attribute accountability for how machine learning models may impact people. There are scenarios with this is important (as in Cathy O’Neill’s Weapons of Math Destruction), but it’s often retrofitting legal concepts onto statistical systems.

yevgeni So can we rely on laws to protect our privacy and ensure the ethical use of data, or will law simply be an “afterthought” to seal the deal?

tyler I think people often look to the law like it’s something magically “right” and “just”. Laws are made by people so, uh, they are biased and imperfect. Even basic tenets like attorney-client privilege can be ethically problematic.

kathryn The same holds for machine learning models.People assume that code and math means “objective” and “neutral,” but models are trained by human designers on human data.

tyler Definitely

yevgeni Right

tyler I think that’s the problem: we all have blindspots so that even the well-intentioned among us will fail to consider something that some other folks would see was a glaringly obvious problem.

For example, when I was hiring data scientists around the time of the George Zimmerman trial (for killing Trayvon Martin), I’d often ask people to talk to me about meaningful ML projects. People often gave well-meaning ideas of helping protesters detect and organize protests.

kathryn I bet they hadn’t thought about AI falling into evil hands. With so much focus around AI creating an existential risk in the future, people orient their fears towards superintelligent overlords as opposed to the power algorithms might have in the wrong hands.

tyler EYS

tyler YES

kathryn Zeynep Tufecki mentions this in her talks.

tyler Oh I don’t know her work.

kathryn Oh she’s great! Here’s a Ted talk from last year.

tyler Lacuna! Hole in knowledge and I’m so well meaning!

kathryn Touché.

yevgeni That is a very real issue. If I understand correctly this relates to a model that is designed well, but is then misused. Is that different from a model that is inherently biased?

kathryn I think it is, yes.

tyler (Agreed)

yevgeni We have agreement!

kathryn For example, I wrote a blog post a few weeks back about what I call the “time warp” of certain AI algorithms. I cited research by Bolukbasi and colleagues that shows that, when constructing word embeddings — vector representations of unstructured text — we can inadvertently recapitulate social trends and biases we think our society has progressed beyond.

tyler And then there’s Joanna Bryson and colleagues’ works that show that the texts we train on have the same biases we see experimentally in terms of race, gender, etc

kathryn It’s quite similar to the work Joanna has done! What’s cool is that Bolukbasi et al developed a technical hack to overcome these biases once we recognize them (edit! see Joanna’s different take in comments below!). Well, not overcome them, but ensure they are not propagated by our models. They used techniques to disassociate certain words from gender biases.

(By the way, super excited to interview Joanna on In Context next month.)

tyler Hey that reminds me of a question I’ve been wondering about. If I’m building a model to decide whom to give a loan to, it’s pretty easy for that to discriminate on things like geography that are going to basically make it a racist model. So is it okay to build a specific “predict race” model on the data so that the factors that matter for THAT model can be subtracted from the “real” loan prediction model?

kathryn You know, we’d assume we want to be blind to protected attributes (like race or gender) to achieve fairness in machine learning models. The assumption would be that if we deliberately leave those protected traits out, we won’t make racist models.But as you pointed out, there are often “redundant encodings”, features that are so closely correlated that one (like zip code) likely entails the other (like race).

So paradoxically, we have to face up and pay attention to these protected features. Treat similar people similarly, as Cynthia Dwork would say.

tyler One thing I talked about at an ethics workshop is the notion that human subjects research is different in technology than in, say, technology. In technology you may need to build something problematic to stop something more problematic.

kathryn Interesting.I’m really fascinated by the difference between using stats for sociology and ethnography and using them for machine learning products. I think that category shift from observation to action — to product — makes a huge difference in how people think about what models can and should do. (Peter Sweeney has some interesting thoughts here.)

Sweeney argues it’s best to view the output of algorithms as observations, not explanations…just like Galileo in this painting.

tyler We’re talking about academics and industry but there’s also a different part of the world: what school teachers do (do you teach white kids about discrimination like a teacher in my home state of Iowa? or do you say that you shouldn’t make kids feel bad even if you’re trying to help more broadly?) And what happens in, say, law enforcement data.

Basically, I can’t stress strongly enough that our basic assumption in ML should be “Oh, this data is biased”. Maaaaaybe that’s not true in some cases but mostly it is.

Core claim from Tyler’s recent Wrangle talk about ethics and AI.

kathryn I had a very similar experience with one of my law students at the University of Calgary. He wrote his final paper as a first-person narrative of a first nations individual subject to a sentence from an algorithm. And he was INCREDIBLY CHALLENGED by what it felt like to actually identify with a minority group — not just talk about it abstractly. I loved that learning experience.

tyler Oh so he chose to put himself in that position?

kathryn Yes, his paper was inspired by this article. I gave my students the opportunity to write first-person fictional narratives to understand the legal issues we were grappling with, and this topic inspired him.

yevgeni That’s fascinating.

tyler ⚡️ What did you/they learn?

yevgeni I will also add — how can these lessons be used for the broader society?


tyler Uff can you teach empathy?

kathryn It’s amazing how fiction can teach empathy. Not reading it, but exercising our creative minds to write and identify with another. Writing fiction.

tyler Maybe you just try to protect against it. For me, it’s also about enabling people OUTSIDE of AI practitioners to understand what’s going on.

kathryn Absolutely, that’s why I’ve always found that artistic applications of AI are great vehicles to help people outside research develop intuitions. My friend Gene Kogan has eloquently argued the same.

yevgeni Guys, we’re drifting. Let’s get back to AI and ML.

Let’s focus on people developing AI models. It’s hard to know a priori if an experiment is biased as designed. How can a designer mitigate the risk of that happening?

kathryn You know, Tyler has a whole list of practical recommendations in the link he posted above.

(I seem to like starting with “you know”. Linguistic tick. Um on slack)

tyler (I keep starting with “oh”, I got it from my friend Eliza…I analyzed my text messages. Wait, sorry, wrong topic.)

My favorite of the practical recommendation is having the team do a pre-mortem — that is, you imagine the project has happened and it’s been a DISASTER. The team writes the story of what went wrong. This is generally useful and it helps you notice if NO ONE on the team proposes something like an ethical/marketing disaster. That should be one of the disaster scenarios.

yevgeni So that’s a good way to get the team to brainstorm about negative outcomes.

kathryn I think, Yevgeni, that no matter how hard to we try dissociate ML from fiction, they two seem to be intertwined. 😀

yevgeni What do you mean by that?

kathryn The pre-mortem technique that Tyler recommends requires that we imagine, that we consider hypothetical scenarios, and, to a certain extent, empathize with people impacted by our work.

tyler Yeah and I think something you’ve been thinking about is that idea of fair representation — like in slide 57 of your talk for the NextAI workshop.

yevgeni Ok, Tyler’s getting dragged to a Sprint planning meeting and Kathryn has an interview, and I should probably get back to feature engineering. So we have to call it a day. But we will definitely continue the discussion at a later date…maybe even at next Wednesday’s AI in the 6ix! (Come join us!)