Humans are fallible, why should we expect AIs to be any different? (Some ramblings from a procrastinator)
If I had to come up with a catchy sound bite for this post it might be “Humans are fallible, why should we expect AIs to be any different?” but that’s not very catchy. Maybe if you say it like Master Yoda it’ll stick? “Surprised you are by robot mistake, hmmm?”
People got quite excited by the Tesla crash a few weeks ago. A model X crashed into a barrier while on autopilot, killing the driver. In a blog post Tesla said:
No one knows about the accidents that didn’t happen, only the ones that did. The consequences of the public not using Autopilot, because of an inaccurate belief that it is less safe, would be extremely severe.
There will be more autonomous car crashes and people will complain that “murderous algorithms are running our lives, and we should rise up against them” (J Connor, 1984). If we’re looking at statistics and not tragedies, we can decide if murderous-algorithms are worse than murderous-humans. Do you care who’s doing the running-over? I don’t. I care about how much running-over is being done. (I also care that everyone is getting run over equally, but let’s talk equal-access road traffic accidents another day.)
At this point, in your role as Luke Skywalker, you are probably going to whinge to Yoda that you don’t see my point and anyway you’ve got a universe to save. I think you should stay here and finish your training. Because I’m pretending to be a Jedi master, I’ve got an idea that I think is incredibly subtle and important. You will think that it’s obvious and if you do then I can smile knowingly because it’s my superb clarity that’s made it easy for you to understand. (It’s actually quite obvious, but don’t steal my thunder.)
Here are some premises:
- Automated systems have observable outcomes
- Automated systems don’t have different behaviour when applied to test cases rather than real ones. (Unless they are the VW emissions system, but that took some impressive deviousness!)
- Automated systems can be tested on sample cases. Either cases that are made up, to test obscure outcomes, or catalogues of previous real cases.
- The internals of the decision making system don’t need to be visible to test a system if you have access to enough test-case/outcome pairs.
- Putting humans into a system reduces its outcome’s reproducibility.
We, as human arbiters of what is right and wrong, get to decide if we are into any particular set of outcomes. So if we decide to abdicate final say to a system, that’s still a human decision.
The important thing I’d like to get to is the product of points 1, 2 and 3. This played out beautifully is some work that we did about 7 years ago. We had finally transitioned off punch cards and we were excited to be able to make software tools ourselves. The second thing we made was an HFBS area allocation tool. Actually Dan made it, I just buzzed around and annoyed him. The HFBS area system was used in New South Wales hospitals to make sure that someone was paying for every bit of area inside a hospital. (Maybe it’s still used, I don’t really know.) It’s pretty time consuming to manually allocate that area because it wiggles about a lot, this tool drew it for you.
As with all “quick” projects, this was surprisingly hard. Initially because the Revit geometry engine is terrible, like so bad it’d make your eyes bleed. Autodesk owns one of the best geometry engines available, but it’s not in Revit. If the ACIS engine is Euclid, drawing with a nicely sharpened pencil, then the Revit engine is like a person who was raised by wolves got drunk and then tried to draw with a crayon taped to a three metre bamboo stick — it’s not very accurate, it’s temperamental, and it’s hard to interact with. Dan had to build almost a whole new geometry engine to be able to do this work!
The rules of the HFBS are pretty simple (or so we thought). They are neatly captured by the first diagram. We wanted to know how expert users would resolve some strange and tricky edge cases. I think that at this point I’d just finished reading Thinking Fast and Slow so I had improved my opinion of experts since I did my diploma thesis on how overconfident everyone is. BVN has a pretty impressive history of delivering big health projects, so I had access to quite a lot of experts. To test this I made a worksheet, and because I’m a careful sort of person I put in a couple of simple exercises to warm up on before we went for the hard questions.
The worksheets had a copy of the rules from the regulations printed on them, which meant that nobody could claim forgetfulness. They had a few simple sets of four squares and some tricky pin-wheels. The task was to draw the boundaries between the rooms. We expected to get almost total agreement on the rectangles. After all we were giving this task to experts, senior people who used these rules professionally with real world consequences. When I got the sheets back from the ten people we asked there were 10 different answers for the rectangular layouts! We were shocked. (There’s an opportunity to make a clickbaity headline in here somewhere!)
We went back to them and talked it through, and most people revised their results when we pointed out what we saw as inconsistencies. A couple of people told me that the rules were wrong and they were right. (Sadly they don’t work at BVN any more, I’d love to go back to them and ask them about it again!) What was really amazing was how many people gave different results. Who pays for a strip of lino the width of your wrist is actually pretty inconsequential, so I didn’t lose any sleep over this particular result. I do lose sleep over what this result implies about other things that humans do. If ten experts who work together can disagree over the application of simple rules then it amazes me that roads work, and it’s a miracle that justice systems aren’t total chaos.
Propublica did an excellent expose of the effects of the automated bail decision system used in a lot of the USA. It had strongly encoded biases that led to it denying bail based strongly on skin colour rather than any more useful marker of recidivism. A lot of people called for systems of this sort to be pulled from use. I think that this is a mistake. The current bail system is extremely flawed, but could easily be made better if it were to be open sourced and inspected by some serious stats people, along the lines of open source crypto systems. But we know that it was being biased because we could test it on sample cases; if we rely on human judges to be unbiased then we’re really in trouble!
In our area measurement tool, we took a task that humans didn’t like, and weren’t very good at, and made it go away. We were able to put the line in the right place in a very high proportion of cases and we produced more accurate drawings. We still had a human check them in the end, but at some point the system gets less fallible than the human and we have to make a call about the morality of letting a human do the job.
That’s going to happen soon with self driving cars. If self driving cars kill a person for every 𝑥 million kilometres they drive, and humans are twice as lethal, is it ethically acceptable to allow human drivers? What other factors should we allow into our deliberations?
Humans are fallible, why should we expect AIs to be any different?
It’s not about making AI systems that are perfect, if we wait that long then we miss out on a lot of good things. It’s about making them better than us, and then working with them to make them better than either of us are alone.
