Data Violence and How Bad Engineering Choices Can Damage Society

Cultural harms can go well beyond search results — which can be bad news for vulnerable communities

It’s February 2018, and a small group of Harvard researchers are standing on a stage in New Orleans unveiling a system that automates the detection of so-called gang crimes.

A wave of concern ripples through the audience: Data on criminal activity is notoriously unreliable and often subject to manipulation and misclassification. In one notorious case, a California state database of “gang” members was found to contain at least 42 babies.

The Harvard researchers didn’t appear to have questioned the integrity of their source data and hadn’t thought through the unintended consequences of implementing a system like this. They hadn’t acknowledged the problem of racial profiling or of damaging innocent people who are wrongly identified. In fact, they hadn’t considered any ethical obligations at all.

When asked how the tool would be used, one of the project’s computer scientists confessed he didn’t know.

“I’m just an engineer,” he said.

This kind of response is a cop-out, and the audience knew it.