“I will never forget her description of what can go wrong: Imagine if a group of people were given a school bus to drive, only they were each given a steering wheel of their own and had to coordinate among themselves which way to go. Introduce power dynamics, and it’s amazing what all can go wrong.”
danah, this is a harrowing and accurate description of bureaucratic decision making. It only gets worse as decision makers confront the complexity of subtle and embedded biases in data sets and algorithms that are empowered to make many decision on their own, and decision makers serve only as “moral crumple zones”.
The challenge you have set for you and your team at Data & Society, to “situate ethics in the practice of data science” is overwhelming, if not impossible, if you assume that data science is is being practiced by large organizations, subject to the vagaries of bureaucratic politics, that wield significant market power, enjoy huge information advantages and exploit those advantages through opaque, proprietary, algorithms to consolidate market power.
Fortunately, there is an alternative. If we can devolve power to the edge, away from the platforms and the rent seeking intermediaries to the creators and consumers of value, if we can as Tim Berners-Lee has said “re-decentralize” the web, we can avoid many (not all) of these challenges by decentralizing decision making in open, efficient, resilient markets. The markets themselves are amoral, but the people exercising their freedom to choose a service provider in a competitive market will bring their ethics to those choices.