The Bias Loop
Y = mX + b is one of the most common formulas in math. I, as the programmer, choose values for m and b based on assumptions about this system. The assumptions used to design Big Data systems are the fundamental issue as to why biases are created and reinforced within our society today.
If I put Y = 2x + 5 into a software, the formula itself will not create bias. The computer cannot manipulate the data to give me an output that isn’t on the line, Y = 2x + 5.
However, the creators of Big Data systems use their own subjective biases, whether consciously or subconsciously. Furthermore, after receiving corresponding outputs, the data is analyzed by the same people, reinforcing those initial biases. Though I am exemplifying this in a simple linear equation, we know that models include numerous assumptions and codes that are only translatable to a small group: data scientists, mathematicians, and programmers. In Weapons of Math Destruction, O’Neil says, “these mathematical models were opaque, their workings invisible to all but the highest priests in their domain” (O’Neil 10). As a result of this incomprehensibility, “their verdicts, even when wrong or harmful, were beyond dispute or appeal” because no one else has the necessary knowledge to question Big Data systems (O’Neil 10).
The bias loop is exhibited in underwriting processes performed by insurance companies. Underwriting is used to set premiums, grouping individuals into risk pools based on demographic factors. For example, when Tom, who is in the low income bracket, applies for insurance, he is deemed higher risk because the firm presumes he has limited access to gyms and/or healthier food. This creates higher premiums that Tom cannot afford, forcing him to opt out of the product. He now cannot afford doctor visits, making it harder to break the poverty cycle.
The assumptions made by the insurance companies to set premiums for potential policyholders may or may not be true, but the algorithms implemented in underwriting programs are biased by the employees who wrote them and reinforced during analysis. Moreover, there are no checks on the biases held by these firms because nobody else has the skills to create and/or read their algorithms. This creates endless bias loops like the one Tom experienced: low income = high risk due to assumed poor health.
We, as a society, need to come together to do what programmers do best — force quit the infinite loop of biases.
