Anticipating Market Pressures, Takeovers, and Hacks
Guard against undesirable uses of data
This is part of a series to provoke dialogue and provide concrete ways to help teams ethically build and design intelligent systems. Read our introduction.
Knowledge is power. You may gather information about people and communities, intending to use it for their benefit. But at some stage others may have less benign intentions for using the same information. Competitive forces, misaligned incentives, outside partners, hostile takeovers, and bankruptcy might strip away the original purpose and intent of a design. Teams change, corporate strategy pivots, and data may be lent or sold to other agents, groups, and companies. That’s why we should always design to guard against potential future harm.
To do this, articulate best-case and worst-case scenarios for how your data might be reused by others differently than was originally intended. Then determine what, if anything, you could do now to avoid potential issues later.
Activities to try
_Imagine that your database gets hacked. Or your company is subject to a hostile takeover. Or that your data assets are put up for sale after a bankruptcy. List a few companies, governments, people, or groups that might try to use these data assets — for good purposes and for bad. What might they use the data for? How could it be put to destructive use? How might you change what data you collect, how you store it, or how you structure it to avoid these potential consequences?
_List what, if any, data should be excluded, guarded, limited, or specially governed within your service. How could you accomplish this without jeopardizing the beneficial intent of your design? For example, would it be easy to anonymize people’s names to make it more difficult to cross-reference with other sources? You also may want to talk to an internal or external information security expert to explore the best available safeguards.
Project vignette
In the spring of 2018, the City of Chicago launched CityKey, a single card that could serve as an ID card, a library card, and a public transit farecard. The card was specifically designed for residents of Chicago who might otherwise have a hard time acquiring a driver’s license or state ID — including undocumented immigrants. In designing the CityKey business processes and data retention policies, the City Clerk’s office wanted to avoid the the experience New York City had faced with its own IDNYC card. New York City had to go to federal court to protect the personal information contained in its applicant database, which the Trump administration sought to use in connection with deportation and immigration enforcement activities.
Chicago City Clerk Anna Valencia worked closely with the city’s then-Chief Technology Officer Brenna Berman to safeguard their applicant database. According to Berman, “By designing the Chicago CityKey system to only require and capture minimum viable data, the program both protects the privacy of vulnerable residents while delivery the services they need. It is a critical design feature to only collect the most basic information needed to operate CityKey, and nothing more.” Unlike New York’s program, Chicago’s CityKey database does not retain images of any documents applicants submit nor any personal information about CityKey holders.
Explore the other posts in this series:
