Optimizing to a Fault
The accelerated convergence of more data, faster machines and better algorithms have led to an exponential improvement in the quality and accuracy of predictive models. From planning global air routes, to making personalized movie recommendations, these models now power or optimize nearly every aspect of our lives.
As has been the case throughout history, cities have not been immune to the effects of these developments. Advancements in data collection and modelling have always played a substantial role in transforming cities and the lives of their citizens. The collection of census data, improved sampling methods and development of spatial analysis techniques are all examples of relatively academic exercises having a huge impact on how cities, public services and facets of community life were designed and planned. Predictably, there has been mounting pressure on cities around the world to start harnessing the power of state of the art predictive analytical tools and to incorporate a more modern, data-driven approach to planning and policy.
The incremental, and at times revolutionary benefits, that have come from our ability to more accurately predict everything from mobility patterns to personal preferences are undeniable. But can our quest for perfection go too far? Is there a point where our obsession with accurate predictions of human behavior and optimizing all parts of a user, or citizen, experience starts becoming counter-productive?
If national elections in the western world over the past two years are any indication, we may already have crossed that point.
On the face of it, debating the merits of objectively “better” algorithms is illogical. But what is concerning is less the quality of the models themselves, but the unintended consequences of their applications, despite seemingly innocuous objectives.
To this end, I plan to explore two key problems with the current trajectory of present-day “Big Data” solutionism.
The first is the disregard for the tightening feedback loop between the data that is output by models and the data that is used to train them. The second is the potentially negative unintended consequences that arise due to the blind spots of decision brokers in the technology industry.
1. The Virtues of Randomness
The central goal of any predictive model is to reduce uncertainty around a chosen outcome or choice. A relatively recent application of this principle has been the “personalization” of products and services. This process entails taking data on users’ revealed preferences and behaviors from a variety of contexts and returning what they most likely “want” from a given interaction, such as a product recommendation or a news article from a trusted source. To that end, a simple metric for success is simply confirming that users see less of things that they don’t “want” to see.
Ostensibly, giving people more of what they want is a noble goal. It is also a goal that for many reasons has remained impractical for much of human history. As collecting data from millions of individuals has gotten cheaper, and more importantly, our ability to aggregate that information in useful ways has improved, we are finally within touching distance. Furthermore, research has shown that individuals themselves are often unable to articulate their desires and preferences consistently or clearly. By using an objective observation of their actual behavior, modern algorithms are often more accurate in predicting future actions than the respective individuals themselves.
Despite this, the validity of all such personalization algorithms is based on a set of a core assumptions about human behavior. For past actions to accurately predict future behavior, preferences must be relatively consistent both internally and across time. But a more critical, and often overlooked, assumption is that human preferences are independent of the model outputs. In other words, the feedback loop between input and output is weak enough that the recommendations generated by a model are not a major influence on future preferences that will be input into it.
In an extreme case where all recommendations that were output were fed back in directly, the model would simply be amplifying a signal that itself generated, rather than providing any new and improved insights. In this case, the more “accurate” the model is, the faster it will converge to this limited strain of outputs.
Thus, when the sources of discovery and recommendation begin approximating a closed loop, our second assumption breaks down, and with it, the accuracy and long-term value of the model. In many important contexts this closed loop already exists online. Facebook and Google curate effectively all the content that their users see. In addition, they are able to track user activity across most websites and across all their devices. More importantly, they now serve as the primary portals for both discovering and developing opinions on world events and popular culture.
Paradoxically, as their influence in shaping their users’ future preferences increases and their models become more accurate, the tech behemoths might actually be generating less insight than their methods merit. The desire to continue optimizing and perfecting their predictive power may have reached an unexpected tipping point, with potentially grave consequences.
As an evocative example of the destructive power of over-optimization, we can look to the political polarization in the United States. Although there is literature that already shows links between social media usage patterns and an acceleration in political divisions in recent years, we will construct a hypothetical case to illustrate this effect.
John, a young, ideologically independent user with limited interest in following politics online, starts using Facebook. Using a rough demographic profile of John and the political views of his friends, Facebook begins gradually prioritizing conservative leaning internal and external content in his News Feed. As he doesn’t follow too many external sources of political news or opinion, the relative abundance of conservative news content and commentary from friends nudge him towards interacting more with that brand of content. This further amplifies the importance of such sources in Facebook’s algorithm creating a cycle in which opposing viewpoints are almost completely shut out, and progressively more and more right leaning content gains prominence. Through this simplified case we can see the clear potential of a vague, inaccurate estimation of a user’s preferences gradually shaping the very preferences that were to be detected in the first place.
This logic is not limited to the political arena. It also serves to potentially inhibit creativity and innovation. Netflix’s funding of original content has largely been based on the types of shows it believes its users demand, based on an interpretation of their historical viewing habits. Similar approaches are now being used by large film studios and even music labels to choose what content to release or market aggressively. Over time an emphasis on optimizing for successful content will reduce the opportunities available to riskier content that could lead customer preferences into new directions. Very few of the creative artefacts that became cultural highlights or turning points were conventionally “predicted” to be a success.
This approach is also beginning to have an impact on city planning. In a visit to Sidewalk Labs New York office, we were presented with a passionate pitch for how their innovative, data-centric approaches to analyzing urban problems could inform far better design decisions than conventional methods. The example they presented was one in which responses given on a survey on present-day and desired transportation options in central San Francisco were at odds with actual the mobility patterns in those areas. What was ignored in that assessment was that the current mobility patterns, which they intended to use as a better proxy for intended travel decisions, were also a function of the built environment around citizens. A large highway could be separating an entire community from a park that they might otherwise like to walk to, or it could be inhibiting a neighborhood from becoming a popular destination for commercial offices. By attempting to “predict” citizen behavior based on past actions, there is always the risk that we may simply be “optimizing” our way around a better way that could have been found by being more tolerant of less accurate, but “fresher” data.
One solution is to introduce a forced random component into models that can be learned from. In fairness, most personalization algorithms claim to do this. But these aspects are often mentioned as an asterisk rather than a headline feature. Introducing an appreciation for the importance of engaging with new, random content to develop fresh opinions, and thus data, should be an integral part of the personalization industry’s messaging. This will serve to both educate the public on the extent of “filtering” they are subject to and also given them more control over the direction their profiles of personal preferences will take.
As an aside, as predictive filtering, categorization and recommendation becomes more and more pervasive, it may also lead to some philosophically undesirable consequences. In a world in which we are all a sum of our past actions and preferences in such an extreme way, we are robbed of the ability to reinvent ourselves or course correct for past mistakes.
This leads to some difficult questions that we need to ask ourselves. Is it possible that the better we get at measuring “individuality”, the less human the concept becomes? Even more importantly, how much control should individuals have on the way the world, and in this case algorithms and faceless corporations, see them? Should we have the power to modify and curate what factors go into our profiles? Is more control even compatible with the benefits of predictive power when we are known to introduce biases into our self-assessments?
2. Who builds the future?
This leads us to the issue of which parties hold the power to make the important technological decisions that have far-reaching implications for society, and where they might be falling short.
Big Tech has long gotten away with their techno-utopianism. For the best part of the last 5 decades Silicon Valley’s approach has been to build out technology to the extent that it is technically possible and to think about consequences later, if at all. Better technology is always equated with creating a better world, and any ill-effects are usually deflected to an imperfect implementation that can only be fixed with unhindered development of more advanced technological solutions. But their day of reckoning might finally be getting closer, if not here already.
Tech companies, and the engineers that form the majority of their workforce, have consistently demonstrated evidence of two kinds of blind spots.
The first are simply a set of practical ones. In an effort to optimize for one, or a small set of metrics, decisions are often made that lead to a cascade of unintended consequences. Although unexpected results are to be expected as a matter of course in any interventional field, what is surprising is the level to which the complexity and interconnectedness of various systems is taken for granted when technological solutions are being rolled out.
A simple example of this can be seen with ride-sharing companies such as Uber making a case for why their pooled ride offerings would lead to less congestion. The ‘Pool’ product was a solution that was meant to optimize for all the wasted and inefficient trips taken travelling to and from similar destinations at roughly the same time. The scale of the potential opportunity from this analysis was indeed correct, and the pooled rides were a huge success. However, what was not planned, or simply willfully ignored, was that as the cost of pooled rides became cheaper, they triggered a strong substitution effect. New users that had previously been happy to use public transportation were now shifting to using ride-sharing cars. This in turn led to a measurable increase in traffic congestion when compared to previous levels. In today’s complex world local optima rarely coincide with the global optimum. As the online and physical words continue to converge and the world becomes even more interlinked, short sighted design choices need to be the exception rather than the rule.
The second set of blind spots are moral. Issues of embedded bias and unethical product decisions are well documented. One such example of a private company making a choice that ended up exposing unintended bias was Amazon’s Boston roll-out for same day delivery. The company identified neighborhoods that ordered frequently from their website and had shown a preference for faster delivery times. They proudly highlighted their use of machine learning algorithms in creating the most optimal map of service areas that would provide the maximum coverage while remaining cost effective. However as soon as the map was made public it was immediately pointed out that all the areas that were outside the delivery zone were predominantly low-income, African-American or ethnic neighborhoods. This led to some justified backlash towards Amazon choosing to exclude an already underserved segment of the city and not fulfilling its duty as a corporate citizen.
Data will always be biased whenever systemic or institutional bias was historically present. The only way to avoid, or work around, problems such as the one mentioned above is to be more intentional about removing bias and understanding the potential implications of any solution that impacts a significant segment of society. This must be the case even if a more responsible solution is not the “optimal” or “most effective” one.
But this cannot happen simply by thinking longer and harder about problems. In order to raise concerns that are not obvious to the people developing the technology, more diverse voices need to be present at design time. In the absence of such organic forces, or in addition to them, there is also a need to impose a balanced set of ethical standards that force engineers to account for more constraints when making decisions. Currently those constraints are too often limited to cost, skill and technological feasibility. A combination of finding ways to inject more relevant voices into the process and educating engineers to think more holistically about the social implications of their technological interventions could go some way in avoiding potentially embarrassing and harmful failures.
