The Abilene Paradox (Sunnyvale Edition)
In a 1974 article, Jerry Harvey, a George Washington University professor in management science, introduced a scenario which he refers to as the Abilene Paradox. In this scenario, a family living in Coleman, Texas is enjoying a warm afternoon. One of the family members suggests going on a visit to Abilene, Texas, many miles away, because he believes the other family members will want to go (though he doesn’t particularly want to). Each other family member, believing that the group as a whole would like to go to Abilene, agrees. The family has a terrible time in Abilene and, upon arriving home, reveal to each other that none of them actually had any interest in going to Abilene.
The intent of the paradox scenario is to demonstrate that individuals may act contrary to their conscious desires if they believe that a group holds those contrary beliefs, even if there’s no evidence to suggest such.
As a computer scientist, I can’t help but view this problem as an example of what’s called the distributed consensus problem. In this problem, a group of people need to reach an agreement on the answer to a particular question by sending each other messages over some communication channel. A simple example of this may be people trying to decide the answer to a multiple-choice trivia question by sending each other messages written on pieces of paper. The Abilene example is more complex: the physical communication channel (speaking) is simple and reliable, but the communication channel between each person’s thoughts and the words they speak is muddied by social pressures and assumptions, which are invisible.
In the world of distributed systems architecture, there are several complicated solutions to the distributed consensus problem, and they have many steps and constraints. As much as it would seem that the computer science sphere of academia, and by proxy the technology industry, has solved this problem, I would argue that it has solved it only in the hollowest way. While our database nodes, load balancing servers and predictive networks have been taught to reach consensus based on truly held statements, we as people, as employees, engineers and members of the technology industry, are spellbound in our daily lives by the Abilene Paradox.
When I read articles of Silicon Valley CTOs decrying free time as a waste of valuable coding effort, or when I hear stories of people in their offices at 2AM on a Saturday, I always wonder: who is it in these companies that really believes that those decisions are right? Let’s deconstruct an example, one which is based on an experience to which many Silicon Valley engineers can relate. Alice, Bob, Charlie and Dave work on a development team together and each arrive at their office at 9:30 AM. As 6:00 PM rolls around, most of the team is finishing up their tasks for the day. Charlie mentions that he wants to start on a new feature and suggests that the team grab dinner nearby and hang around to work on it. Alice wants to go home, but is nervous that at her next sprint planning she will seem like a slacker compared to Charlie. Bob and Dave feel similarly, so all three agree to the plan. After dinner, the four attempt to work together, but it being late, each one finds themselves unfocused. They go home at around 1:00 AM. The feature is left an incomplete, buggy mess by the morning.
This example doesn’t seem to fit: it looks like Charlie just dragged three others along with him. But Charlie only suggested it because he believed that Alice, Bob and Dave were still hard at work on their tasks, and he thought harnessing their focus and energy could help him tackle a tough to implement feature. He didn’t want to stay late either, but assumed that the others would stay at the office and figured he could make the most of the situation.
It’s easy for this scenario to play out repeatedly, with large groups of employees staying in the office late, without making any productive work, simply because they believe they will seem like an unmotivated slacker to their peers if they don’t. If dragged on long enough, this can lead to people becoming sleep-deprived and distracted, ultimately ruining their output entirely. It sounds far-fetched, but it’s a common problem in high-energy tech startups.
I’ve come to believe that the Abilene paradox has dug its roots deep into Silicon Valley culture and drives the way we work and interact. It tells us to work late, ignore our families, skip vacations, abandon hobbies, and burn up millions of VC dollars on hilariously uneconomical business plans. We all collectively reached a group consensus that those are the Silicon Valley things to do, even though not a one of us actually wants to do those things. Tech culture is drilled into a bedrock made of chalk, a facade of strength, intelligence and vitality hiding a group of people disillusioned and exhausted.
I’d like to coin this phenomenon the “Sunnyvale paradox”, after the South Bay town that is simultaneously the beating heart of the Valley and an uninspiring commuter town which hosts none of the biggest players in tech. In some odd way it seems like an apt representation of something everybody agreed on but wasn’t really what everyone wanted: the mean average of the Valley, suspiciously absent from the underlying data.
A solution to the real life Abilene paradox is not as simple as applying the lessons we learned from Lamport or Liskov about reaching consensus. Humans behave in unusual ways and our internal communication channels are easily sabotaged. I wish I could propose a solution to fix the industry’s caustic and ruthless culture, but I can’t, other than for people to be honest about what they really want in their lives. If you want to go home, you should, sprint planning showoffs be damned.
