The idea of the open plan office is very entrenched today in the IT field, and so only very courageous decision-makers are going to go against that dogma. And as you point out, offices with doors also cost a-lot more — in an organization in which programmers have doors, the CFO is going to scream: “Why is our real estate cost per person so much higher than the industry norm for programmers?”
Many, many respected people have already shouted that the Emperor Has No Clothes on the open plan office — Susan Cain, Daniel Goleman, Cal Newport, and others. However, the Agile community — which is seen as leading innovative thought in the programming community — is deeply invested in the open plan idea.
Also, paradoxically many of the Agile community’s thought leaders are extroverts who seem to think that programming is like sports or military squadrons (it is not), where everyone stands at attention, shouts out their status, then roar hooah! and all run to their desks and code furiously, with ideas being shouted back and forth like a football being passed. One of the most prominent thought leaders — Jeff Sutherland, inventor of Scrum — is a former fighter pilot and doctor, and yet, through his methodology, he is telling an entire profession of programmers how they should work, and they are gobbling it up.
Programming is not like sports or a military squadron at all: programming needs focus, which requires maximizing working memory. Distraction and interaction sap working memory. Being hyped up generates cortisol, which reduces the ability to think deeply. Programming requires calm.
It is going to take a respected thought leader in the programming community to call out the obvious cognitive dissonance — to say that the Open Plan Office Emperor Has No Clothes. This happened with test-driven development (TDD), a longtime favorite practice of the Agile community: when David Heinemeier Hanson blogged that TDD does not work, a firestorm ensued, but his ideas stuck, because he is so respected, and now, a few years later, TDD seems to be dead among those who are on the leading edge. Microsoft, which has a very compelling DevOps story to tell, has completely de-emphasized unit testing in favor of automated integration and acceptance tests.
We need a giant of the programming community to come out and say, “Programmers need quiet”.