The Magnificence of Flow
Long before the personal computer revolution, before the information technology industry, some of the few companies paying people to write software had already noted that some engineers were a lot more productive than others. Not 20% or 30% better but hundreds of percent. Superficially these stellar developers didn’t seem that different from others; they didn’t type that much faster, they didn’t measure as significantly more intelligent, they didn’t put in extraordinarily long hours. The distribution of productivity and work quality wasn’t a continuum or a bell curve, these developers were …
… in a class by themselves.
Hospitals had noticed something similar with patients after heart attacks; some died and some with identical clinical presentations recovered and went home. Again the distribution wasn’t like the normal distribution of height or intelligence, there was a hidden factor. The obvious lifestyle differences were quickly eliminated.
In this case and to the surprise of all the factor most strongly correlated with recovery from an infarction was having a pet at home, specifically a furry pet. Intensive care wards relaxed rules on visiting pets and saw a sharp rise in recovery. Since then we have learned that stroking the fur of a cat or dog is almost therapeutic.
The software productivity hidden factor proved somewhat more elusive. But it was investigated and it was found. And it’s not a genetic gift like perfect pitch, it’s a work habit that you can learn.
The Significance of Flow
What distinguished these super-developers from their peers turned out to be something simple to describe but difficult to achieve, and the factor was that
Great developers are able to maintain prolonged periods of unbroken concentration.
This was abbreviated as “flow” and it is this term I will use henceforth. To avoid loaded descriptions like “stellar” or crude language like “super-programmers” please allow me to coin the compound noun “flow programmers.”
Having found the pivotal factor the researchers set out to characterize this condition and quickly learned a few features of flow:
- It is a state of mind difficult to enter but easy to leave; in fact it is quite fragile
- Interruptions break flow, even brief ones, and it is difficult to resume, usually lost for the remainder of the day.
Many reading this have likely recognized that what they call “good days” on the keyboard were days when they didn’t have to break from work for meetings and phone calls, nor get sidetracked with burdensome processes. When they finished one task they went immediately to the next, clicking off tasks and bug fixes and almost joyously brain-surfing.
Flow isn’t just getting a lot of work done; flow feels good.
Enabling Flow: The Early Days
I began programming as a hobby on a Radio Shack Color Computer with an added floppy drive and taught myself the C language from Kernighan and Ritchie’s first edition of The C Programming Language. After a year at a small company I was a tester at Microsoft; five months after hire I was a developer at Microsoft, working on its flagship product, OS/2 3.0, codeword Cruiser, released later as WindowsNT.
Those were glorious times. In retrospect, it was clear as could be.
The entire Microsoft company culture was centred on creating conditions that allowed us to enter and maintain flow.
Everyone had an unshared office, we were discouraged from interrupting each other, meetings were few, email was preferred over phone calls. There was no Internet yet so the distractions of social networks weren’t available. Cell phones were years in the future. Computer games were casuals and not as distracting as they are now.
Coffee and drinks were free, most buildings had cafeterias. Nobody ever knocked on an office door to ask for change for a dollar. The company had internalized the idea and this was, indisputably, the origin of its long lost greatness.
We loved working there. We worked long and late because we loved our work and we loved how well we were treated and not because we were terrified of the semi-annual review. At that time the review was a relaxed session of friendly feedback and nothing like the reign of terror it is now. The difference between these days and the company today cannot be overstated.
Flow was for many of us a natural condition. It was almost an inverted inebriation and when I attended my first annual meeting my heart swelled with pride. Our tools were few and simple, we had minimal process to encumber us, there were no quality gates or code coverage metrics.
We were passionate about our work, we regarded our coworkers as teammates, not competitors in a stack ranking. We weren’t burdened with management fads and once we’d passed the already standard whiteboard interview we were not degraded with impediments to our work. We were relaxed, and the pressures of crunch time were the only pressures. It was a happy time.
It didn’t last.
The Shadow Lengthens
The extant version of Windows was 2.11 and it was used solely by hobbyists. I saw it in a business exactly once. It was not an operating system, it was a piece of software that some people ran on their 8088 desktops with 640K and low-resolution monitors. We did our work in MS-DOS; nobody used Windows all day, we returned regularly to the DOS command line.
Our text editors were minimal, with none of the features we now enjoy. If you had a second monitor it was “green-screen” text-only monochrome and it put you on the leading edge, debugging your code on the terminal and running your Windows project on a low-resolution colour. This was nicer than it sounds.
On May 22, 1990 Microsoft released Windows 3.0. The company went overnight from the most familiar software outfit (for people who didn’t work with mainframes) to the only software company worth mentioning. The Redmond campus grew new buildings like toadstools after a rain.
And things changed. The egalitarianism we had enjoyed yielded to an ever starker hierarchy; org charts appeared everywhere, it felt less like a team and more like a business, and not in a good way. And the business culture insinuated itself more and more into our work. Suddenly we shared offices with coworkers who were often on the phone and had tobacco on their breath, we had a lot more meetings. The change in company culture the year after release was profound, and many who had been there during IPO called it a day.
Flow went from the normal condition to an exceptional one, even harder to achieve and more frequently broken. It is a fragile condition, and the culture that had enabled it began to break down.
Every other company followed Microsoft’s lead, just as they had with the abhorrent whiteboard interview; single-occupancy offices were more and more reserved for managers, not coders, then came cubicles, and the culture of flow was replaced with a frosty insistence that we should be able to leave our work for a meeting and return to as before. This was of course delusion. The original studies had shown that this didn’t happen; break the flow and it’s gone.
Some people were better than others at getting it back, but flow programmers became less common and the halcyon days drew to a close. Software companies that had once been run by developers-turned-managers were now run by business majors who had never coded and to whom a meeting was a productive use of time.
While some programmers are still able to enter this superlatively productive condition, the recognition of flow by software executives has vanished without a trace. At many modern companies, a developer spends double-digit hours in recurring meetings every week, interruptions are routine and no effort is ever made to spare them. Management fads come and go like the illnesses they mostly are, and most management fads involve regular meetings; a Daily Scrum that starts your day with a rush-hour commute so you arrive at work already worn out and peevish.
We have to endure managers and even fellow developers passionately evangelizing transparently counterproductive models and we have processes that rob us of time all day. On the Windows Vista project (“Longhorn”) checking in one’s work was a four-hour slog and only after a queue of check-ins hours longer.
But nothing demonstrates how far we have fallen from the great days of flow like the practice called pair programming; sitting in uncomfortably intimate proximity with someone you possibly can’t stand puts even ordinary concentration completely out of reach and flow isn’t even on the radar. You can’t concentrate while arguing every keystroke. This is pure folly, but it has its enthusiasts and lots of them.
Are the Good Times Really Over for Good?
They don’t have to be. At most companies now the management paradigms prevail, cubicles have all but replaced offices, but more to the point the recognition of flow, however advantageous to the company, has left the building. The early greatness at Microsoft was a direct result of its unapologetic flow-enabling culture and its current mediocrity is directly attributable to the abandonment of that.
Your workplace probably won’t help you. Business managers sadly tend to regard software engineers as menials and our need for uninterrupted concentration as effete. Meetings are good in their eyes; in ours meetings are interruptions.
It would be in the company’s interest to restore the conditions that enable the productivity and work quality of the early 1990s but such people are not likely to reduce time in meetings and are likely to take the attitude that prevailed under Steve Ballmer’s tenure, that developers are already unreasonable in their expectation of decent salaries and that prolonged freedom from interruptions is asking too much.
Processes that interrupt work with tiresome and repetitive operations are likely here to stay, though these can be integrated into a smooth workflow unless they are poorly designed and unreliable, as they often are. Overenthusiastic Git branching will shatter your concentration with near-perfect reliability. A tedious and complex check-in system that leaves you aggravated will break your concentration too.
Your writer works exclusively from home and is able to get into flow mode fairly easily and regularly. But remote work goes in and out of esteem.
What You Can Do
If you can’t change workplace practices you can still work to shut out interruptions. Headphones are very good for this, but don’t try to concentrate while listening to music that commands your attention. Whatever your musical tastes, music with a vocalist is probably worse than open ears. Lyrics will command your attention as much as an officemate on a cell phone.
Solemnity works better than exciting rhythms; ambient music (Steve Roach), Renaissance liturgical music (The Tallis Scholars) work for me; experiment with musical genres but shrill or twitchy music will likely impair you. Your mileage may vary. Don’t listen to spoken word like podcasts.
Do your work on the largest monitor you can; a laptop screen is like peering at your code through a keyhole and while it lets you work on a plane or a bus you should cable it to some acreage most of the time. Do everything you can think of to reduce interruptions.
Discussions with managers are problematic and even the most well-intended and politely-presented suggestions of reducing meeting frequency are likely to be ill-received, but you can try. Decline meetings and stay at your workstation. Keep meal conversations work-related and low-key. Eat alone if you have to.
And eliminating distractions is your responsibility too; if you succumb to social networks, blogs, games, the loss is as bad as to external distractions.
What We Can Do
RESIST. OK, that’s probably not a good idea. For some ineluctable reason there is heavy resistance in IT toward any kind of organization; I’ve never heard of even a suggestion of a union and it would take that kind of linkage of arms and a united front to overcome how executives from business think things should be.
At Intel we were instructed to refer to problems as “opportunities,” a foolish corp-speak begging for mockery (“the building is on fire. What an opportunity!”). In Agile we are told to refer usage scenarios as “stories,’ a humiliation and a steep drop in accuracy. Nomenclature is powerful, both as harm and benefit. This is a game two can play. Make a point of referring to meetings as “interruptions.” Ask if we really need more than one brief recurring meeting per week and what can better be done in email. Don’t relent. Twenty hours a week of meetings are impediments to basic work, much less flow.
But remember: you will do your best work and more of it if you can concentrate without interruptions. We should all work to restore recognition of the benefit and power of flow and its advantage isn’t limited to software engineering. Reading books, playing musical instruments, any endeavour that depends more on mind than muscle goes better with flow.
Eliminate interruptions. Concentrate. You won’t be able to do it every day. But the days you do, you will soar.
If we can restore consciousness of the productivity and work quality enabled by flow, and further get across that working from home is a better way to achieve that, maybe we can make some headway at getting out of offices.