Programming productivity: understanding where your personal energy goes. Flow and “the zone” versus running and obstacles

For the first time since I left Google I am working on a large system from scratch. As background: Christian Schafmeister and I are having a company bringing his chemical computation system Cando and the Clasp compiler to something other organizations can use. One thing I’m doing here is basically a secure, multi-computer dependency graph resolver. Aka a distributed make(1). That is what will actually execute molecular dynamics computations (Amber for now) by using pools of computers with high-performance GPU or CPU resources. The point is that we enable more people to improve chemistry by using molecular dynamics packages, by providing both a graphical front-end and a new computation job scheduling system (plain Amber is a solid nightmare for non-programmers to use). None of these specifics actually matter, this writeup is about productivity.

Runner analogy

I like to express programming difficulties in terms that non-programmers understand. One such explanation is to compare programming to the different kinds of athletics, specifically running. When you plan your software package, and you know you can re-use all these great tools out there, programming can look like a series of long-distance runs. Every day you get up and you run something up to a Marathon, and on bad days a lot less distance, maybe a few days where you only fix your shoes. At the end you covered the distance you want and you have a software.

That is not a useful mental model.

Programming is more like an unpredictable hurdle race. You have passages of flat running, sure. But there are sections of hurdle racing, and in the fog you can only see a couple meters ahead of you. A passage of hurdles can pop up at any time.

Runners (the flat-plane kind) know that you will get into the zone where running suddenly becomes easy, just as The Zone enables us to program 5x as fast. That is how runners carry large distances, just as we write the bulk of working code in The Zone. Now imagine that you, the runner, will find yourself up to 400m of hurdles at any random point in time with little warning. It breaks your rhythm. You also need to restrict power levels during the flat parts, so that you preserve enough energy for the hurdles. So even if you get into the zone, you cannot use it for maximum effect.

I am going through the rounds of “this supposedly easy step took me a day and a half to debug”. Today’s favorite is Apache’s reverse proxy and websockets in authenticated https. This is the most normal thing in programming. I find that I am more sensitive to this now, even after my Google experience that involved … say … overly interesting hurdles ended. Today I can do whatever I want, pick any tools I want, don’t have to tangle with Blaze’s Lisp module anymore and there is neither an open seating office space nor a US-CAM style uncontrolled-climate room (aka “H-WHAC” system). Still, the unpredictable change in rhythm does occur and it does do damage. Organizational independence does not mean you lose the connection to unruly third-part software.

I have no plans to come up with a solution for the hurdles as such. This is all about minimizing the damage.

Damage catalog

We need to break up what the damage is coming from. Obviously you run slower over hurdles than on flat ground, but there is a lot more in play here. I want to express this in terms of energy. Personal energy, which is a resource that you have to spend wisely[1].

A quick pass of energy drains looks like this:

• unexpected debugging. Something should have been a straightforward step, but it isn’t. This drains energy from anger and disappointment, and possibly resentment against yourself for not foreseeing it, or for not being as smart as you thought you are.
• worse, unexpected debugging is not really against a computer. Chances are that your hurdle has been put up by a human, or a group of humans. Some piece of software you need to use doesn’t work like you thought, and there is a human that made it that way. Doesn’t really matter whether somebody actually screwed up, whether it is a freak accident, somebody took the package into a direction you don’t approve of, you are in error, you are holding it wrong, or whatever. This can cause anger directed at living persons, which is a huge energy drain.
• it pushes you into corners when you have to explain yourself. Even I have people to sync with, business partners, money givers etc. It is a huge energy drain to explain why your progress in the last 2 days was NIL and you spent time in “assorted misc”. All known ways to explain this involve energy drain at least from having to revisit all the pain you already went through the day before.
• it causes anxiety. With every unexpected hurdle you are forced to consider how many more of them are out there, and that questions many of your predictions, even your self-image. Anxiety is a top-rated personal energy leech.

Add this up, you will see that energy leaks all over the place, large amounts of it. We cannot do anything about having to spend energy to overcome gravity on the actual hurdle jump, nor can we do anything that allows us to run the flats at higher power depleting the energy we will need for the hurdles. However, the energy drained in total here is multiple times what the actual hurdles take, and our allowable flat-cruising speed between hurdles is reduced accordingly.

Preserving energy

Now, what do we do about those energy drains that are not going into actually jumping?

• experience helps. You have all those unruly software packages you have to use, that suddenly don’t do what you thought they should do out of the box. It really helps playing around with them. Do pet projects with them, where you use those packages for what the designers are proud of. Even if it doesn’t have anything to do with your own task. Redirect frustration downtime toward playing with your tools. You need to get into the mindset of the people who made the software the way it is today, even if that means following those late developers who took the package into a direction you don’t want. People matter. Only using a software package the way you need it to work right now is like banging your head against the wall. You are probably better off doing an excursion using it the way the authors envisioned it.
• build a self-image that treats those hurdles right. It is not useful to put yourself into the position of being a master at holding zoos of third-party packages working together. This can only lead to huge energy drains when you have unexpected problems. Adjusting, or worse — defending — your self-image is draining. Many of us are objectively very good at that kind of zookeeping, out of necessity[2], but it is not useful to make it part of your self-image.
• You need to get emotions out of those parts of programming that are not under your control. Emotions are good, you need them for a wide range of things, e.g. motivation. You need emotions to get into “the zone” where you are cruising in that great zone between challenge and difficulty. That zone is pure emotion, and software packages written entirely outside of that zone are late, bad and never work right. You need to optimize where emotions are anchored. Your distribution of emotions should have heavy weight on things you can control, and less weight on things you do not control.
• Reduce turnaround time. This goes back to “Software Development at 1 Hz”. If you have unexpected debugging to do and it takes too long for experimental changes to be observable you multiply the damage. This is not a joke. Apart from leaking your personal energy in several ways (including loss of context and focus) it is also a straight time sink. Having too slow turnaround is doing a 400m hurdle race not over those track-and-field hurdles but over “Full Metal Jacket” training obstacles you have to climb up.

This is where I want to be: being engaged and In The Zone when I happen to run a flat part of the journey. I want have a less emotional reaction to hurdles appearing out of the fog. I take note of how much energy I currently have left so that I can later fine-tune the power expenditure during the flat parts. I don’t charge into the hurdles like a bull. Focus is lost anyway. Can as well take a step back, recharge the batteries a bit, vent a bit. I want a transition from the good kind of emotion-driven programming toward an energy-preserving sequence of try-n-error with less emotions.

Another thing that might help is a certain amount of parallel development. Maybe have a couple of places in the system that you are making progress at the same time. That slows you down a bit while they are on the flat stages. On the other hand it gives you flexibility when dealing with stalls. If you are well in the zone then you might have the option to stay in the zone by switching to a different “thread”. You will bunch up the obstacles, so will have to do them eventually. But it might be an overall advantage to use “the flow” for longer periods, and batch up the obstacles. Think of it as threads on a CPU. One thread stalls on a memory load and you get more utilization (aka work done per time) by having something else to put into the hot pipeline, or by re-ordering things.

A mental model of those hurdles as pipeline stalls seems useful. You can study how other systems deal with pipeline stalls (such as CPUs) and try to apply those lessons.

Previously-recognized drains and your choices

As a closing point I also want to mention bad software, and people and organizations making it. While we all have to deal with unpredictable software we have to use, it is also undeniable that there are patterns. One of my earliest personal rules is “don’t use software written by mad people”, and it holds up over the decades. People matter, and the reward systems they are a part of. It is no coincidence that the language with the iffy license terms (Java) is the same one starved to death of badly needed features. It is no coincidence that pulseaudio and systemd, which are screwed up in so many similar ways, are made by the same people. Primary directions of open source software change extremely rarely. The people who like the current strengths of one package over another join the former, making that package’s strength’s stronger, and thereby increase the difference in weaknesses (because those who have a problem with that weakness went to the other package). Just because one package is more popular doesn’t mean its weaknesses get more attention. People follow paths, reinforcing strengths and weaknesses. For 25 years now the same weaknesses and strengths persist when comparing Linux and FreeBSD. It is oh so much easier to get any advanced kernel trick deployed under FreeBSD — assuming it has that trick. Keep track of people, and of reward systems that people are in, and direct your choices of tools based on people. All these systems improve, but they tend to improve more in those areas they already are strong in. Software is people, and organizations which are made from individuals. Look at people and their reward systems. Use your gut feeling. Do not use software made by people who are mad in your unfiltered opinion. Do not use software made by organizations with a reward system that by your standards rewards doing the wrong thing.

Same with toxic communities. Software that has a toxic community has people joining who tolerate or even like such a community. The people who have a problem with toxicity contribute elsewhere instead. Toxicity does not change unless there is a huge crisis, and even then examples are rare.

You need to be unemotional about choice here, to avoid emotional drain later during the entire project. These problems are representing a cost, and the currency is personal energy. Those cost factors need to appear on a spreadsheet where all the costs of packages are compared to make choices. If you are choosing for a team then choose wisely based on what kind of future team you envision.

As always I appreciate feedback. My other writings have improved a lot based on your feedback. Comment here, or on twitter, or maybe there will be a reddit thread.

Martin

References:

• In case you are wondering what our project is about, here are two videos:

• Distractions, focus and attention pathogens are a separate topic. I wrote about it here: https://medium.com/@MartinCracauer/on-attention-focus-and-autism-in-the-tech-workplace-8246526fbbc0
• There is a “tactical” problem in programming, and that is slow tools. In neurology they often throw around the number of 7 seconds, which is the timeout after which you lose current working memory if you have to lift your mind off the task. If your development tools need more than 7 seconds to react you have a problem at your hands. https://hackernoon.com/software-development-at-1-hz-5530bb58fc0e
• That writeup had huge resonance in the programming community https://news.ycombinator.com/item?id=12577283
• When it comes to choosing software packages, you might be interested in “Command and Control in Open Source — the right to split”. I evaluate how direction changes in open source software work: https://medium.com/@MartinCracauer/command-and-control-in-open-source-the-right-to-split-2135ecdf6c53

Footnotes:

[1] To put cost into business terms: replenishing your people’s personal energy is very expensive, and there are no shortcuts. This is a resource more difficult to manage than money. You can always hope to replenish money by begging, investing, giving out shares or whatever. But there are very few ways in which you can drop a big bucket of personal energy on your team. (no, free food does not work)

[2] I want to compare this to “if you assume you are talented you are screwed — regardless of whether you actually are talented or not”. That recently came up as a meme on twitter (anybody got a link?). If you assume you are talented and you encounter difficulty you have additional energy drain from a broken self-image. If you assume your success so far involved a lot of practice and other hard work you are better prepared for the passage of pain that is ahead of you right now. Note that it doesn’t matter one bit whether you actually are talented or not. Completely irrelevant detail. In both cases it is better to have an experience and hard-work based image of your success. Talent is for resumes, not for actually getting work done.