Why Self-Organization Is a Luxury

I continue exploring deep reasons behind various phenomena in agile software development. It’s amazing to watch how the universal principle of entropy has manifested itself in the relatively short ~5-10 year mainstream history of this movement. Looks like I have another eye-opener.

Entropy in softdev organizations

What is entropy, in simple terms? This is the principle of bouncebacks as in a pendulum. If there’s too much emphasis on one side, a cultural or a methodological pendulum bounces to the opposite point, as if to compensate this overdoing. We have too many decisions to make and we’re tired of that? Let’s invent the panacea, called Big Data. We got tired of deadlines in waterfall software development, and even Scrum didn’t help, because we got tired of those time-boxed iterations? All hail Kanban, because it has no deadlines. We are sick of managers who treat us as if we are mechanical tools, not humans? We want to break free, so we delve into the tempting freedom of self-organization and no managers at all. I sketched the visual for this pendulum, to make the metaphor clearer.

Now, the universal law of bouncebacks is not a problem per se. It’s a given. There’s no way to prevent seasons of nature from changing; and they say that if you laugh too much, you will soon cry. Anything in this world has a bounceback, if overdone. What all those bouncebacks have to do with the problems of my company, you might ask? That’s what I’m getting at. There’s a composite of forces that start a trend or a movement. If the critical mass of companies are having similar problems, the pendulum gains the powerful momentum to the opposite side. That’s how it happened with Agile. Some companies hopped on board as early adopters, some have not yet accumulated enough energies for that move, some are in the process of swooshing with the pendulum swing. Each and every company has their own place in this swing with their unique combo of energies and driving forces. The way I see it, strategic management has little to do with catching the swing and rushing headlong where everyone else is rushing. To follow the “join everyone else” call is the easiest thing to do, and a certain component of luck might help here as well, if a company has caught the early tide flow, and benefited from the swoosh. The harder part would be to identify where your organization is now, at which particular point of the swing, and how standing there influences your org’s productivity and ability to deliver software to your customers.

I’ve spent enough years in software development, with customers, developers and managers, and I speak from personal experience. It feels good swinging along with the pendulum as it bounces from the left to the right, as on the picture. The agile movement and self-organization used to make sense to me, at a certain period of time. However, later on I started connecting the dots in the bigger picture and saw some signs that self-organization and several other values declared in the agile manifesto are not universal laws. Rather, they mark the points of a bounceback. There’s no universal proof that self-organization or face-to-face communication yield the highest productivity for any company. Keep in mind that agile manifesto signatories were software craftsmen. They postulated the principles to declare that software development is a craft, and they are a noble guild that wants to serve customers. However, the founders of agile movement made no mention of constraints, times-to-market, expectations of stakeholders, and financial goals.

Is it a human instinct to deliver projects on time?

Some complexity theory studies compare the laws by which organizations function with those of colonies of ants or flocks of birds. According to those theories, ants and birds know instinctively how to self-organize to fetch food, or to fly safely to their destination; and the researchers somehow concluded that the IT companies must follow the same pattern. I shared this belief until a realization struck me. I beg your pardon, but is it our natural human instinct to deliver a release on time? Or to a meet a harsh deadline? Can you imagine the Great Wall of China built by a self-organized team? Of course, no. Why am I asking this? Well, developers are not ants and birds, and, speaking of instincts, their basic high-profile instinct as software craftsmen is to craft whatever they do to the point of perfection. Given endless time, they would indeed self-organize to craft at their own pace. Lucky is a software craftsman, and lucky is the team that can afford the luxury to release at their own pace, with this deep feeling inside that they have crafted this piece to perfection. However, we live in the world that is full of constraints. A two-pizza-sized team of craftsmen eventually arrives to a point where they are supposed to come up with a yet another awesome software. This move requires more hands and more coordination with stakeholders, both internal and external ones. The castle of software craftsmanship is invaded by merchants, who want deadlines met and sales goals hit.

So, when the harsh reality of the market hits our noble craftsmen, they go by inertia supposing that self-organization (or flat structure) would allow them to meet those very non-instinctive deadlines and sales goals. That’s what happened to the guys from Everpix, oblivious to the fact that the caste of merchants deserves some of their attention, too. For the Everpix, the pendulum urged them to swing backwards, down-to-earth, to numbers. They didn’t.

If you haven’t done that yet, it’s time now to take a pragmatic look at the eulogies they sing to flat organizations and self-organized teams. The opposition of hierarchy and flat goes first in the list of the pendulum bouncebacks (see the sketch above). As for the other three pairs, I’ve covered them elsewhere. The flat company structure is related to the concept of laissez-faire leadership. Literally, laissez-faire is a French word for “let do”. While this is an excellent approach to leading a company from the standpoint of individual learning and skills acquisition, it does have faults if what a company needs most at this very point in time is maximum productivity. Quoting the article that I linked to above: “Researchers have found that this is generally the leadership style that leads to the lowest productivity among group members.” Again, this type of leadership represents a bounceback from the rigid hierarchy. It does allow people to make decisions and contribute to the activities that are beyond their direct responsibilities, but people have only so much energy, and their contribution strongly depends on the individual working styles. It could be that someone who has a lot to contribute to good decisions is not OK with the combined need to make decisions and to produce deliverables. Organizing and decision-making is work, the chore that drains energy from the performance component of such an individual. If a UX designer or a senior developer who is at the same time responsible for the implementation of a feature or a design spends too much time on decision-making activities unrelated to their main work, this represents a serious hindrance to their productivity. Oh, and don’t forget that personal discipline and responsibility is a must for that leadership style, as well as high cross-competence in all aspects of software development. What happens then if team members lack competence, and what they do is learn? The momentum spent on learning is subtracted from production. Here’s a visual to help explain the idea quickly:

The background colored shapes show how each of the components would take from the other two, if overdone.

Balance learning, time and productivity?

There is a way, though, to get the best of productivity and keep the learning in place. But it does require smart thinking behind the daily setup of company operations. Back to the pendulum, the company has to be somewhere around the pivot point, with no rush to extremes, and with a balanced mix of agile and hierarchical management. This mode of pivotal operation implies razor-sharp intensity and focus in anything and everything, the foresight in knowing and defining which issues need whose attention, and when, and setting the boundaries as far as to the point of worshiping individual productive flows, those of the decision-makers and those of the performers. It takes some hard thinking, and taking hard choices: can we afford 5 meetings with 7 people to discuss this issue? Is it worth it? Does it really matter for productivity that everyone reconciles and has the same viewpoint on any given problem? How the skills and competence of people can be applied optimally, to keep them working and have them learn new things? As I read stories about companies that seem to be successful with the flat structure, I noted their discipline and foresight with work-related interactions. So, if stakeholders are willing to copy-paste the laissez-faire style, they need to be realistic in the assessment of where their company is in the pendulum, and which priorities overrule at this very moment. Hands-off leadership, which is the same as self-organization, is a luxury that not everyone can afford.

This article has been re-written from the earlier version.