Why Corporate Innovation is so Hard

I loved this piece by Tim Harford in the FT about why corporate innovation fails not least because it emphasises one of the more under-acknowledged challenges with generating disruptive innovations in large organisations.

Harford references J F C Fuller’s ‘Plan 1919’ from the First World War, a pioneering and ambitious strategy to use new British tanks to roll over the German trenches and strike a decisive sledgehammer blow to the German army that would end the war. Fuller, Chief Staff Officer of the nascent Tank Corps, originated a plan to amass 5,000 heavy and medium British tanks, 3,000 of which would be used to penetrate German defences along a 90-mile front supported from the air. 800 faster-moving medium tanks would then proceed to attack the German Army’s string of headquarters miles behind the trenches to disrupt the command structures. A further 1,200 medium tanks, supported by artillery, airpower, cavalry and truck-mounted infantry would then move rapidly to penetrate far behind enemy lines.

Fuller’s lightning thrust plan was revolutionary. Tanks had until then only been used to open up gaps in the enemy trenches through which foot infantry could advance a few miles. But Fuller was proposing a new form of mechanised warfare that could end the attritional stalemate of trench warfare and focus on disorganising the enemy. He wrote:

‘Tactical success in war is generally gained by pitting an organized force against a disorganized one.’

Fuller’s biographer Brian Holden Reid called Plan 1919 ‘the most famous unused plan in military history’ and yet as Harford notes, Fuller had actually created an entirely new military strategy that would be studied by the Germans and implemented to devastating affect in 1940. Fuller had in fact invented Blitzkreig.

The great irony was that after Fuller’s plan failed to see the light of day in the First World War, many nations still neglected to see the value in his strategy and believed that tanks should be used in small pockets to support infantry. The Army even went as far as stopping the publication of Fuller’s book for several years yet Heinz Guderian, the mastermind behind Hitler’s blitzkrieg, still managed to read Fuller’s work after the war and used it to great impact.

The story is an excellent analogy for why organisations so often look at the new through the lens of the old and ignore the kind of ideas and concepts that can be truly transformational, even if those concepts are originated inside their own organisations (Xerox Parc’s personal computer with mouse and graphical user interface, Steven Sasson’s first digital camera for Kodak, Sony’s Memory Stick Walkman, the IBM Simon, the first touchscreen phone). In his piece about why companies squander great ideas, Harford quotes Joshua Gans, author of The Disruption Dilemma and economist at the Rotman School of Management:

‘Disruption describes what happens when firms fail because they keep making the kinds of choices that made them successful.’

In other words, companies get stuck in the strategies and thinking that have made them successful before.

And perhaps there is an under-acknowledged reason why this happens. Harford’s article references a 1990 paper by Rebecca Henderson and Kim Clark (Architectural Innovation: The Reconfiguration of Existing Product Technologies and The Failure of Established Firms) in which the authors distinguish between the components of a product and the way that they are integrated into the system, setting out four types of product innovation:

• Incremental innovation: this may strengthen the core components of the product but it also maintains the existing linkages between them (an example would be improving the performance of a car component like a driveshaft without impacting the way in which the car is put together)
• Modular innovation: which may change the fundamental technology of the component but still doesn’t change the way in which the system links together (like an automatic transmission)
• Architectural innovation: this may change the design but whilst the components may not change significantly the way in which they link together does (like front-wheel drive transmissions)
• Radical innovation: which is the most extreme, and involves changing both the technology of the components and also the way in which they link together (electric vehicles for example)

Henderson and Clark make the case that ‘architectural’ and ‘radical’ innovation can more fundamentally challenge the existing organisational structure and processes which makes it more difficult for incumbents to respond:

‘…architectural innovations destroy the usefulness of the architectural knowledge of established firms, and…since architectural knowledge tends to become embedded in the structure and information-processing procedures of established organisations, this destruction is difficult for firms to recognise and hard to correct.’

Incremental and modular innovation are less challenging to established structures since the system doesn’t fundamentally change. Radical innovation establishes a new dominant design and a new set of design concepts embodied in components that are linked together in a new architecture. Architectural innovation involves the reconfiguration of an established system, linking together existing components in new ways. So it can be harder to perceive and make sense of since it involves similar component parts to the problem that are put together in new ways and so have very different relationships with each other. When an organisational structure and information flow has grown up around the old system, it becomes very difficult for the company to respond in suitable ways. The structure gets in the way.

Henderson uses the example of IBM to demonstrate how a company can respond well even to radical innovation if it fits the structure that already exists. IBM successfully dealt with significant developments such as the semiconductor, the integrated circuit, the hard drive, and the shift to mainframe computing since it was not dissimilar in structure to producing mechanical tabulating machines. Yet when it came to the PC revolution IBM’s initial success came only from going against existing strengths and the advantages of its extant structure and eventually internal politics rose up and the PC division struggled to cope and was sold off. In the First World War, says Harford, the invention of the tank did not fit existing systems and structures for fighting the war and so the real potential to use it in a very decisive way was missed.

Reading Harford’s piece reminded me of Noah Brier’s write up of Conway’s Law. In 1968 programmer Melvin Conway wrote that:

‘organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.’

In the context of software for example, the designers of two different component modules need to communicate with each other in order to ensure that the modules work effectively together. So the interface structure of software naturally needs to show congruence with the social structure of the organisation that created it. This similarity between organizations and designs he called homomorphism, noting that ‘the very act of organizing a design team means that certain design decisions have already been made, explicitly or otherwise’.

American Computer Scientist Fred Brooks noted the wider application of this concept in management theory, observing in his early book on software engineering (The Mythical Man-Month):

‘Because the design that occurs first is almost never the best possible, the prevailing system concept may need to change. Therefore, flexibility of organization is important to effective design.’

The key points are that when we’ve started to design any system the choices that we’ve already made can fundamentally affect the final output, that organisation structure and design/architecture are intrinsically linked with the former impacting and constraining the latter (rather than the other way around), and that organisational flexibility is required to organise quickly to fix mistakes and adapt.

Noah Brier builds from this idea, referencing the work of Harvard Business School professor Carliss Baldwin whose work around the mirroring hypothesis shows that mirroring between technical dependencies and organisational ties becomes evident as a way of preserving scarce cognitive resources in solving complex problems:

‘People charged with implementing complex projects or processes are inevitably faced with interdependencies that create technical problems and conflicts in real time. They must arrive at solutions that take account of the technical constraints; hence, they must communicate with one another and cooperate to solve their problems. Communication channels, collocation, and employment relations are organizational ties that support communication and cooperation between individuals, and thus, we should expect to see a very close relationship — technically a homomorphism — between a network graph of technical dependencies within a complex system and network graphs of organizational ties showing communication channels, collocation, and employment relations.’

In other words mirroring makes outputs and new products easier to understand because it aligns nicely to current ways of organising.

It may be that in some instances (particularly in the context of incremental or modular innovation) mirroring will result in effective innovations and outputs. But with more radical, architectural innovation that challenges the architectural knowledge, information flows and structure of an organisation, mirroring can be counter-productive.

Rapidly changing contexts or emergent capabilities characterised more by unknown unknowns do not suit rigid systems and ways of working that mirror existing ways of doing things. Instead we need to work back from the system design we need into a structure that can reflect it. Rigid and deeply hierarchical structures may be good at delivering incremental and modular innovation and supporting optimisation and efficiency, but do not lend themselves to the flexibility required for adaptive, emergent problem-solving. These types of challenges require small, cross-functional teams that can move quickly, unconstrained by ingrained systems, architecture and thinking.

We need to fundamentally redesign our organisations to reflect both of these needs.

For more like this, order your copy of Building the Agile Business Through Digital Transformation, or you can join our community to access exclusive content related to the book.

Originally published at Building The Agile Business.