Wrangling complexity

Businesses today are struggling to survive and thrive in an ever more complex and rapidly changing world. The good news is that a lot of the problems of addressing complexity and change have already been solved. They have been solved by the very same people who started all the complexity problems in the first place: technologists. They solved these problems because they had to.

Technologists started wrangling with complexity before anyone else. The wave of complexity, change and coevolution that is now cresting across the business world first appeared in the technology realm, as computer scientists tackled complex software design problems and struggled to interweave multiple systems into large-scale “systems of systems.”

Since the 1950s, technologists have adopted new approaches that allow them to better address complexity and ongoing change. One, called agile development, is a different way of doing work. The other, called service orientation, is more focused on how bits of work are connected to other bits. Both of these approaches emphasize continuous learning, adaptation, and distributed control, rather than planning, prediction and central control. They are specifically designed for managing work in fast-changing, complex, uncertain environments.

These approaches function like complex adaptive systems, where the parts of the system can learn, adapt and coevolve like a biological community.

Agile development.

As early as the 1950s, IBM programmers were working on software for things like submarine control systems and missile tracking systems, which were so complex that they could not be conceived and built in one go. Programmers had to evolve them over time, like cities, starting with a simple working system that could be tested by users, and then gradually adding more function and detail in iterative cycles that took one to six months to complete. In a 1969 IBM internal report called simply “The Programming Process,” IBM computer scientist M.M. Lehman described the approach:

“The design process is… seeded by a formal definition of the system, which provides a first, executable, functional model. It is tested and further expanded through a sequence of models, that develop an increasing amount of function and an increasing amount of detail as to how that function is to be executed. Ultimately, the model becomes the system.”

This iterative approach to software development, where programmers start by creating a simple, working seed system and expand it in subsequent cycles of user testing and development, has become a common approach in software design, known under a variety of names such as iterative development, successive approximation, integration engineering, the spiral model and many others, but in 2001, when a group of prominent developers codified the core principles in a document they called the Agile Manifesto, they gave it the name “agile” which seems to have stuck.

Agile is about small teams that deliver real, working software at all times, get meaningful feedback from users as early as possible, and improve the product over time in iterative development cycles. Developing software in an agile way allows developers to rapidly respond to changing requirements. Agile developers believe that where uncertainty is high there is no such thing as a perfect plan, and the further ahead you plan, the more likely you are to be wrong.

Service orientation.

Early computer programs were written as sets of instructions, like recipes: First do this, then that, if the user does this, then do this; otherwise, do that, and so on. This worked just fine for simple programs. But software tends to get more complex over time. When programs reached about a million lines of code, they hit a complexity ceiling and started to break. And as software and systems were connected with other systems, the number of dependencies and interconnections increased to the point where the tangled web of interdependent functions was impossible to modify or adjust in one place without breaking something somewhere else.

In the 1960s, computer researchers started to code modular, reusable building blocks instead of procedural instructions. Computer pioneer Alan Kay named the approach, calling it object-oriented programming:

“I thought of objects being like biological cells… only able to communicate with messages,” he later explained.

Object orientation allowed programmers to design software as a system of interacting objects instead of a list of instructions. They could modify a single object without worrying about complex interdependencies. Each object could be seen as an independent machine with its own roles and responsibilities within a larger system.

Object-oriented programming was primarily used inside large enterprises, and not so much for interactions between companies. But the advent of the internet added another layer of challenge and complexity — as well as opportunity. Suddenly it was feasible for software to exchange information not only within the business, but between a business and its partners, suppliers and customers.

The next phase in programming’s evolution, service orientation, emerged to solve this problem. It delivered a way for software objects to interconnect with each other over the internet, at a massive scale.

Software services are very similar to software objects. They are modular functional units that can operate independently and interact with other services using an agreed-upon set of common standards. The big move forward comes from the way that they interact with the larger world. In service orientation, technologists have now agreed to a set of standards that allow any service to interact with any other service, over the web, regardless of the service’s underlying technology.

Services can be made available over the web or any other network. They can be made available to the general public, or to a defined set of authorized users. The power of a service-oriented architecture is that each service can learn, adapt and coevolve without wreaking havoc on the overall system, just like species coevolve in a biological community.

Three principles for the core of service-oriented design: service contracts, composability, and loose coupling.

Service contracts.

A service contract is a simple description of the service, including what the service provider needs from customers, what it will do for them, and any rules about how the service provider and customer will interact. Like any business contract, it represents an agreement.

Business examples abound. The contract doesn’t need to be specified in writing as long as both parties understand the agreement. For example the service contract of a fast-food restaurant is different than a sit-down restaurant. The agreement is that customers will stand in line and order by number in exchange for faster service. If you sit down at a table in a fast-food restaurant and wait for a server to come and take your order, you will be waiting a long time. The reason fast-food works is that providers and customers both understand the promise of the service and agree to work together in a certain way.

A service contract specifies what the provider will do, but it doesn’t specify how the work will be done.

The advantage of this is that a service can hide its internal complexity, and even change the way it operates, as long as it continues to keep the promise of its contract. This is important because it allows the service to independently evolve and improve its operations without affecting customers or other services. A service can be as complex as it likes internally, so long as it provides a simple contract describing what it does and how it will interact with its customers.

Most services have some kind of complexity that is invisible to customers. For example the kitchen and dishwashers in a restaurant are not usually visible to diners, and most stores have areas such as storerooms and shipping/receiving docks that are not obvious to customers. The iOS operating system that powers iPhones and iPads hides a lot of internal complexity. There are no files or folders. There are only apps that you access to do things. Amazon customers don’t have to know anything about Amazon’s warehouses or distribution systems. They just order on the website and sign for the package when it arrives.

The reason to hide complexity is that it makes a service easier to understand and use. Since customers see only the things they can act on, make decisions about or buy, they can make better, faster choices.

Composability.

Most services are combinations of other services.

For example, any kind of food service, from a vending machine to a five-star restaurant, must provide a few core services: it must be able to take orders. It must be able to take payments. It must be able to store and deliver it to customers. Every food service must make decisions about how it will do each of these things, and how it will combine these services with other services to deliver value to customers.

Common standards make services more useful by making them connectable and composable, so they can be easily combined into larger services.

For example, consider a restaurant with a bar and a kitchen. When they use a common ordering system they can work together more effectively. Wait staff can easily access both the bar and kitchen services, and the total charge can show up on one bill. This means the larger service works better, and it’s more convenient for the customer. At the same time, the bar and kitchen services are separate in the sense that they are not dependent on each other — they can exchange information, but each can also operate independently of the other. If the bar shuts down people can still order food, and vice versa.

Loose coupling.

Loose coupling simply means that services agree to use a set of common set of rules about how to connect. So as long as a service follows the rules, it can update, change or modify itself without having to worry about the impact on other services in the system.

Web pages, for example, are loosely coupled, because one web page can link to another without knowing anything about the other page, beyond its address and the rules for connection, which in this case is HTTP, the protocol common to all web pages.

The opposite of loose coupling is tight coupling, where elements on both sides must be designed to complement and fit one another. For example, most mobile phone companies have a unique interface for attaching the charger to the phone. There’s really no benefit to customers in this. The primary reason is so they can sell more chargers. This is why you have a drawer full of perfectly-good chargers that are useless to you or anyone else.

But there can be good reasons for tight coupling. Things that are designed to work closely together can deliver better performance, more efficiently. For example, most of the components of your car are tightly coupled, because each part is designed to fit smoothly and integrate with every other part. You can’t take a door, or an engine, out of a Honda and attach it to a Ford, at least, not easily.

But your car is loosely coupled with many other elements in the road-and-car system, and for good reason. For example, when you pull into a gas station to fill up your tank, you don’t have to worry about whether the pump nozzle will fit, because there is a standard for that. If you need to put air in your tires, you don’t have to worry about whether the air hose will fit your tires, because there is a standard for that. Cars are tightly coupled internally but loosely coupled with the overall system that they operate in.

Service-oriented architecture works in the same way. Internally, a service can be as complex as it wants to be, just like your car. But when it needs to interoperate with the larger system, it follows a common set of rules.

Standards can be proprietary and closed, such as Apple’s iOS, Microsoft Windows, and Facebook’s application development platform, which are provided and managed by a single company; or they can be open, like HTTP and TCP/IP, which govern web interactions, and the electrical sockets in your home. Open standards are defined and managed by technical communities, or sometimes they just evolve naturally over time, like the standard for the width of cars and roads, which can be traced back to the width or Roman roads, which was determined by the width of the two horses it took to pull a Roman chariot.

Regardless of whether the standards are open or closed, it is the number of people and businesses that have adopted them that make them valuable. The more that have adopted the standard, the more valuable it is.

Service contracts make services simple, modular, understandable and easy to access, like building blocks. Composability makes services combinable and connectable. Loose coupling is the standardized interfaces and connections that make it all work.

Organizing for agility.

Agile and service-oriented approaches are designed for complex, uncertain, fast-changing environments. They are proven methods for organizing systems and work. And the same approaches that solved complex software problems can also work in business.

For example, Whole Foods Market operates on agile and service-oriented principles.

Small, agile, autonomous teams.

Each store is an autonomous profit center made up of about ten self-managed teams, who manage various aspects of the store, like produce, deli and so on.

Teams determine their own staffing levels and manage their own part of the store. Teams are responsible for all operating decisions within their group, including pricing, orders and point-of-sale promotion. Teams buy locally and stock the things they feel will be most interesting to local customers.

New hires are subject to peer review: after a one-month trial, team members vote and a two-thirds majority is required to keep the person on the team.

Service contracts.

Each team is bound by a service contract, that specifies what it is accountable for, how it is measured and how performance will be rewarded.

Each team is measured and managed as its own profit center. Every four weeks, the company calculates profitability for every team in every store. Teams get bonuses when they meet or exceed profit targets.

Ten times a year, a regional leader or executive from company headquarters conducts a surprise inspection and gives the store a report card that rates it on 300 items. Once a year the company does a survey to probe for employee morale issues.

Composability.

The teams are also composable: Each store is composed of about ten teams. On average, ten teams make up a store, and each team’s leader participates in a higher-level team that manages the store. The team leaders of each store in a region make up a regional team, and the six regional presidents make up the team that manages the company.

Loose coupling.

Each Whole Foods team operates as an autonomous unit that has control over its own fate. Performance data is available to all the teams, so they can compare their performance against other teams in their store, similar teams in other stores, or against their own team’s historical performance.

Teams also have access to detailed financial data, like product costs, profits per store, and even each others’ compensation and bonus information. They can look up the best-selling items at other stores and compare them to their own. Employees at Whole Foods are so well-informed that the SEC has designated all employees “insiders” for stock trading purposes.

This data transparency both builds trust and fuels a spirit of intense competition between teams and stores, since every team can compare itself with every other team and try to raise in the ranks.

Whole Foods has created a platform that makes it possible for the company’s stores and teams to compete with each other so they can tune and improve their performance over time.

At the same time, each team has the autonomy to make local decisions as they see fit to improve their performance. So every Whole Foods store carries a unique mix that is tailored by self-managed teams for that particular location. Whole Foods’ agile, team-based strategy allows it to target extremely small locations with customized stores. They are starting to open small stores in suburbs and college towns where rents are lower and competition less fierce.

Customers like the system. The industry average sales per square foot is about $350, and Whole Foods is one of the top ten retailers in the US, with sales of about $900 per square foot, higher than Best Buy and Zale jewelers. Not bad for a grocery store.

Employees like it too. Whole Foods has made Fortune’s “100 best places to work” list every year since the list was started in 1998.

Most companies are not built for agility.

Most businesses today are not designed with agility in mind. Their systems are tightly coupled, because their growth has been driven by a desire for efficiency rather than flexibility.

Consider the difference between a car on a road and a train on a train track. The car and the road are loosely coupled, so the car is capable of independent action. It’s more agile. It can do more complex things. The train and track are tightly coupled, highly optimized for a particular purpose and very efficient at moving stuff from here to there — as long as you want to get on and off where the train wants to stop. But the train has fewer options — forward and back. If something is blocking the track, the train can’t just go around it. It’s efficient but not very flexible.

Many business systems are tightly coupled, like trains on a track, in order to maximize control and efficiency. But what the business environment requires today is not efficiency but flexibility. So we have these tightly coupled systems and the rails are not pointing in the right direction. And changing the rails, although we feel it is necessary, is complex and expensive to do. So we sit in these business meetings, setting goals and making our strategic plans, arguing about which way the rails should be pointing, when what we really need is to get off the train altogether and embrace a completely different system and approach.

This seems simple when you think about it. But it’s difficult to do. It’s hard to even think about it, especially when you are sitting on a business train that’s going a hundred miles an hour and you feel like it’s headed in the wrong direction.

Excerpted from The Connected Company by Dave Gray.

Dave Gray is the Founder of XPLANE and author of Selling to the VP of NO, The Connected Company and Gamestorming.