The first quotation comes from the blog of ITSM consultancy iCore. Andy Roberts observes that the emergence of DevOps is forcing the Change Manager to shift their role towards:

“…governance of the change processes, rather than the management of individual changes, evolving from a policeman to a supervisor”

This viewpoint is increasingly common, and it can seem reasonable at face value. The ITSM community is generally aware of its reputational issues. It knows that its own bulky Change Management processes have historically contributed to that problem. Roberts’s blog, in common with many similar commentaries, argues for a future version of Change Management in which DevOps teams must:

“…prove, to the Change Manager, that the processes that define their deployment pipeline can be certified to be categorised as being standard changes.”

I’ve written before on the cultural challenge undermining this viewpoint, but let’s set that aside, and move on the other quotation, which comes from a very different perspective: the world of cloud-native software development. Tony Hansmann is a platform architect with Pivotal. In an operations-focused episode of that company’s own “Conversations” podcast (timestamp 43:00) he argued the case for complete automation:

“There are no untestable things. You must first accept that if you’re dealing with computers you can always test an assertion. Once you come to accept that, then ITIL can be entirely subsumed. ITIL does not require a person to validate. If you have to have X, Y and Z processes, great. Just write the assertions for that process, or automate that process: You can keep your ITIL.”

Each of the statements here reflects the same goal, of course: Everyone wants to be able to implement stuff quickly, without breaking things. However, we can’t use this alone as an argument that Continuous Delivery can be aligned easily with existing ITSM norms. In fact, these two quotes highlight some stark differences.

To illustrate those differences, it is first important to understand that “Standard Change” has a very specific definition (well, of course it does) in ITIL. To be classed as “standard”, a change must be recurrent, well-known, and “proceduralized to follow a pre-defined, relatively risk-free path”. It must also be the “accepted response to a specific requirement or set of circumstances”. If these conditions are met, ITIL allows for advanced approval of changes. Hooray — no Change Advisory Board meeting!

ITSM, defined this way, puts the Change Manager in a position of overarching governance. But while Roberts describes centralisation: a single certificating authority, led by a human persona, with sign-off responsibility over each pipeline, Hansmann advocates decentralisation and automation. There is, he argues, no need for incidental validation by a human “Change Manager”.

Continuous Delivery is effectively stating that “Yes, we govern releases so that they don’t break things. But not your way”.

It’s enlightening to illustrate this difference with real world examples. Let’s look at a state-of-the-art delivery methodology: Google’s “Site Reliability Engineering”. The result of over a decade of evolution and innovation, SRE is described with great detail and clarity in this interview with Ben Traynor, a Google VP of Engineering. Here’s a summary of some of the key characteristics:

• If you don’t quite make it through the Google software engineering interview process, but you came close, Google might hire you instead as a Site Reliability Engineer.
• SREs are in an operations role, but they are developers. Google seeks to hire people who “are inherently both predisposed to, and have the ability to, substitute automation for human labor”.
• SRE teams are responsible for “availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning”.
• At leat 50% of an SRE’s time is expected to be dedicated to coding.
• Site Reliability Engineers act in a supporting role to software engineers, but also in an enforcing role.

To incentivise developers not to break things (and therefore create extra work for SREs), Google has developed a concept called “Error Budget”:

The business or the product must establish what the availability target is for the system. Once you’ve done that, one minus the availability target is what we call the error budget; if it’s 99.99% available, that means that it’s 0.01% unavailable. Now we are allowed to have .01% unavailability and this is a budget. We can spend it on anything we want, as long as we don’t overspend it.

If its error budget is not being eroded by bugs, the development team gains significant autonomy. They can take risks: if they want to release a new feature early, with a reduced test cycle, they can do so. If something breaks in the process, that’s not held against them, provided they stay within their error budget. However, once that budget is overdrawn, SREs withdraw their support of the development of that application. At that point, the developers can only action critical fixes, until their error budget is back in the black.

In contrast to “standard change”, the permission-to-proceed given to Google’s software engineers is not limited to “accepted responses to specific requirements”. SREs are the change managers, but even in that role they don’t care what the requirement is, or what the response to those requirements is. You are allowed to do anything you want, provided you have not broken too much already, and you don’t do so in future.

With no centralised certification of the deployment pipeline, each application development team is instead accountable to targets which are measured and enforced by its assigned SREs. SREs are even free to move from application to application (which is another incentive to build solid applications: if you make a good SRE’s job hard, they can move to another application).

Of course, the innovation pipeline itself is undergoing continuous development and innovation, because the SREs themselves are coders, spending more than half of their time innovating that pipeline. Hence, we don’t even have “a pre-defined path”.

The methodology, therefore, is clearly differentiated, on a number of grounds, even from a liberal reading of ITIL’s “standard change”.

I remain convinced that IT Service Management has a huge amount to offer a digitally transforming enterprise shifting towards agile methodologies, software innovation, and continuous delivery. I am very sceptical, as it happens, of Hansmann’s assertion that every validation of a change can be automated, because in practice this relies on being able to quantify not just every technical check, but also many pieces of tacit human knowledge which will be much more difficult to express as code. Real life isn’t always Turing Complete.

However, the fact that two processes have similar goals does not mean that they are the same process. As Google, and countless other organisations, have illustrated, continuous integration and delivery is evolving, rapidly, past the semantic constraints of ITIL’s “standard change”. That’s a problem for ITIL.

I think for example of Google’s “error budget” concept which enables Dev teams to do stuff beyond the usual constraints, provided they stay within their availability bounds. Want to rush out that new feature on limited testing? Sure, just don’t let it blow your 99.9% target. That 0.1% is all yours to “spend" as you wish.

One famously visible feature of the Toyota Manufacturing System is the Andon cord. Installed at each station of Toyota’s Lean manufacturing lines, this cord was within reach of every production worker. Anyone on the line can pull the cord (or push a button), to indicate that something is wrong. Pulling the cord triggers remedial action. It may even result in the stoppage of the line, enabling the issue to be resolved before production restarts.

With Andon, if something is wrong, it does not need to be escalated laboriously up a chain-of-command. Any worker on the production line, whatever their pay grade or experience, has the right to pull the cord.

As a result, it represents a degree of autonomy which exceeds most industrial convention. This autonomy is one of the cultural building blocks of Lean methods. It is important for any outside observer, seeking to understand a Lean-derived movement like DevOps, to recognise this.

I have frequently discussed IT Service Management with DevOps practitioners. Sentiment isn’t always positive, particularly when certain topics are raised: If Andon is one symbol of the Lean movement, then ITIL’s Change Advisory Board (CAB) meeting is a potent symbol of ITSM for many in the DevOps world.

Remember: a key goal for DevOps teams is the establishment of a high cadence of trusted, incremental production releases. The CAB meeting is often seen as the antithesis of this: a cumbersome and infrequent process, sucking a large number of people into a room to discuss whether a change is allowed to go ahead in a week or two, without in reality doing much to ensure the safe implementation of that change.

This perception is perhaps unfair to a degree, but it has significant grounding in reality. I have myself sat through dozens of these meetings in previous roles. There were occasions when I’d probably have pulled the ceiling down, had there been an Andon cord attached to it.

Fortunately, ITSM has evolved its thinking over the years, and plenty of effort has been made to adapt its messaging. This article from ITIL custodians Axelos, for example, points out that…

Having the CAB review every single change request isn’t efficient, and it’s definitely not common sense. However, having the CAB review change requests of unknown risk, when parts of the business need to be consulted because they might be impacted, makes a lot of sense.

This might seem reasonable at first glance. However, does it really pass what might be called the “Andon test”?

One issue that stands out with the sentence is the notion of changes “of unknown risk”. Much of the DevOps movement’s achievement has been built on the removal of the concept of “unknown risk”, through automated testing and a safe-by-design deployment pipeline. This is illustrated nicely in Gene Kim’s brilliant article about the transformation of HP’s Laserjet firmware team:

To support self-testing builds, they built a set of automated unit, acceptance and integration tests, which would continually be run against trunk. Furthermore, they created a culture that “stopped the line” anytime a developer checked in code that broke the build, broke a unit test, etc.

This is Andon, automated: nothing gets to production if anything is going to break production.

But even if we set aside this point, there is still another significant issue with the Axelos statement: the assertion that CAB meetings “make a lot of sense” for identifying and managing stakeholder risk.

Here’s one problem with this: DevOps champions make the case that DevOps is reducing risk far more effectively than legacy processes. The 2016 State of DevOps Report (a commercially sponsored paper, but one which builds on academic research by Nicole Forsgren and Jez Humble) argues that the difference between “high performing” and “low performing” organisations is considerable. High performing organisations — that is, those who have embraced the best practices of Agile development and Devops — were found to be achieving incredible numbers:

• 200 times more frequent deployments than low performers
• 2,555 times faster lead times.
• 24 times faster recovery times
• Three times lower change failure rates.

Fundamentally, these are all leading or trailing measures of the effectiveness of the Change Management process. So what does this do for the assertion that a structured CAB meeting “makes sense”? Why should this particular structure be accepted as the means of achieving the goal of determining impacts and involving stakeholders? DevOps seems to be doing just fine without it.

In his book Toyota Kata, author Mike Rother highlights the problems with a focus on existing solutions over new ways of solving challenges:

Toyota opens its factory doors to us again and again, but I imagine Toyota’s leaders may also be shaking their heads and thinking, “Sure, come have a look. But why are you so interested in the solutions we develop for our specific problems? Why do you never study how we go about developing those solutions?” Since the future lies beyond what we can see, the solutions we employ today may not continue to be effective. The competitive advantage of an organization lies not so much in the solutions themselves but in the ability of the organization to understand conditions and create fitting, smart solutions.

There is plenty of opportunity in enterprise IT to find such solutions. DevOps isn’t perfect, because no methodology is perfect. The business structure of an IT organization is broader and deeper than DevOps, and there are many challenges to be solved now and in future. ITSM may offer the best solutions to many of those challenges, but that value needs to be proven, not asserted.

Indeed, Oracle’s stuff is there already:

This is, I’m sure, a welcome development for anyone wishing to deploy Oracle stuff in a Docker container. These are official Oracle docker images, and will probably save people a lot of manual effort (or persuade them not to go down the more questionable path of using an unofficial image from the public Docker Hub. These do exist, by the way, even though they feel akin to buying a CD-R with “ORACLE” scrawled on it in marker pen).

The keynote included a hypothetical demonstration scenario, in which one of those Oracle images was used. The story goes something like this:

• Two developers are grabbed by the CEO (whose neck they have already saved in a previous phase of the demo, of course), because the company has just acquired another business.
• The company they’ve bought uses a ‘traditional on-premise application’, and there is concern that the cost of migrating it into modernity will impact the next day’s merger announcement.
• Once again, the hapless CEO is saved by the clever developers, who use Docker’s clever tools to enact a rapid transformation of the clunky old application. Hooray, now it can run in Docker containers and be deployed to the cloud!
• Oh, and one last thing… the application uses Oracle, so in a flash, they pull an Oracle database server image off the Docker Store, and deploy it to run with the application.

Okay, this is an impressive story (notwithstanding the argument that HypotheticalCorp might want to ask a few big searching questions of their merger due-diligence people). But there’s a serious gotcha here: as any Software Asset Manager could point out, these actions could have just cost the company a pretty staggering amount of money. How? By falling foul of Oracle’s notoriously complex licensing system.

Let’s look at a few key points here:

Firstly, the company has just merged with another company. You might as well paint a big bullseye on your HQ, and force all staff to tape “AUDIT ME” on a piece of paper to their back, because this is one of the primary triggers of vendor compliance checks.

Secondly, this application is now moving from a dusty-old on-premise installation, to whatever “the cloud” looks like in this case. That might be a public cloud provider, in which Oracle has you covered here (PDF), provided you are using Amazon or Microsoft Azure (if not, then congratulations, you’ve entered the well-known software licensing realm of “who the hell knows?”). You’re still going to be paying money, but at least it’s only for what you use, so just watch that sprawl, okay?

The real issue comes if the stuff is being deployed on a cloud which is hosted on some identifiable physical infrastructure. Now, things can get very, very expensive.

(A quick history lesson: until some time last decade, it was very easy for big middleware software vendors to figure out how much to charge their customers. They’d find all the metal-boxes-with-blinking-lights on which their stuff was deployed, count the CPU cores, and charge accordingly. But then, VMWare and others came along and muddied the water. As their customers could now wrestle a whole lot more effective utilisation out of those metal boxes, many of those vendors had to change their formulae, to avoid being paid a lot less money. To run those products on VMs, customers now needed to purchase expensive licenses corresponding to the capacity of the physical machines underpinning the virtual environment.

As virtual technologies evolved, it just kept getting worse. Many companies simply failed to manage it, often through carelessness or ignorance. Software auditors filled their boots, creating a whole new software sub-industry worth literally billions of dollars. It’s a cycle of complexity that just gets worse with each new development in infrastructure. I wrote about this stuff five years ago, and that article still holds true: IBM now has something like 50 different virtualisation capacity counting methods.

Anyway, we’ll just pause for a moment to let the Open Source people stop laughing).

The specific issue with Oracle is that they have a pretty hardline approach to what needs to be licensed, as defined in their ‘Partitioning Policy’. If you’re using one of the technologies listed in that document as “hard partitioning” (such as, for example, capped Solaris Zones, or IBM LPARs), then you’re in luck: you only need to pay license fees based on the CPUs underpinning those partitions.

However, pretty much any other virtualization technology likely falls into Oracle’s definition of “soft partitioning”. Oracle now want their software to be licensed for every CPU in the physical server farm.

The plucky developers in Docker’s example might just, therefore, have exposed the company to a huge licensing demand. This has happened before, one example being the huge spat in 2016 between Mars and Oracle over the former’s use of VMWare.

I’ve presented on numerous occasions to Software Asset Managers and ITSM people about the implications of a rise of containerization as a defacto virtualisation standard in enterprises. SAM practitioners know only too well about the complexities of Oracle’s licensing in virtual and cloud environments. People in the Docker world are now starting to notice too:

I have to make a disclaimer here: Oracle licensing is bloody complex, and it’s entirely possible that a goalpost or two might have moved by the time you read this.

But this scenario is a really good example of the importance of aligning the rapid, wonderful, software-is-eating-the-world brilliance of agile, DevOps and Digital Transformation, with some experienced heads from the “old worlds” of ITSM and IT Asset Management, at least once-in-a-while. In Docker’s story, developers saved the day, but a Software Asset Manager might save the company.

The article revealed that an investment in Apple stock in 2009 would have yielded a healthy 370% return by 2017. However, an investment into the comparatively unglamorous Dominos would have returned the investor’s stake more than 20 times over.

In fact, Dominos has outperformed a raft of extremely successful tech companies over that time.

The story predictably generated a mixed and sometimes incredulous reaction. Much of the analysis (including the article linked above) focused on pizzas, but missed the key point: Dominos Pizza is one of the most digitally transformed enterprises in the world. To adapt an overused cliché: they are now a technology company that sells pizza.

Take their British subsidiary, for example. Dominos UK only took its first mobile phone order in 2010. At that time, online sales accounted for less than 30% of revenue. Seven years on, digital transactions account for 80%, with the UK smartphone app accounting for more than two thirds of those sales.

Worldwide, the company has innovated and experimented, embracing almost every popular consumer technology. In some countries, customers can opt to order favourite pizza as soon as their app opens (a ten second countdown is the only protection from accidental deliveries). They can order via a Facebook messenger bot, a Tweet, or any of an increasing number of smart devices.

The company has embraced agile delivery. 2014 saw the launch of the “Pizza Mogul” service in Australia, enabling customers to design and market their own pizzas, sharing revenue with Dominos. This came to market after less than five months of development.

In short, the company has relentlessly focused on a technology revolution. CEO Patrick Doyle recently pointed out to Harvard Business Review that fully 400 of its 800 head office employees work in software development and analytics.

In a remarkably short time, Dominos has changed from traditional walk-in/phone-up chain retailing, to being one of the great digital success stories.

Critically, it has done this by focusing its technology resources on the front-line. While Enterprise IT shops have focused inwardly on the “Consumerisation of IT”, Dominos has concentrated on “IT’izing the consumer”. In doing so, they have carved big share out of an intensely competitive market.

This is a hugely important point. Most Enterprise IT is built on models and processes that pre-date this kind of transformation. The customer is the employee. Even when it looks outwards, ITIL still defines an external customer in business-to-business, rather than consumer, terms:

A customer who works for a different business from the IT service provider.

However, this is not an accurate description of the kind of customer who may experience an issue with an application like Pizza Mogul. They are members of the public. They may or may not work for a business (maybe they are simply living off their Pizza Mogul successes).

A customer experiencing an issue with an externally facing system is more likely to interface with the business’s customer service department than its IT channels (and hence critical issues may first appear in the CRM system, rather than in the normal trouble ticketing tool). Marketing, rather than IT strategy, becomes the primary driver of changes. Those changes (or fixes), incidentally, are delivered rapidly to the front-line directly by the software developers, with the path to production smoothed by DevOps practices which evolved outside the ITSM ecosystem.

This illustrates the big disconnect between the traditional model of Enterprise IT, and the kind of innovation that has driven such success at Dominos. Corporate IT does, of course, have a very important role to play in enabling and engaging employees. The Digital Workplace is a hot topic, and it will create huge benefits in areas such as operational efficiency, communication, and even employee retention. However, it’s difficult to imagine any such internal initiative producing the equivalent of Dominos’ 2000% growth.

Every CEO knows this, and as a result, when most enterprises talking about digital transformation, they are looking outwards. While few have gone all-in like Dominos, many have significantly increased their commitment to in-house development of customer-facing applications.

I’ve had many conversations with both developers and ITSM representatives at these companies, which have highlighted consistent and increasing challenges: Even high-quality customer service departments lack the technical sophistication to deal with incoming technical issues and transfer them to the right team in the correct way. IT service departments struggle to get visibility, particularly once tickets are thrown “over the wall” to developers’ SDLC tools. And developers lack context on the customer, which makes prioritisation difficult. Often a large number of developers are integrated with support processes for the first time (a travel company told me last week that as many as 3000 developers might receive tickets, a number far bigger than their internal IT organisation), which may amplify these issues.

Most enterprise IT functions were built around service provision, rather than product innovation, and around the needs of employees rather than external customers. Digital transformation will change this to a greater or lesser extent for every company.

This morning, I received a LinkedIn post notification which made the following statement:

“At its core, DevOps is simply about better collaboration between development and operations teams. In fact, it should be seen as a means for improving ITIL® processes”

This statement feels very representative of much of the current dialog (there are plenty of other recent examples to be found). However, this logic, while reasonable (and sensibly aspirational) risks missing a wider point. Yes, DevOps is about collaboration, but it is merely one set of practices and philosophies which have evolved as part of a much broader digital transformation revolution. Many other things have also changed, from the underlying technology stack right through to the recipient customer.

History provides numerous examples of organisations which got very good at optimising what they did, only to find that the world had moved on without them. While IT organisations have great opportunities to add huge value on an ongoing basis, we can not assume that digital transformation will be best served by “ITSM done better”.

DevOps is coming to IT enterprises, whether they are ready for it or not. In this article, I aim to do something that might be considered brave: I will argue that the current organizational structure of the vast majority of IT support organisations is fundamentally flawed.

More importantly, those flaws will make it difficult or impossible for those enterprises to achieve successful integration of their emerging DevOps practices, with their existing structures for technical customer support.

I will propose, instead, that the emergent practice of “Swarming” is an enterprise-ready methodology that is ideally placed to being technical support into the DevOps era.

Background: The Three-Tier Support Orthodoxy

We need to start with a short overview of the management structure which underpins the vast majority of large enterprise IT support functions.

The classic organisational structure for IT Service Management is the three-tier support hierarchy:

• Level 1: A frontline Service Desk, directly fielding incoming customer communication (typically by answering phone calls).
  Most Service Desks are set up to provide a moderate level of generalised technical support, with the aims of presenting a consistent level of customer support to users, while resolving a significant number of incoming issues at the first-line.
• Level 2: A second tier of support, often closely associated to the Service Desk, but with deeper general or specialist skills.
  Level 2 support agents may, for example, have additional training in support of common operating systems (such as Microsoft Windows) or hardware, and hence be able to resolve more complex issues affecting common technologies.
• Level 3: Specialist support teams focused on specific technologies and applications. For companies which develop software in-house, it is typical to find specific Level 3 support teams assigned to individual applications or services.

Deconstruction of the three-tier structure requires a brief analysis of the business motives for it. It is almost ubiquitous in enterprise IT Service Management, and there are a number of business benefits driving this, which include the following:

• Customers are presented with a single communication channel to the IT support organisation, regardless of the nature of their issue.
• The general technical support skills needed to work in Tier 1 and Tier 2 support are easily found in the workforce. This also makes outsourcing of one or both of these layers straightforward, and as a result this is commonly seen.
• Specialist technical resources can be insulated from direct contact, ensuring that only properly triaged issues reach them.

The journey of a customer’s case through this structure may start and end at the first line (in fact, in many organizations, customers have the opportunity resolve their issue through automated self-service — often described as “Level zero”).

There are inevitably many issues, however, which are not resolvable by Level 1 support. These progress to Levels 2 and 3 through a process of escalation:

Level 2 support agents typically handle fewer cases than their Level 1 counterparts, but these tend to be more complex, with a longer average effort on the part of the agent.

Tickets which make their way to Level 3 (either from a secondary escalation from Level 2, or directly from Level 1) typically account for a small volume of the overall incoming caseload, but they are also the most complex issues, requiring the most specialist skills, and the most time to resolve them.

There have been frequent attempts to benchmark the comparative cost of resolving a ticket at each level of support. This 2014 study, for instance, assesses the average cost of a Level 1 resolution as $22, with Level 2 resolutions costing $62, and Level 3 resolutions $85 (other studies have calculated the Level 3 resolution cost to be several multiples higher than this number).

Why Three-Tier is a problem, especially for DevOps

Challenging such a ubiquitous structure is challenging. However, the Swarming movement aims to do just that, on the basis of some significant addressable disadvantages to tiered support. Many of these disadvantages have particular consequences for DevOps:

• Tiered Support creates multiple queues.
  While Level 1 support tends to be reactive and realtime, any case that can not be resolved at this level immediately enters a queue. Its nature changes, turning it from a current activity into a backlog item.
  As such, Levels 2 and 3 are stores of Work in Progress, a very problematic concept in the Lean philosophy which underpins the DevOps movement. Successful adoption of Lean practices like DevOps fundamentally requires assertive steps to reduce systematic Work in Progress. This alone is a major barrier to DevOps practitioners’ adoption of IT service management practices
• Tiered Support blocks the route to the correct resolver.
  DevOps aims to promote increased ownership and autonomy. Developers are encouraged to take responsibility for the support of their own code. The highest performing DevOps organizations are achieving significantly faster resolutions on this basis (24 times faster, according to the 2016 State of DevOps report). However, this all comes to nothing if the ticket still crawls through several triage queues on the way to that expert. As one support manager at BMC put it to me, when discussing the company’s adoption of Swarming for customer support, “why were we putting our best people at the back of the process?”.

• Tiered Support leads to cases “bouncing”
  In tiered support, the method of moving a case between them is simply to change the team which is assigned to it. This step is typically carried out unilaterally by the assigning team to the assignee team. The first time the new assignee sees the case ticket is when it arrives in their queue.
  Unfortunately, it is very common to see the ticket bouncing right back, either because the more specialised team requires further information to proceed, or because the assignment to them was completely incorrect.
  DevOps is fundamentally built on collaboration between operational and development professionals. The vertical and horizontal silos inherent to Tiered Support, and the passive handovers of work between them, are the antithesis of this inter-disciplinary collaboration.

• Tiered Support does not solve the problem of Subject Matter Experts becoming overwhelmed
  While one positive outcome of multi-tiered support is the prevention of easily-solved tickets finding their way to teams overqualified to work on them, it does not protect key specialists from high volumes of difficult cases. IT Service Management is plagued by “heroes”. These are typically very clever people who — at face value — appear to be incredibly valuable contributors to the success of the organization, repeatedly producing miracle fixes to tough issues. In reality, the hero is an overworked, burnout-prone single-point of failure, deliberately or inadvertently acting as the custodian of knowledge that the organization badly needs to be disseminated more widely. Tiered support, being a linear and siloed structure, does nothing to prevent the cult of the Hero. Arguably, it simply reinforces it.
  As enterprises shift to DevOps, we are already seeing the perpetuation of this scenario, with key DevOps team members appearing at the end of the escalation chain for high-volume spikes of tickets. The damage in a DevOps scenario is arguably worse: key developers are taken away from innovation to deal with a firehose of escalated (and already delayed) support issues.

Introducing Swarming as an alternative

“Collaborative communities can reach across the usual disciplinary and organisational silos that inhibit cooperation, learning, and progress”
(Don Tapscott and Anthony D. Williams, in “Wikinomics”)

“Swarming” appeared late in the last decade as a proposal for a new framework for technical support organisation. It explicitly rejects the three-tier orthodoxy, in favour of a model of networked collaboration:

A key pioneer for IT support was Cisco, who set out their new “Model for Distributed Collaboration and Decision Making” in a 2008 white paper, “Digital Swarming”. The concept was subsequently adopted by the Consortium for Service Innovation, and developed into a vision entitled “Intelligent Swarming”. Some of its core principles, in direct opposition to the orthodoxy, are that:

• There should be no tiered support groups.
• There should be no escalations from one group to another.
• The case should move directly to the person most likely to be able to resolve it.
• The person who takes the case is the one who sees it through to resolution.

Swarming in practice: an example structure for DevOps

There is not a single definitive structure for Swarming, particularly as it is a relatively new and unadopted concept. However, the example illustrated below (based on customer support swarming methods in place at BMC) is typical, and has generated significant improvements (as presented at the UK’s Servicedesk and IT Support Show in 2015).

Swarming starts as soon as any issue is not immediately resolvable at the point of customer contact. A rapid initial triage results in the distribution of case tickets to one of two “Swarms”:

Each “Swarm” is actually a small team, focused in near-realtime on the incoming flow of customer cases:

“Severity 1” Swarm

• Three agents working on a scheduled weekly rotation.
• Primary focus: Provide immediate response, and resolve as soon as possible.

A Severity 1 swarm is focused on the most critical issues. They coordinate the response to an acute situation, bringing appropriate people into the effort to resolve severe cases as quickly as possible. This process is not in itself different to the Major Incident processes typically employed in traditional tiered support. However, the other type of swarm used at this stage, to which the much larger volume of tickets goes, is different:

Dispatch Swarm

• Meet every 60–90 minutes
• Regional, product-line focused
• Primary focus: “Cherry pickers”. What new tickets can be resolved immediately?
• Secondary: Validation of tickets before assignment to product line support teams.

Dispatch Swarming addresses a key shortcoming of tiered support: many cases can be solved extremely quickly by the right expert, but they get lost in the backlog. Hence, a five minute resolution may actually take days.

The Dispatch Swarm is encouraged to “cherry pick”, disregarding anything that can not be resolved very quickly. In doing so, they are able to dramatically shorten the time spent achieving resolution for a significant subset of escalated cases.

There are significant secondary benefits, too. The inclusion of inexperienced frontline support staff in these Swarms gives exposure to knowledge that would otherwise only start to be gained after eventual promotion to more specialist teams. Meanwhile, conversely, third-tier support agents are brought closer to the customer.

Dispatch Swarming achieves rapid resolution of a significant minority of cases (at BMC, typically this is in the order of 30%), but the remaining cases will end up in the queues of more conventional-looking Product Line support teams. Here, many tickets will be quite straightforward for regular team members to work. However, another subset (perhaps 30% again) may be challenging enough to warrant attention from the best support people available, regardless of team structure.

This is where a third type of Swarm is used: the Backlog Swarm.

Backlog Swarm

• Meet regularly, typically daily.
• Primary focus: Address challenging tickets brought to them by product-line support teams.
• Secondary: Replace the role of individual subject matter experts.

Backlog Swarms bring together groups of skilled and experience technical people, crossing boundaries such as geography and department, with the objective of focusing on the most difficult cases. Cases are referred to them by local engineering and support teams, who are no longer permitted to directly engage individual subject matter experts. They must, instead, always refer those cases to the appropriate Backlog Swarm.

The Case for Swarming, as Enterprises adopt DevOps

The failings of tiered support in traditional enterprises are magnified in a DevOps scenario. The system actively creates Work in Progress backlogs. It restricts autonomy and agility. It is fundamentally siloed. These issues are the antithesis of the DevOps philosophy, and this is becoming a major industry challenge as large, traditional Enterprises seek to build DevOps capabilities.

It is already possible to observe the negative outcomes resulting from this.

• DevOps encourages the teams building software to take ownership of supporting it — sometimes colloquially summed up as “you wrote it, you fix it”. However, in an enterprise-scale support enterprise, the tiered support structure is typically the primary path along which these cases arrive. As we have seen, the layers of separation between the front-line and the DevOps team can result in tickets arriving late, and poorly qualified.
• “Throw it over the fence” integrations between ITSM ticketing tools, and the software development lifecycle tools used by DevOps teams, result in a lack of situational awareness for users of each.
• The attempt to force a rigid vertical and horizontal silo structure creates boundaries to the cross-collaboration that is key to good DevOps practice.

Swarming, conversely, is built on many of the same principles that underpin the success of DevOps:

• Dynamic cross-functional collaboration, bringing different skills together into combined teams.
• Flexible team organization, rather than rigid, hierarchical structures.
• Individual autonomy, rather than dogmatic process (a key example being the opportunist “cherry picking” of the Dispatch Swarm).
• A focus on the avoidance of build-up of backlogged Work in Progress.
• Cross-pollination of skills and experience.

Conclusion

“The enterprise space doesn’t move slowly because they’re stupid or they hate technology. It’s because they have users.”
(Luke Kanies, founder and then CEO, Puppet Labs. Configuration Management Camp, Belgium, 2015)

DevOps has grown rapidly as a fundamental challenge to an established orthodoxy, bringing together the previously siloed roles of development and operations, and aggressively targeting long-established inefficiencies and dogma. It has also largely (if not entirely) grown in a new generation of organisations, often with the benefit of limited legacy and technical debt.

Importantly, it has done so tremendously successfully:

Now, however, DevOps is reaching traditional enterprises, where it will inevitably meet new challenges, just as those organisations will frequently struggle to adapt to it. That they need to is hard to deny. It is not just a matter of improvement; it is a matter of survival. Change, in the form of “creative destruction”, is a constant existential threat to large enterprises. Only 12% of the Fortune 500 in 1955 were still present in it in 2014.

As a result, IT organisations must adopt fresh thinking, and challenge existing orthodoxies wherever possible.

The Swarming movement has begun to deconstruct and attack the prevalence of the tiered support model, but the progress in enterprise IT Service Management has been slow, limited to a few forward-thinking organisations. However, Swarming’s similarities to key elements of DevOps thinking are undeniable, and the real-world organisational problems addressed by it are amplified into fundamental challenges for DevOps adoption.

There is therefore a compelling and urgent need to rethink the tiered support model, with a methodology that harnesses and enables the benefits of DevOps, while doing so on an Enterprise support scale. Swarming could be the answer.

More related recent articles:

The Andon cord and ITSM’s DevOps challenge
On digital transformation, ITSM, and DevOps. Oh, and pizza.
We’ve been getting “DevOps vs ITIL” wrong

Simply meshing processes together because they look like they fit, in my experience, tends to result into “throw it over the fence” system integrations, and a lot of WORN (“write-once, read-never”) data.

In 2014, DevOps pioneer and “Phoenix Project” author Gene Kim, wrote an article for the ITSM Review: “Trust me: The DevOps Movement fits perfectly with ITSM”. It’s an good read, because it frames the subject of integration in terms of opportunities to create value:

“…find the automated infrastructure project (e.g., puppet, chef) that provisions servers for deployment. We can help that team with our release management readiness checklists, security hardening checklists and so forth”

“… information from IT Operations is radiated to Development… IT Operations is where value is created, and this feedback is required in order to make good decisions.”

DevOps is, essentially, an application of Lean thinking to software development and deployment. A core principle of Lean is the removal of inefficiencies from the value chain. If it can’t be demonstrated that any integration adds value or removes cost, then no principled Lean practitioner should feel able to adopt it.

Hence, it’s not adequate for ITSM just to ask DevOps to conform to its processes, no matter how brilliantly optimised and streamlined they are. It’s critical to start with why.

None of this should be a surprise. The Agile Manifesto, effectively the DevOps movement’s Declaration of Independence, explicitly values individualism over process, and reactiveness over structure. The manifesto is the specific antithesis of the traits seen in that negative perception of ITSM.

ITSM commentary on DevOps, meanwhile, is inconsistent, ranging from outright confusion to sheer overconfidence. The complaints of the DevOps community are frequently acknowledged, but they are often waved away on the basis that ITSM is “just a framework”, and hence it should be perfectly possible to fit Devops within that framework. If that doesn’t work, the framework must have been implemented badly. Again, this is a reasonable point.

But there’s a recurring problem with the debate: it tends to focus primarily on processes: two ITIL processes in particular. ITSM commentators frequently argue that Change Management already supports the notion of automated, pre-approved changes. DevOps “is just mature ITIL Release Management”, stated an opinion piece in the Australian edition of Computerworld (a remarkable assertion, but we’ll come to that later). Some of the more robust sceptics in the DevOps community focus on ITSM’s process silos and their incompatibility with the new agility in software development.

Certainly, the ITSM community has to realise that there is a revolution happening in software production. Here are some statements which are easy to back up with real-world evidence:

• DevOps methodology fundamentally improves some of the inefficiencies of old, waterfall-driven processes.
• Slow, unnecessarily cumbersome processes are expensive in themselves, and they create opportunity costs by stifling innovation.
• Agile, autonomous teams of developers are unleashing creativity and innovation at a new pace.
• With small, frequent point releases, systems are growing in a more resilient way than monolithic releases tended to achieve.

Unarguably, the new methodology is highlighting the shortcomings of the old. Can anyone argue today that a Change Advisory Board made of humans, collating verbal assurances from other humans, is preferable to an effective, fully-automated assurance process, seamlessly integrated with the release pipeline?

We know, then, that DevOps methods dramatically improve the speed and reliability with which technology change can increase business value. But that’s where the arguments on both sides start to wear thin. What is that business value? How do we identify it, measure it, and assure its delivery?

In my experience, there is little mention of the customer at DevOps events. DevOps is seen, correctly, as a new and improved way to drive business value from software development, but the thinking feels very “bottom up”. There is huge willing to drive business value, but is there enough top-down, customer-first thinking? ITSM commentators seem to have taken the same starting point: drilling into minutiae of process without really considering the value that ITSM should be looking to bring to the new world.

To highlight why a lack of service context is a problem, let’s take the simple example of frontline support. When developers push out an incremental release to a product, customers start to use it. No matter how robust the testing of that release was, some of those customers will encounter issues that need support (not every issue is caused by a bug that was missed in testing, after all).

The Service Desk will of course try to absorb many of those issues at the first step. That is one of its fundamental aims. To do this effectively, it needs to have reasonable situational awareness of what has been changing. It is not optimal for the Service Desk only to become aware of a change when the calls start coming in. Ideally, they should be armed with knowledge of how to deal with those issues.

No matter how effective the first line of support is, some issues will get to the application team. Those issues will vary, as will the level of pain that each is causing. Triage is required, and that is only possible if there a clear understanding of the business and customer perspective.

Facing a queue of two tickets, or ten tickets, or one hundred tickets, the application team has to decide what to do first. This is where things start to unravel for an idealistic, full-stack, “you break it, you fix it” devops team. Which issues are causing business damage? Which are the most time critical? Which can be deferred? How much time should we spend on this stuff at the cost of moving the product forward? This is the stuff that Service Management ought to be able to provide.

Effective Service Management in any industry starts with a fundamental understanding of the customer. Who are they? What makes them successful? What makes them tell other potential customers how great you are? What annoys them? What hurts them? What will trigger them to ask for a refund? What makes them go elsewhere altogether? And, importantly: what is it that we are obligated to provide them with?

An understanding of our service provision is fundamental to creating and delivering innovative solutions, and supporting them once they are there. This is where ITSM can lead, assist, and benefit the people pushing development forward.

The “S” in ITSM stands for “service”, not process. The heavy focus on process in this discussion (particularly two specific processes, close to the point of deployment) has been a big mistake by both communities. It is wholly incorrect to state that DevOps is predominantly contained within Release and Change Management. Code does not appear spontaneously in a vacuum. A whole set of interconnected events lead to its creation.

I have been in IT for two decades, and the DevOps movement is by the biggest transformation in software development methodology in that time (I still have the textbooks from my 1990s university Computing course. These twenty-year-old tomes admonish the experimenting “hacker” and urge the systems analyst to complete comprehensive designs before a line of code is written, as if building software was perhaps equivalent to constructing a suspension bridge).

The cultural change brought by DevOps involves the whole technology department… the whole enterprise, in fact. Roles change, expectations change. There are questions about how to align processes, governance and support. We need to think about the structure of our teams in a post three-tier world. We need to consider new support methodologies like swarming. We need to thread knowledge management and collaboration through our organizations in innovative new ways.

But the one thing we really must do is to start with the customer.