DevOps isn’t about automation!

In this blog post we (Welder Luz, Gustavo Pinto, and Rodrigo Bonifácio) reflected on our JSS 2019 paper of the year.

Using a mixed-method approach, we investigate what practitioners believe as a successful DevOps adoption, what are the common steps that these practitioners followed when adopting DevOps and that other may also follow in order to improve their odds in successfully adopting DevOps, and whether these steps do make any sense when applied in practice.

Read on if you want to know what DevOps is about. For a detailed reading, please refer to the original paper.

DevOps is a set of practices and cultural values that aims to reduce the barriers between development (the “dev”) and operations (the “ops”) teams.

The ideas behind DevOps are not new. Even before the existence of the term, companies like Flickr had already pointed out the need to break the existing separation between the operations and software development teams.

Since then, the term gained strengths and interests from companies that perceived the benefits of applying agile practices in operation tasks. The expected benefits of DevOps include 1) increased organizational IT performance and productivity, 2) cost reduction in software life-cycle, 3) improvement in operational efficacy and efficiency, 4) better quality of software products, and 5) greater business alignment between development and operations teams.

Unfortunately, there is a lack of guidance for companies interested in adopting DevOps. As a consequence, many software companies interested in adopting DevOps may struggle to get the basics right, potentially delaying or even compromising the adoption. The lack of detailed guidance to support newcomers interested in adopting DevOps also poses many questions, for instance: 1) Is there any recommended path to adopt DevOps? 2) Since DevOps comprehends multiple elements, do these elements have the same relevance, when adopting DevOps? 3) What is the role played by elements such as measurement, sharing, and automation in a DevOps adoption? To answer these questions, we need a holistic understanding of the paths followed in successful DevOps adoptions.

To gain this holistic understanding, we started by interviewing 15 practitioners that succeed in adopting DevOps. After analyzing and curating their views and reasons for adopting DevOps, we designed a model for helping other software companies interested in adopting DevOps. Finally, we also employed this model at a governmental institution, here after called ABC, which was bogged down trying to adopt DevOps, although repeating the same non-DevOps problems.

In the rest of this blog post we distill our paper.

What we did?

We conducted online semi-structured interviews with 15 practitioners of companies from Brazil, Ireland, Portugal, Spain, and United States that contributed to DevOps adoption processes in their companies. We started with five open-ended questions: (1) What motivated the adoption of DevOps? (2)What does DevOps adoption mean in the context of your company? (3) How was DevOps adopted in your company? (4) What were the results of adopting DevOps? (5) What were the main difficulties?

We used a classical Grounded Theory (GT) approach to analyze the interviews. We actually started from an area of interest: successful DevOps adoption in industry. Following a GT guideline, we employed three investigation steps: Open Coding Data Collection, Selective Coding Data Analysis, and Theoretical Coding. Throughout the process, we wrote memos capturing thoughts and analytic processes; the memos support the emerging concepts, categories, and their relationships.

Data collection and analysis were iterative so the collected data helped to guide future interviews. The first moment of the analysis, called open coding in GT, starts immediately after the transcription of the first interview. Open coding lasted until there was no doubt about the core category of the study.

The first core category candidate was automation, but we soon realized that this category did not explain most of the behaviors or events in data. The sense of shared responsibilities in solving problems, and the notion of product thinking are examples of events that could not be naturally explained in terms of automation. We then started to understand that collaborative culture also appeared recurrently in the analysis and with more potential to explain the remaining events. Thus, we asked the respondents explicitly about the role of automation and how the collaborative culture is formed in a DevOps adoption process.

Taking into account the previous analyses and the respective memos written during all the process, after the tenth interview, we concluded that collaborative culture was the core category regarding how DevOps was successfully adopted. Following three more interviews and respective analysis, we realized that the new data added less and less content to the emerging theory. That is, the explanation around how the collaborative culture category is developed showed signs of saturation.

After understanding the practices that support a successful DevOps adoption, we curated a model that represents it, and we also applied this model in the context of ABC, a governmental institution which was struggling to achieve the benefits of DevOps. We assess the use of our model at ABC by observing the behavior of the team (the first author works as a software developer at ABC) and, to reduce the bias of our perception about the scenario of adopting DevOps at ABC, we also conducted a Focus Group with four professionals of the company for an empirical evaluation of their perception about DevOps adoption.

What we found?

What are the characteristics of a successful DevOps adoption?

The collaborative culture is the core category for DevOps adoption. A collaborative culture essentially aims to remove the silos between development and operations teams and activities. As a result, operations tasks — like deployment, infrastructure provisioning management, and monitoring — should be considered as regular, day-to-day, development activities. This leads to the first concept related to this core category: operations tasks should be performed by the development teams in a seamless way.

“A very important step was to bring the deployment into day-to-day development, no waiting anymore for a specific day of the week or month. We wanted to do deployment all the time. Even if in a first moment it were not in production, a staging environment was enough. […] Of course, to carry out the deployment continuously, we had to provide all the necessary infrastructure at the same pace”

As a result of constructing a collaborative culture, the development team no longer needs to halt its work waiting for the creation of one application server, or for the execution of some database script, or for the publication of a new version of the software in a staging environment. Everyone needs to know the way this is done and, with the collaboration of the operations team, this can be performed in a regular basis. If any task can be performed by the development team and there is trust between the teams, this task is incorporated into the development process in a natural way, manifesting the second concept related to collaborative culture category: software development empowerment.

“It was not feasible to have so many developers generating artifacts and stopping their work to wait for another completely separate team to publish it. Or needing a test environment and having to wait for the operations team to provide it only when possible.These activities have to be available to quickly serve the development team. With DevOps we supply the need for freedom and have more power to execute some tasks that are intrinsically linked to their work.”

A collaborative culture requires product thinking, in substitution to operations or development thinking. The development team has to understand that the software is a product that does not end after “pushing” the code to a project’s repository and the operations team has to understand that its processes do not start when an artifact is received for publication. Product thinking is the third concept related to our core category.

“We wanted to hire people who could have a product vision. People who could see the problem and think of the best solution to it, not only thinking of a software solution,but also the moment when that application will be published. We also brought together developers to reinforce that everyone has to think of the product and not only in their code or in their infrastructure.”

We also found other concepts such as straightforward communication, blameless, and shared responsibilities.

Another important set of categories is regarding DevOps enablers, that is, a set of concepts and practices that support the adoption of DevOps. There are two main categories here: automation and sharing and transparency.

Automation is important because manual proceedings are considered strong candidates to propitiate the formation of a silo, hindering the construction of a collaborative culture. If a task is manual, a single person or team will be responsible to execute it. Moreover, automation increases the confidence between teams, which is an important aspect of the collaborative culture.

“Before we adopted DevOps, there was a lot of manual work. For example, if you needed to create a database schema, it was a manual process; if you needed to create a database server, it was a manual process; if you needed to create additional EC2 (Amazon Elastic Compute Cloud) instances, such a process was also manual. This manual work was time consuming and often caused errors and rework.”

Transparency and Sharing represents the group of concepts whose essence is to help disseminate information and ideas among all. Training, tech talks, committees lectures, and round tables are examples of these events. Creating channels by using communication tools is another recurrent topic related to sharing along the processes of DevOps adoption. Sharing concepts contributes with the collaborative culture. For example, all team members gain best insight about the entire software production process, with a solid understanding of shared responsibilities. A shared vocabulary also emerged from sharing and this facilitates communication.

“Nowadays . . . everyone knows what everyone else is working on. That’s why we have some structured actions. For instance, if one person wants to share some content relevant to the rest of the team, she is completely free to book a room and carry out her own tech talk. . . . Considering this investment in removing silos, the guy knows that every procedure should be transparent and shared to anyone that is interested on that. This creates a positive cycle that increases the sense of collaboration and makes one to share more details about what she does”

We also found categories that correspond to the expected consequences of the adoption of DevOps practices, including agility and resilience, as well as categories that work as both enablers and outcomes. However, we will skip then for now, otherwise this blog post would become really lengthy 😅. What is important to have in mind so far is: To successfully conduct an effort for DevOps adoption, practitioners suggest the focus on building a collaborative culture (the essence of DevOps), which should be achieved through Automation and Transparency and Knowledge Sharing. Without setting up collaborative culture as the main goal, the chances of failing to achieve the expected benefits of adopting DevOps (e.g., Agility and Resilience) increase.

What is the receipt for a successful DevOps adoption?

the most important categories used when adopting DevOps

Figure above represents our understanding for DevOps adoption. It considers the following steps.

1. In the first step, a company should disseminate the relevance of building a collaborative culture between development and operations teams, as the main goal of a DevOps effort.
2. In the second step, a company should select and develop the most suitable enablers according to its context. The enablers are means commonly used to develop the collaborative culture and its concepts.
3. In the third step, a company should check the outcomes of the DevOps adoption in order to verify the alignment with industrial practices and to explore them according to the company’s need.

Our model assumes that following the patterns identified in companies that were well succeeded in DevOps adoption can be a good way to reduce the risks of failure during the process.

Does this receipt for a successful DevOps adoption really work?

Our proposed model has been applied to guide the DevOps adoption at ABC where the first author of this study works as a software developer. More details about ABC could be found in our paper.

In ABC, communication between teams occurred basically through the use of a corporate service-desk tool. The SLAs mostly ensured deadlines for the operations team to perform the procedures that the development teams requested through the service-desk. The procedures include provisioning of infrastructure and database servers for development,testing, staging, and production environments, requesting access to server logs, and the most controversial of all activities: deployment of new versions of applications. The deployment involved a long chain of a week-long process, whose complex chaining of responsibilities oftenran out of schedule, causing months of delays in software deliveries. In addition to delay the software delivering, the silo structure caused a devastating blaming g ame between the teams. If there were delays in the delivery process, or errors in some functionality, the focus was not on solving the problem, but rather on pointing out which team was responsible for the problem.

To start disseminating our model at ABC, we conducted a series of lectures to explain both the model itself and the way it was formulated. After knowing the model, the professionals began to understand that if ABC wants to succeed in DevOps adoption, tooling and automation would not be enough. We consider that the teams changed their focus to build a collaborative culture. This change was only possible due to two aspects: (1) the engagement and sponsorship of the IT managers and (2) the explanations about the process of constructing the model: although ABC is a governmental institution, its IT professionals recognize the importance of staying in line with industry practice to avoid an already before faced weight of dealing with completely legacy software.

Looking to the concepts within the collaborative culture category, the first practical action at ABC was to facilitate communication between teams. The use of tickets and SLAs were then abolished in most of the scenarios. In its turn, the concepts of software development empowerment and shared responsibilities have found some resistance from some professionals. The strategy to mitigate this cultural resistance involved the use of enablers and awareness about: (1) fortifying the collaboration will bring benefits to all involved and especially to ABC and (2) following a model built upon well succeeded experiences is a low-risk strategy for ABC to explore the benefits of DevOps. Finally, we consider that the efforts to disseminate and promote the collaborative culture are continuous, but the kick-off was given and it is possible to notice that the idea has been widely accepted and applied by most of the involved professionals.

In our evaluation, the ABC change of focus from tooling to collaboration was decisive to increase the confidence between teams, reduce the blame game and, consequently, to enable the company to sustainably exploit the benefits of adopting DevOps. In addition, our model has served as a roadmap that allows the teams to focus on the collaborative culture. So,the outcomes already present in ABC DevOps Adoption are compatible with those foreseen in the third step of our model.

To cross-validate our perceptions, we conduct a focus group with four professionals at ABC, two of each development and operations team. The focus group lasted approximately three hours.

The first action indicated and discussed in the group was the provision of VMs for the installation of tools that are related to the development work. This action was exemplified by the successful installations of the Elasticsearch and Kafka tools. The previously existing problem was that when a developer had a need of tools like these, he/she would have to open a request for the operations team to provide it — with very long deadlines that often make the use of the most adequate solution unfeasible. With VM provisioning and cooperation between the two teams, these tools became available quickly for use and the responsibility for its management is joint. This is a clear example of application of the concepts of software development empowerment and shared responsibility of the core category, collaborative culture.

Next we discussed about the reduction of bureaucracy in communication between the teams. It was pointed out that although there is still much room for progress, this can already be considered as one of the advances of the ABC related to DevOps adoption.During the discussion, the ceremonious communication process in the scope of the deployment SLA was recalled. There is a current guideline to avoid using service-desk for simple problem solving. The problems had to be solved in a collaborative way, preferably face to face and the use of the Slack tool has been institutionalized and facilitated the contact between the two teams. We highlight the concept straightforward communication of the collaborative culture category as part of this point of discussion.

Then, the use of multi-disciplinary pipelines in the most recent applications of ABC was pointed out and debated. These pipelines involve everything from the build, through automated tests and static analysis of source code, execution of containers using Kubernetes and publication isonomically in the different environments (development, staging and production). Jenkins is used as a tool to describe and execute the pipelines. One single trigger executes a set of steps that previously required several comings and goings between the teams and a long time to complete. The group agreed that the construction of multidisciplinary and collaboratively produced and maintained pipelines like these ones is only possible when the collaborative culture is fostered.

Overall all participants of the focus group agreed that the proposed model has great utility in DevOps adoption at ABC. They remembered that most of the actions discussed in the previous topic were direct result of the model development and, therefore, its application is already being effective and producing results in expanding DevOps usage throughout the development of ABC enterprise applications.

In summary, in our JSS 2019 paper of the year, grounded in data collected from successfully DevOps adoption experiences,we present a theory on DevOps adoption, a model of how to adopt DevOps according to this theory, and a case of applying it in practice.

As one could see, behind this paper there is a lot of dedication and effort. In this case, most of the dedication and effort made behind the scenes was superbly done by our first author, Welder Luz. If you happen to find this paper interesting and want to congratulate us, please congratulate him!