What is DevOps?
DevOps is definitely something to do with software delivery. The word has stuck. And it has spawned others: BizOps, DataOps, ArchOps, FinOps, ChatOps, SecOps, WinOps etc.
DevOps consultants create DevOps strategies to implement DevOps transformations with DevOps teams and DevOps Engineers who do the DevOps.
What is it?
I have been in a DevOps Engineer position for the past one and a half years and during the course of this I’ve tried to understand what DevOps is meant to be.
Objectives
It is a set of software delivery principles and practises with the objective of producing better business outcomes and better quality of work-life outcomes for those in the field.
It (namely) centres around breaking down barriers to communication and collaboration between Software Development and Operations teams. Those who build software and those who look after it as it runs. Where a developer may consider a feature completed when it had been developed and was running on their machine but there would be great pain in achieving the same outcome in a real environment. Work would be thrown over the wall once complete from one business unit to the other.
Each team has conflicting objectives. Dev wants to introduce new features whilst Ops is resistant to change due to skepticism over quality and long, complicated deployment processes which put the business at risk.
Competing goals and rigid responsibilities create blame and antagonism and produce an uncomfortable working environment. DevOps aims to get both teams on the same page and further to smooth interaction for all internal and external stakeholders. Not just ‘Dev’ and ‘Ops’ but Test, Security (InfoSec) and the business as well. (This just wouldn’t all fit into a single catchy portmanteau.)
No time for heroes
DevOps aims to remove the necessity for heroics. Where teams or individuals need to constantly go above and beyond expectations to solve problems and meet deadlines creating an obvious vicious cycle.
There are quality or deployment problems with software. This means at the very least more work is required. More overtime and a persistent heightened sense of urgency -> less happy people. It will likely mean cutting corners. Problems need to be dealt with under great time pressure which means working at pace under increased stress which increases chance or error. This means manual intervention, risky changes and manual testing. When there is eventual success then tech-debt has piled up. Questions need to be answered and knowledge shared and new work planned to plug gaps. But of course, heroics is the norm. And so this is all de-prioritised because there is another fire to fight. The work naturally falls to those most talented and those who understand the most about the system. These are also the people who will find it easiest to leave if they feel mistreated.
They do leave. The pile of tech-debt is now mountainous. Teams are resentful because they believe other teams are causing their problems. Everything is horrible. The cycle continues.
We want to deliver valuable things continually with the team working at a constant and stable pace that can be continued indefinitely. This means a better day to day existence for IT delivery professionals; less person-dependence in the organisation and less burnout[0] which is a common problem in the technology industry and others.
Origins
The term must have been coined sometime before the first DevOpsDays technical conference in Ghent Belgium, 2009[1]. Now the event is coming to it’s tenth anniversary. Give or take marking ten years of DevOps-ing.
The 2013 novel ‘The Phoenix Project’[2] illustrates DevOps through the incremental IT transformation of a company. The book is modelled on ‘The Goal (1984)’[3], a business change novel set in a machine assembly factory which introduces readers to the management philosophy; the ‘Theory of constraints’[4], explaining the importance of ‘bottlenecks’ and other components which are very closely linked to the ‘three ways of DevOps’ (explained in the next section). Both novels highlight the difficulty of affecting positive change.
Whilst in many ways software production is unique. In many other significant ways it parallels other physical manufacturing processes. Shipping code is not dissimilar to other production lines. Software can benefit from lessons already hard-learned in other industry.
In this vein DevOps also takes influence from Lean practises[5,6] which originate in the automotive industry with Ford and Toyota. The Toyota Production System[7] focusses on eliminating waste everywhere it can find it whilst never making compromise on quality. The core values have been codified as principles in ‘The Toyota Way’[8]. The four sections are as follows:
1: Long-term philosophy.
2: The right process will produce the right results.
3: Add value to the organisation by developing your people.
4: Continuously solving root problems drives organisational learning.
The Three Ways
Gene Kim’s 2012 blog post ‘The Three Ways: The Principles underpinning DevOps’[9] is perhaps the best concise distillation of DevOps. These are the core principles guiding The Phoenix Project and the 2016 DevOps anthology ‘The DevOps handbook’[10] which provides more breadth and depth on the subject.
The first way: Systems Thinking
Moving things left to right.
Focus should always be on the entire value stream. From concept and analysis, to development to running a new feature in production. Micro optimisations within the stream are useless if changes are getting throttled at some other point in this end to end pipe. Lead times are a key metric. When a feature has been built, how long does it take before this is running in production? Where is it getting blocked? Solve for this.
Flow is the rate at which features and new changes are moving through the end to end pipeline. SPF or Single Piece Flow is the ideal. Where a new idea can be built, tested and deployed to production both rapidly and safely so it can deliver value to the business. Small, frequent changes enable better throughput whilst introducing less risk.
The second way: Amplify feedback loops
Feeding things back right to left. Measurement and visibility.
This is important at every stage so we can pinpoint and address areas to improve effectively.
Continuous Integration: Every change committed should be subject to automated build, deployment and test immediately so any negative mutation to the codebase is found and fixed ASAP.
Highly visible automated testing should occur checking different things across the entire pipeline so we can quickly identify where we have unexpected results and focus in on the problem.
Deployments should alert on success or failure. Every successful deployment is cause for celebration and every failure presents an opportunity to quickly learn and improve.
We should have excellent system observability and alerting in all environments. Then we can obtain clarity on production issues as well as ensuring that the tooling is fit for purpose as it is already used on a daily basis to explore and understand none-production defects.
Internal retrospectives should occur after each development cycle so we can understand what has been going well and what needs to change.
All Incidents and Defects should be fully understood through blameless postmortem and this knowledge passed back to the team for growth.
Finally, we should ensure we are obtaining user feedback. Directly and indirectly. We need to know if the features we are releasing are being used and we need to understand exactly what customers really want so we target our efforts in this direction.
The third way: Culture of continual experimentation and learning
This is the final hurdle in culture change. Instead of being fearful of change we need to get to a place where we are making lots of small changes and thus have great confidence in the process. This can embolden us to take risks and explore the art of the possible. Where we can initiate controlled experiments with the confidence that if anything goes wrong we have the resiliency and ability to quickly recover and learn from the outcome.
Along with the emphasis on measurement and feedback above. We are attempting to inject some of the scientific method into software delivery. Hypothesis, experiment, measure, learn, iterate.
Learning should always be one of a companies highest values. If individuals are always learning they will always be improving and changing. This cascades to an ability to innovate and either adapt to or force changes in the market place.
Isn’t this just good software architecture?
So, we want to make various stakeholders happy, this means satisfying both functional and non-functional requirements. Isn’t this just good software architecture? In a sense yes, but…
1: Increasing flow and feedback should be high priority objectives when making key architectural / process decisions and technology choices.
2: Treat others how you would like to be treated. It is not just the intention to do quality work but how it is done that can be the difference. DevOps places focus on culture and stakeholder collaboration structures outside of application and platform architectures. It is paramount that it is both perceived and felt that teams are working together for a common goal.
3: There is an emphasis on the left-shift of activities so that they are worked on early and continually. So it isn’t just acknowledging NFRs are important. But actioning this from day one. The definition of done should extend for each feature to ensure it is has been tested, is running in a production-like environment with security and monitoring and documentation has been written (operational / consumer / guides ).
Is it Agile?
Agile and DevOps are entirely compatible and I see DevOps mostly as an Agile extension. Agile++. The novelty of DevOps seems to be built on an agile foundation.
Both favour communication and interaction, frequent delivery of working software, cross-functional teams, sustainability, quality, simplicity and reflection[11,12]. Both take influence from Lean.
In agile development you respond well to change. The core objective of flow in DevOps enables this. By delivering small increments of things that work you are able to pivot quickly as you are not stuck in a long state where lots of different things are sort of kind of done.
In agile development you focus on customer collaboration — business people working with technology teams consistently. This is taken as a given for DevOps and we just extend / clarify the agile notion to ensure Ops is included in the collaboration.
DevOps is a bit more prescriptive in how we can achieve this end. Automation, infrastructure-as-code, continuous integration, visibility etc. but it’s all hyper-compatible.
Both DevOps and Agile are equally incompatible with all-encompassing, un-adhered-to, complicated JIRA workflows.
ITIL?
IT Service Management is as important as ever. Prominence is given to stakeholder and relationship management and the responsibility is distributed over the entire team not just leadership and management.
Automation is used to ensure quality and efficiency in configuration and change management. Defects and incidents need to be swarmed with a good sense of togetherness, good lines of communication, good telemetry and tools and good post-issue learning sessions which result in new actions and automation avoid similar things happening in the future.
The IT Infrastructure library is constantly updated for relevance and continues to be a valuable and complimentary framework to understand and adhere to[13,14].
Should we do the DevOps?
If you work in software then you will have experienced common problems which keep popping up across the SDLC. DevOps acknowledges these and attempts to address them in a reasonable way painting a positive picture to aspire to. Where problems exist they should be solved for.
Puppet and Splunk publish an annual report on the state of DevOps[15]. This research attempts to quantify DevOps success and highlight the latest lessons learnt in the market place. This is an interesting read and may be more convincing for business leadership.
How to DevOps
Get everyone on the same page: Generally everybody wants to contribute to good work and enjoy the process. This is what DevOps attempts to provide. All of the IT staff of an organisation need to learn about what DevOps is and what it could mean so that they are enthusiastic and open to changes which may need to occur in their daily work rituals.
Extend role definitions: To help smooth pain points teams need to not just understand principally what to do but to have some concrete targets of actions they will take to help improve the end to end delivery pipeline; actions that were not expected of them before.
New roles and teams: To ensure each project team can execute on it’s functional and non functional requirements efficiently without constant reinvention there needs to be elements of a shared platform. Automation, environments, monitoring systems, security and various patterns and support. These platform components need to be managed and improved upon carefully and continuously.
Cross functional teams: project teams should be small and include business people, developers, testers and ops people who can feed into design decisions and help the teams with automation and integration into platform components. These teams have better communication and can then fully own their features and autonomously deliver them.
Implement: Work out within your team how you are going to start to start implementing the 3 ways of DevOps focussing on automation and NFRs continually from day one.
Executive buy in and planning: To enable all of the above there will be a good deal of organisational change both structural and cultural. There is no one size fits all DevOps strategy for organisations. It needs to be carefully planned and incrementally executed. This requires buy in from the top.
I’m a beginner — Where do I start?
So am I :)
Hopefully, start with this post. Understand the gist of things and what is really important. Read some of the material linked in the Resources section below. Read blogs on Software Development and watch various DevOps conference talks on Youtube. Maybe attend one of them yourself! (I haven’t been to one yet but I think it would be cool to do so.)
Think about how you or your team can better collaborate with your immediate stakeholders. Try to take some action even if it’s small. See what happens and be proud of your proactivity.
Value and cultivate your soft skills. Communication is key. Empathy, patience, emotional-resilience and the ability to quickly and gracefully change your mind when it appears you are wrong all come in handy.
Remember that DevOps is not a set of tools or a single role / team. DevOps goals are supported by plenty of tools and patterns that span many functional areas such that what could be a DevOps Engineer is far too broad and causes confusion. I understand that certain tools and technologies come to mind when talking about DevOps and I would like to address this in detail in another post.
Undoubtedly there is a theme of automation which runs through DevOps. Before worrying about what the latest build server can do establish comfortability in the terminal. Basic scripting and SysAdmin material will suffice. Learn to code. Algorithms 101 and some modern high-level language. Try to adhere to good engineering standards — testing, source-control, versioning, clean code[16] etc. Learn the traditional Software Development Lifecycle (SDLC) and move more deeply into areas and sub-areas of IT which interest you as you blaze your automated trail.
Resources
Throughout the post there are several really great references. I’d encourage all of those really interested in DevOps to read them all over time. If you are looking for some DevOps gurus to follow then just check out the authors.
[0] Why we need to talk abut burnout in the tech industry?, Laurence Bradford, Forbes, 19/06/2018.
https://www.forbes.com/sites/laurencebradford/2018/06/19/why-we-need-to-talk-about-burnout-in-the-tech-industry
[1] About DevOpsDays.
https://devopsdays.org/about/
[2] The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, Gene Kim, Kevin Behr, George Spafford, IT Revolution Press, 10/01/2013.
[3] The Goal: A Process of Ongoing Improvement, Eliyahu M. Goldratt, Jeff Cox, North River Press, 01/07/2019.
[4] The theory of consraints.
https://en.wikipedia.org/wiki/Theory_of_constraints
[5] What is Lean?
https://www.lean.org/WhatsLean/
[6] Lean history.
https://www.lean.org/WhatsLean/History.cfm
[7] Toyota Production System.
https://en.wikipedia.org/wiki/Toyota_Production_System
[8] The Toyota Way: 14 Management Principles from the World’s Greatest Manufacturer, Jeffrey Liker, McGraw-Hill, 2013.
[9] The three ways: principles underpinning DevOps, Gene Kim, 22/08/2012 IT Revolution.
https://itrevolution.com/the-three-ways-principles-underpinning-devops/
[10] The Devops Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations, Gene Kim, Jez Humble, Patrick Debois, John Willis, Trade Select, 01/12/2016.
[11] Agile Manifesto: Values, 2001.
https://agilemanifesto.org/
[12] Agile Manifesto: Principles, 2001.
https://agilemanifesto.org/principles.html
[13] What is ITIL?
https://www.axelos.com/best-practice-solutions/itil/what-is-itil
[14] ITIL and DevOps — arch-enemies or complementary models?, Dave Blodgett, Axelos, 17/02/2017.
https://www.axelos.com/news/blogs/february-2017/itil-and-devops-arch-enemies-complementary-models
[15] The Annual State of DevOps Report, Puppet, Splunk.
https://puppet.com/resources/whitepaper/state-of-devops-report
[16] Clean Code: A Handbook of Agile Software Craftsmanship, Robert C. Martin, Prentice Hall, 01/08/2008.
