DevOps explained — Venture Capital perspective
DevOps are getting more and more popular for a reason. They enable a quick, secure and reliable way of developing, testing and shipping new features to the customers, so no wonder that many companies can’t imagine life without them. That’s why we have decided to investigate the topic and prepare a lightweight summary that will help non-technical audiences understand the concept and tools used in the process as well as highlight potential investment opportunities for us — the Venture Capital investors.
DevOps revolution
The DevOps movement gained momentum in 2018, but it had its beginnings around 2009 when John Allspaw and Paul Hammond from Flickr presented how Dev and Ops can cooperate to deliver 10+ deploys per day (deployed code is the code shipped to the customers). According to a survey from 2017 DevOps is becoming the norm. Roughly 2 years ago almost 50% of companies had DevOps implemented to some extent and the next 30% was planning to follow suit.
The best and biggest companies such as Netflix, Amazon, Facebook, CD Projekt, Ikea or Lego, have DevOps engineers in their ranks, but you can also spot dozens of them at startups like Revolut, Docplanner, Booksy, or Brainly.
But what exactly is DevOps?
Developers (Devs) build & ship features as quickly as possible while the IT Operations (Ops) deploy code from Devs and make sure systems never go down and fix them if they do so. Before DevOps, there was a conflict of interest because Ops were protecting the stability of systems, and they didn’t welcome changes made by Developers with open arms as they were constantly bringing instability.
After DevOps was introduced the conflict still exists, but now the relationship between these two parties is much more productive. But going back to the main question: the ultimate goal of DevOps is to build digital pipelines taking code from the Developer and shipping it to customers in an automated, reliable and secure way. DevOps is what you get when you take Devs and make them do Ops. They repeatedly automate themselves out of their (current) job by building tools that replace previously manual processes. It wasn’t possible for Ops, who usually don’t have enough coding experience.
Brief story of DevOps
Long time ago IT companies used to have servers in their basements and System Administrators (IT Operations) had to configure them by hand. At the beginning it was manageable as you had 1 or 2 servers only, but over time you needed more and more of them and you started to experience troubles. Servers had different configurations and it was causing unexpected errors that were difficult to resolve as you had to check manually each of the servers to find out how they were configured. And time was money — each hour of outage can be measured in thousands of USD lost.
At some point tools used in the Development department were slowly introduced to the Operations e.g. Version Control Systems. After some time it was possible to define the target state of the infrastructure as a code (IaaC) and from now on Ops were aware of what’s going on at any time, what were the changes, and who did them. It made scaling of your infrastructure much easier, because thanks to IaaC you can simply copy-paste the servers’ configurations instead of setting up everything manually step by step. It also enabled testing of the infrastructure configuration before it was deployed, which decreased downtime. Less manual work resulting from automation means that the deployments can be more frequent, so the code can be tested in smaller batches, the bugs can be found quicker, and the features to the customers can be shipped faster. In short, we can say that DevOps is the application of Dev methodologies to Ops work, and it’s getting more and more common.
The time it takes to go from code submitted by a developer as a ready to the moment when customers see it in their product (deploy lead time) is probably the best metric to assess a company’s maturity in terms of DevOps adoption and the top companies are able to do it in minutes, while it still takes months for laggards.
Shorter deploy lead time allows you to amuse your customers with new features faster than your competition. By looking at the table above you can also see how fast tech is moving forward: in 2011 Amazon did more than 7,000 deployments daily while in 2017 the number surpassed 23,000/day (don’t forget that in 2009 there was a conference talk on how to do 10 daily!).
The DevOps movement manifesto
According to Google, the DevOps manifesto can be summarized in 5 points:
1. Reduce organization silos — It’s about merging your Dev & Ops teams — sometimes it’s as easy as putting them in one room. These two teams should share ownership and use the same tools.
2. Accept failure as normal — It’s almost impossible to develop without any bugs at all, and the only way to deal with them is to have a plan of how to resolve the problems e.g. have a script to roll back deployment when it fails.
3. Implement gradual change — Long time ago some companies used to release large, annual software updates which caused total chaos and disruption. If you deploy 100k lines of code at once finding all bugs would be a miracle. It’s much safer to do it more often in small chunks — preferably in CI/CD methodology.
4. Leverage tooling & automation — Ops had a lot of manual work that doesn’t scale — think about installing packages, monitoring, log analysis and so on. In the DevOps world, it has to be automated to increase the velocity of deployments as well as stability & reliability — people are bad at dull, repeatable tasks and often make mistakes.
5. Measure everything — You always have to have numbers to prove that what you are doing makes sense and it’s worth to continue living the DevOps way. You can do it by introducing measurement, monitoring (e.g. how much CPU you are using), and alerting tools which can tell you or alert you if any of the crucial parts of your services go down. Alerting should be very selective and focused on symptoms that are painful for end-users.
DevOps Stack
Let’s have a look at the DevOps pipeline, most popular tools, and their pros and cons.
Continuous Integration & Continuous Delivery/Deployment
Integration can be understood as merging code from different Developers into one. More frequent integrations mean that the merge conflicts are easier to resolve, so there is a preference for doing it several times a day — almost continuously. Before the code is integrated into the mutual repository, it should be built (building is converting source code into working software) and tested.
Continuous Integration (CI) tools are often combined with CD tools which can stand either for Continuous Delivery or Deployment. Continuous delivery is an extension of Continuous Integration and means that the code is not only built and tested but also ready to deploy with one click of a button. Continuous Deployment also automates this last click and automatically ships the code to your customers.
CI/CD tools usually don’t bring much value on their own, but they serve as a central place to which other parts of the pipeline are connected to. Jenkins is probably the most common tool in this space, but as it was created almost a decade ago, there are some opinions that it’s complex to set up and maintain. It has tons of plugins and integrations which, unfortunately, happen to break from the time to time, so it’s worth giving a try to newcomers like Buddy, or Buildkite, which recently raised $20m Series A.
1. Version Control — when you are programming alone it’s not a problem to store code in a local directory and make copies of it as you go. However, startups and enterprises tend to have dev teams consisting of hundreds or thousands of employees. In such a setup you need to have a tool helping you to avoid code merge conflicts and to recover previous versions of code. This category of tools is called Version Control Systems (VCS) or Source Control Management (SCM). The king of the category is GitHub, much more user friendly than clunky solutions from the past like SVN.
2. Build Automation — tools in this category automate the compilation process (compilation = building = turning your code into a working software), and you usually need to find a tool specific for the programming language you use in the project. There are a lot of open-source options available. Depending on the type of projects, Maven has 35% to 60% market share.
3. Artifact Repository — the definitions of artifacts are quite fuzzy but in general they are one of the software development by-products e.g. project source code, dependencies or diagrams explaining how the software should work. Artifacts can vary between versions of software so you need to have a way of managing them. As JFrog has one of the leading Artifact Repository Managers (ARM), it has prepared whitepaper with 10 reasons to use it.
4. Testing — As DevOps make development more rapid, security is getting more and more important. Tests can (and should) be run at many points of the pipeline, not only after the building process. Recently this area received a lot of attention, so we will dive deep into it in the next blogpost.
5. Infrastructure Provisioning (IP) — Tools for launching your infrastructure. In the past this part of the process was lengthy — you had to order hardware (servers), wait for it to arrive, then set up everything, connect via wires and so on. With the cloud, it’s much easier, and tools like Terraform enable you to manage your infrastructure as a code (IaaC). Spacelift, on top of it, enables collaboration between DevOps, introduces more security, versioning and understanding of your costs.
According to DevOps 2019 survey from JetBrains, 65% of the respondents didn’t use any IP tool, which might be partly explained by the fact that some of the Configuration Management tools have IP features, but using them for this purpose has some cons e.g. lower control of your infrastructure.
6. Configuration Management (CM) — is another part of IaaC. It lets you maintain IT systems, servers, or software that you have on your infrastructure in a desired state by managing settings, patches or updates. Interestingly 27% of DevOps teams has a custom CM solution. The leading commercial tool with a 23% share is Ansible, which is especially recommended for small infrastructure. Although CM tools have some IP features as already mentioned, and some IP tools have CM features, the best practice is to have separate IP tool like Terraform and CM tool like Ansible.
7. Cloud — for scaling up you need to choose your cloud provider. The most popular ones are Azure, AWS, and Google Cloud Platform.
8. Containers — in the past there were Virtual Machines with separate operating systems (OS) on the servers. Now, containers are taking their place. Instead of dividing infrastructure into blocks with their separate OS like Virtual Machines, containers are dividing the infrastructure at the level above the OS. They are much faster and more lightweight as a result. In 2018 Docker had 83% of the market (99% in 2017). Containers require orchestration tools to automate their deployment, management and scaling.
The future of DevOps
What are the most important trends that emerge or might emerge in the DevOps space?
- Safety first — Implementation of Security in the DevOps pipeline is getting more and more popular. It means not only changing the name from DevOps to DevSecOps but also change in mentality and implementation of many solutions. This trend is so broad, that I decided to prepare a separate blogpost about it.
2. Containers everywhere — Containers are more efficient and cheaper than Virtual Machines, so no wonder that their adoption is increasing and they become a standard.
3. Keep it simple — the DevOps pipelines vary from organization to organization and sometimes even inside a single organization, but they have one thing in common — usually they are very complex and are based on many different solutions and plugins. Maybe in the future we will see end-to-end tools aggregating higher chunks of the pipeline and removing boilerplate work needed for setup and maintenance. The interesting approach to this trend is Lean DevOps and SlackOps propagated by Tristan Pollock from cto.ai.
Recommended reading:
I hope that this blogpost increased your understanding of the DevOps space. If you have noticed some security gaps in the pipeline — I did it intentionally. In the next analysis, we are going to focus solely on DevSecOps. If you have any comments, see any non-security gaps in the pipeline or if you know (or are building) better solutions for DevOps that I haven’t mentioned above, make sure to reach out to me on LinkedIn or comment below.
Inovo Venture Partners backs early-stage, post-traction startups that can grow 100x. We partner up with ambitious founders like Stefan from Booksy, Maja from Zowie, or Marcin from Spacelift. We invest between €0.5–2m in startups from Poland and the region.
