Does investing in DevOps pay off?
--
fastforward has invested heavily in setting up a top-notch DevOps infrastructure. Our software is built, tested and deployed in a well defined and highly automated process. This has cost us a truck load of money. So why did we do it?
What is “DevOps”?
Developers write code, which is then tested and finally put into production by the operations team. At this point, the system needs to be tested again as the development environment usually differs a bit from the production environment.
Since testing is a big effort, the number of updates is reduced, which leads to less flexibility in the software and a bigger time to market for new features. And your competitors will be happy about that!
This is where DevOps comes into play. It is the combination of development and operations and consists of all the practices and tools which help to produce, test, and release software at a high pace and with the least amount of effort possible.
Processes covered by the fastforward DevOps
With our agile and well trained team, there was no real effort needed to set up the development culture and practices needed. However, we had to invest heavily in tools to set up a highly automated DevOps infrastructure covering the whole process from writing code to monitoring the running systems:
Source control system
At fastforward, we store our code in a “Git” repository. It is integrated with our project management and our build system. This, for instance, allows us to create new “branches” (versions) of the software for each ticket with the click of a button. This way, multiple team members can work on the same project without interfering in each others’ work.
Automated building…
When a developer adds new code to a branch in the source control system, it automatically triggers the build process. All code is compiled, the dependencies are fetched, added and more. The result of the process is communicated to the team via automated message, to allow it to react quickly, in case the change broke the build.
…testing…
In order to be able to release new versions quickly and with confidence, the software needs to get checked by automated tests. This way we can ensure that changes made to the code in one area will not cause problems in other areas. This is absolutely crucial and the basis of every good DevOps process.
We use three kinds of tests:
- Unit tests check the functions of small “units” of the software. These tests are usually quite small, run quickly and check one single function of the code.
- Integration tests check the collaboration of multiple parts of the software and can even assess if the system still works with other systems, e.g. via a network etc. These tests usually take considerably longer to run and often depend on a certain state of the environment they run in (e.g. availability of a database or a network). However, if two systems should work together, exchanging data, etc., this type of test will also tell us, if the other system was updated and broke the interface — maybe a system which is not under our control.
- End-to-end tests, finally, test the user interface. In our case, this happens through a script which controls a web browser, telling it to open a URL, click a link, enter some data in a form and so on. End-to-end tests are usually quite slow, so we don’t have them automatically executed upon every little code change. However, when a developer is finishing up her task, it makes sense to trigger these tests before her work is merged into the system. Also when creating a new release, these tests should be run.
…and deploying
Once the code is checked in, built, and tested, it is time to deploy it to a test server, aka a test environment. After all, it is nice to have automated tests checking the code, but the stakeholders need to play with the new features themselves to see if they meet their expectations. Maybe the requirement was not precise enough, maybe something was forgotten or misinterpreted, or — quite often — the scenarios of the automated tests do not yet cover all features.
In most DevOps infrastructures you will find multiple environments, e.g. “DEV” as a playground for the developers, “TEST” to verify the new features, “INT” as an environment which should be a twin of the “PROD” environment for final “real life” tests and maybe also performance tests. And finally “PROD”, the production environment, where the users are. But what if developer A would like to test his new feature, but DEV occupied by developer B, who is not done checking her update?
Of course one could add a few more environments and hope for the best. At fastforward, we opted for “ad hoc” environments to solve the problem: After a code change successfully completes the automated build + test pipeline, it is deployed on an environment which is exclusively created — of course again automatically — for this build. It can then be used for reviews and — you guessed it — is afterwards automatically discarded. This gives us a lot of flexibility to work on code, test it, and present the updates to the stakeholders.
How much did it cost?
DevOps is not only an infrastructure, but also a mindset. As for the mindset (high test coverage, small but frequent releases, quick feedback cycles), this should not cost anything, as it should be part of every good development team’s evolutionary process. But as you can imagine, setting up and running such an infrastructure is quite an effort. Our DevOps team continuously expanded and improved the system over the span of about a year, and small tweaks are still required on an almost daily basis in order for the system to keep up with the ever changing requirements and technologies. Therefore it is difficult to put a precise price tag on it, but the investment is definitely north of $100k.
What do we gain from it?
Automated installations give us stability. If the installation was successful on INT, it will also be successful on PROD. On top, the installation is much quicker than it was before. Nobody has to set the alarm clock for Sunday at 2am in the morning to roll out a new version. Also, the developers can roll out their code themselves on a test environment and thus get quick feedback from stakeholders without having to bother operations about it.
Automated tests give us peace of mind. With good test scenarios, we know that our update does not break other features. And if it does cause problems, the tests will tell us right away what is going wrong and where. This not only greatly reduces the need for manual testing but also allows us to release updates at will. Anytime. Instantly. And if something should still break, then a new test is added to cover exactly that problem. Once the test is successful, the next release is ready to be shipped.
Automated builds give us instant first feedback about our code. Maybe the developer does have the required library on his machine, but he forgot to add it to the build script. Thanks to the automated build system, the team will know about it instantly. This way, other developers will not have to spend a lot of time when they start working on new tasks based on a broken system.
Conclusion
All this automation combined with the matching mindset and development culture of the team gives us a lot of flexibility to react quickly to new requirements. We can create a new version with the click of a button, implement a new feature and present it to the stakeholder immediately. At the same time, all of this is available with greatly improved stability while having reduced manual testing efforts. It is by no means a guarantee for flawless software, but it is the basis for quality in complex systems. All of this leads to a much better time to market for new features.
So at the end of the day, comparing the cost of DevOps with the opportunity costs of not having such a system in place, it is quite obvious that there is a lot of value in good DevOps. Our customers of course benefit greatly from our high investments in this area: They get the full features at a fraction of the price, since we can split the cost between all of our customers. And our developers get a better night’s rest, because there are a lot less unpleasant surprises. I would call this a win-win-situation.
