The future of DevOps — Assembly Lines
Being in a world of DevOps ain’t easy. It is not so different than in other parts of the mysterious “IT”. Everything around us evolves — so in order to keep up — we have to evolve as well. In many articles, posts and discussions I can see one term rising above the others — the Assembly Lines. What is that? How does it work? Bear with me for few minutes and I’ll do my best to show you where we’re going.
The past
Not so long ago our main focus area was mostly Continuous Integration (continuously checking whether changes to the baseline didn’t break anything), Continuous Delivery (continuously delivering new pieces of software or infrastructure to our clients) and all kinds of build/test/release automations…
The next big step was when the whole concept of pipelines emerged, it simply reorganized previous approach to more modular one. In “classic” CI approach — sets of instructions for building, testing, packaging and deploying our applications to a single environment were executed one-by-one in serial mode.
If you wanted to get your app tests results or build warnings results you had to wait until the whole process ended and results would be published.
The present
Pipelines gave us a lot more flexibility and control over our processes by simplifying concurrency, adding lots of useful variables out of the box just waiting to be used. In average and classic approach to a CI job — if single process for java mvc app was spinning for five to ten minutes — deployment for all environments (dev, sit, uat, pre-prod, prod) could take an hour or more to complete..
They also gave us a way to see results faster, in more atomic and simpler format. By splitting multiple test runs to multiple stages that will be executed parallely we can see only information that are interesting for us and get all results faster!
The whole concept of “pipes” is beneficial because you don’t need to write 3 jobs for complete CI/CD process (build, ship/package, deploy) and figure out how to trigger one job from another (creating monstrous trees of dependencies inside your platform). Now you can create a single job that will cover all of these stages at once with very smart flow control!
Such flow gives us also more control because we might decide that if one of the test stages is not critical — we may push the software down the pipe in order to fulfill a deadline or to integrate our app with our ecosystem. Been there, done that. Who hasn’t, cast the first stone.
The future (?)
DevOps has adopted many prefixes over the years… BizDevOps (business and analytics), SecDevOps (security wants to be here as well) and many others.
I think that it’s completely normal that we want to do “shift letf” on everything that may save our money in the further stages of project lifecycle because finding any issue at an early stage of your project will cost 10 times less than finding the same mistake on production.
We woke up one day and realized that concerning only about CI / CD processes is over. We have many, many more concerns these days. We all know these:
- configuration management
- infrastructure as a code
- monitoring
- networking
- storage maintenance
- deployments
- releases
- log management
- …
Assembly lines are trying to organize day to day tasks from all teams that collaborate together for the glory of the Project. Imagine project setup that consists of:
- Developers writting new features
- Security doing audits and patching
- Infrastructure creating and maintaining infrastructure
- Build team creating new CI pipelines
- QA developing integration test cases
- Release to production creating pipelines and packages for uat/pre-prod/prod deployment
- …
You see that each of these teams produce something that should be integrated into one, complete chain (or a pipe to be precise). CI is only one of the links in the chain. Each of these tasks must be integrated and connected with others to push the “artifacts” throughout all activities. (compare example below with modular approach above)
Each color on the diagram represents one “team”. Blue means “CI”, i think the difference is self explanatory.
Some companies decide to invest money and time to create bulletproof assembly line that will last for long period of time (especially when their tech stack is pretty much constant). Some will do the duct-tape solution — for now only, for testing — that will probably last as long as the good one but will be tough to maintain.
I’m “working is better than perfect” kinda guy, with strong accent on “continuously fixing my crap” ;-) ( to the stage i’m not embarassed to ask my colleagues to use it…)
In my opinion, the key features of a good assembly line are:
- maintainability
- reusability
- ease of integration (with IaaS / PaaS for spinning new vms/networks/clusters/etc, VCS — for clonning and tagging and pushing the code, storage — for creating persistent volumes, repositories — for storing and promoting binaries, runtimes — java, python, nodejs, CI systems, etc.)
- auditing and monitoring
- visibility into all stages with logs access and status visibility + strong flow control!
- metrics for performance tuning
- security and access management
Assembly lines are here to help us organize and integrate various activities in consistent, scalable workflow. This can be achieved by leveraging CI/CD platforms (like Jenkins or Azure DevOps) and implementing generic and reusable frameworks for pipelines, integrating various tools that may help in the process, eg.:
- ansible for configuration management and security patching,
- terraform for infrastructure management,
- vault for secrets management,
- owasp for security scan on a build level,
- docker & kubernetes for build and deployment management.
The old days are gone, and we must focus on hollistic approach — how to help deliver value to our customers faster and in more reliable and scalable way.
Originally published at http://blog.mylocaldevstack.pl on April 15, 2019.
