Sep 29, 2020
Learn From The Pros! DevOps Round-Up
Intro
When we talk about the way systems or software behave, it’s easy to forget that there are actually people behind them. Everything we interact with online is thought of, built, and maintained by other people like you.
There are project managers, software developers, DevOps engineers, designers, and many more important people involved. It helps to get some perspective on issues that the people in these roles face every day when they’re trying to keep things going for users.
In this round-up, we’ll focus on DevOps engineers. Usually, no one goes to them unless there is something wrong with a deploy pipeline or a pipeline needs to be set up. They might be responsible for managing server resources, monitoring up-time, and making sure that all of the parts of an application are getting updated and deployed correctly.
Thankfully I was able to get some insight from a few professionals that work in DevOps to see what it’s like for them. Here are the questions they were asked.
The Questions
- What’s the hardest part of setting up a CI/CD pipeline for you?
- Do you have an approach for debugging problems in your pipelines?
- What’s something you would tell people who are trying to get into DevOps?
The Pros Answer
Now that you know the questions, let’s get to the fun part and see their answers!
@dmarklein
1) What’s the hardest part of setting up a CI/CD pipeline for you?
The edge cases. Regardless of your language/framework, you can surely find a CI platform that makes it crazy easy to build a “Hello World” app in your language of choice. HOWEVER, it is so often the case that if/when you need to add a new step that does something a little bit outside the box, you have to jump through hoops upon hoops to make it happen. This problem gets compounded by the poor documentation that seems to be so common — in many cases, you are destined to fight through trial and error to get your edge case working reliably!
2) Do you have an approach for debugging problems in your pipelines?
Logs, logs, logs. Read the logs. No, really, _read_ the logs. Just about every CI platform worth its salt has great support for logging, and you would be surprised how often the answer to your problem is write there in the logs, if you actually take the time to read them. CI logs are often incredibly dense and repetitive, but they hold a TON of information if you dig in.
If the problem doesn’t immediately pop out in the logs, it’s often helpful to pursue the “minimum reproduction” strategy: take things out and turn things off until you get a successful build, then start adding things back until your problem resurfaces. This is often difficult or time-consuming if you have a lengthy pipeline, but that’s just another reason to keep your pipeline as fast as possible!
3) What’s something you would tell people who are trying to get into DevOps?
Never forget why we’re here: we’re trying to make it fast, easy, and safe for developers to ship changes to production that provide value to the business. Without the business (whatever that may be) we wouldn’t be here.
Oh, and everyone should read the Phoenix Project and the Unicorn Project. :)
@ce0la
1) What’s the hardest part of setting up a CI/CD pipeline for you?
Early on, the hardest part for me was having to run a pipeline containing large numbers of unit and integration tests. The main problem wasn’t setting up the pipeline or fixing idiosyncrasies related to a test suite, it was the back and forth in moving from Local > Git > CI and also the time it took for some of these tests to complete. Regularly, tests in the pipeline break when they are almost done or, even worse, the coverage reports they generate — after the tests run fine — are sometimes rejected by Sonarqube (software for code quality scan). Some of the tests take about 20–30 minutes to run (almost 2 hours one time) and this considerably slowed down my iteration/feedback process and is a major difficulty I experienced while setting up CI pipelines.
P.S: I found a suitable way around this was containerization; anything and everything that can be containerised should be. It saves valuable iteration times and faster feedback while setting up a CI pipeline.
2) Do you have an approach for debugging problems in your pipelines?
As I already mentioned, a top and less frustrating approach for me to solving problems in my pipelines would be containerisation as it helps with uniformity in environment and you can simultaneously test various iterations of it on your local workspace. Also, meticulously comb through your CI error logs; seems like a no-brainer, but I have seen people (myself included) get stumped by not properly reading their error logs — start and/or end of the logs usually contain a definitive summary of what is causing the trouble.
P.S: You can join developer/system administrator/DevOps communities and engage in active troubleshooting sessions with them. The problem you help solve today, might be the 3 days, coffee-filled, zero-sleep, debugging session you avoid tomorrow.
3) What’s something you would tell people who are trying to get into DevOps?
Well, there are a lot of peculiarities to be considered. For starters, it is strongly advised, and I agree, that you have a background as a System Administrator (Ops) or Software Developer (Dev) before venturing into the field.
I entered into DevOps having intermediate knowledge of Linux and a working knowledge of software development (built a couple personal apps/projects). I am generally who you’d call a self-starter and an autodidactic learner, so I was able to pick up quite fast, but not without trouble and quite a lot of bumps.
Getting into any new field is hard, but the difference between an Entry-level Dev and a Senior Dev is more often than not the amount of problems they’ve been through and how quickly they can get through new ones. And what better way is there to solve various types of problems than getting into them?
@HoardingInfo
1) What’s the hardest part of setting up a CI/CD pipeline for you?
Trying to keep it simple. Starting with a quick win that I can build on. Often times I end up with a on-legged Frankenstein setup. The reason is because with every new step in the flow, or added automation, it seems like 5 new things come up that could be automated. This vicious loop often results in making little progress in having a complete and testable pipeline.
2) Do you have an approach for debugging problems in your pipelines?
No not really. I know the desired state, but the approach is unfortunately similar to applications in prod. Wait for something to break, an start the hunt.
3) What’s something you would tell people who are trying to get into DevOps?
Do remember that DevOps is not an end. It’s a journey. If it were an end, then it’s destined to be the next “waterfall” where people have architected themselves into a corner. While DevOps is both a function and a practice. The function is pretty clear, and not the hard part. The hard part is the practice, which needs to be part of the entire organization.
@pollock
1) What’s the hardest part of setting up a CI/CD pipeline for you?
Uncovering what to automate first. Often you’ll hear the phrase ‘automate everything’ but the fact is that you need to map out the process and automate the most difficult and frequently repeated tasks first. Then you should track your pipelines with the right delivery metrics in order to understand their performance. Then you can add “Continuous Improvement” (so to speak) to the list.
2) Do you have an approach for debugging problems in your pipelines?
We have an Op (or workflow automation) in our open source registry that will notify us in Slack when there’s an issue. We also track our cloud-first logs and monitor wherever there is a breakdown via The Ops Platform dashboard and control plane in Slack.
3) What’s something you would tell people who are trying to get into DevOps?
Start simple. We have a Lean DevOps philosophy at CTO.ai. We simplify our tools via workflow automations in order to make them more accessible to the entire team. We focus on reducing complexity and increasing velocity in straight forward, understandable ways. Our thought is that a 10X engineer is actually a 10X development team that makes 5 devs 2X as efficient.
Conclusion
Don’t forget that we all start somewhere and we all ended up somewhere completely different. DevOps is the same. Hopefully seeing these kinds of responses from experts in the field gives you more perspective on what it’s actually like. It takes some time to learn about all of the tools and how they work, but you’ll get it.
Everything has its difficulties and with enough practice, you’ll be able to cruise through them just like these DevOps engineers.
