My Journey To DevOps
If you are like me, it was very hard, but meaningful growth is hard.
To understand why it was hard for me, you should understand the preamble. I’m skipping past 15 years in IT as a systems administrator for places like Lockheed Martin and the Federal Government. I held a dozen certifications, a degree and a ton of diverse experience. Keep in mind it was all manual administration and some scripting. I was good at my job, nothing amazing, but I was certainly a strong traditional ops guy.
That being said …
I moved into a job where I was administering over 500 VMs. Awesome. But wait, I was volunteered to be part of a huge AWS buildout in the cloud. I forget that VMWare farm and those EMC SANs and all that other data center nonsense. Everything was about the cloud. I knew so little about it, but it came quickly apparent to me that it was wild and new and all the rules of engagement were very different.
Like, the seasoned admin I was, I began building on the cloud like I was building a data center, but that was the wrong way. My misguided efforts didn’t last long because a few months into the project, our PM said:
“We are bringing in DevOps.”
What the **** is DevOps.
I totally scoffed. It’s just a buzzword and I just have to weather it and get past this project.
We had several consultants working “sorta” on our team. One guy they hired was the Chef guy, Craig. For the most part, I had no idea what he did. I made the mistake of asking him one day and he explained he was building a Gem Server, Berkshelf, Jenkins, Gitlab, and the Chef server … all on linux. I was traditionally a Windows admin, so after my head melted, I just backed away, literally.
Craig tried to show me how to add a user in a Chef cookbook the following week. I was so frustrated seeing all this Ruby and trying to understand why we were going to build a test before we wrote code to create the user. WTF, a test? The whole time I kept thinking, I could make this user in 10 seconds in the GUI, or make a script to do it. As Craig was explaining how we could check code into a repo, my mind just shut down.
All that session I was thinking the following things (in no particular order):
This has nothing to do with Windows.
I’m not a programmer.
This is too hard for me.
This is irrelevant to my job.
It was all fear, real fear. I told myself those things because I didn’t want to admit that I didn’t understand the value and disregarded these skills as just being outside of my capabilities and career. At the time, I was completely right.
A month or so later, Justin came in. He was the new unicorn DevOps rockstar (IT is just full of cool slang). Justin knew AWS, Chef, all of it. His seat was right next to mine. He opened his MacBook (of course he wouldn’t be on a PC) and after an hour, he was pulling in sprint tickets, writing tests and working on Chef cookbooks. I was blown away. I peeked at what he was working on a few times that day. He was jumping between terminal screens, committing code, simulating builds in test-kitchen. This is all on day f**king one.
We had a demo later that day. Craig was going to show us what he was working on in Chef. His task was to automate one of our loan processing applications end-to-end (I figured this would be just like a script).
Then I watched the magic happen. He pushed a button on a Jenkins job, code for the automation was checkout of git, an AWS EC2 Instance was launched and test-kitchen ran through RSpec and Serverspec. Now, for those who aren’t tracking what all this means, I watched a vanilla server launched in the cloud, dozens of tests were run to validate the automation code, then acceptance tests were executed to actually configure the box and validate that it was, in fact, configured correctly. It didn’t stop there, the Chef cookbook launched an Elastic Load Balancer, created a certificate and attached it, and placed the server behind the load balancer. The cookbook did everything.
I was watching this all unfold in amazement. After all the validation was finished, the server was destroyed and the code was “ready” for promotion in Chef. Now, when a new loan processing server was launched, it would be automatically configured and ready to work — no scripts or manual intervention required.
I could see it clearly now, the end of my days. Man, it burned.
The next day Justin came in and sat down next to me. He was jamming at his awesome code and I turned to him and said, “Hey dude, I got to ask. Did you go to school for software development? How did you learn all this stuff?” I genuinely wanted to know where these unicorns came from. His response was a knife in my chest.
“Nope. I was an admin, just like you. I just decided I didn’t want to do that dumb shit anymore.”
In that moment, my whole career changed.
I sat in my chair for a good hour just staring at my screen. I honestly wondered if all my experience didn’t matter anymore. My ego would have you believe that people like me would always be in demand, after all, there are so many Windows Servers out there needing to be managed (manually)!
But I was lying. I realized that I was staring at obsolesce in my laptop reflection. And it left me with bitter resolve.
After a lot of drinking, I came in the next day with the attitude of how hard can it be. After all, these weren’t unicorns, they were IT folks just like me. At first I wondered what kind of training I could take. Maybe I just needed to get a few certs in this stuff and I’d catch up to them. That thinking got me far, but not nearly far enough, fast enough. I decided to just dig down to basics and get my hands dirty.
A few months later, two more DevOps guys joined our team, Lars and Roman. By this time, I’d learned about Agile and had a basic understanding of git workflows, building a Jenkins job, writing a few RSpec tests, but for the most part I was still new and got lost in the methodologies quickly. It often felt like I was learning to drive in reverse, in England with the music blasting. I was overwhelmed and really had to resist just cheating and going back to using a GUI or resorting to my powershell scripts. I was like an addict who used wizards and a mouse for far too long. But I was determined, and my teammates were great.
Fast forward a year. I’m still not very good, but I’m writing all the Windows cookbooks (since everyone on my team refused to touch Windows). I could take on sprint work. I could write tests (mostly). I could build Jenkins workflows. Suddenly, I was “okay” at this DevOps thing.
I will tell you, up until this DevOps path, I’d never apply for a job I didn’t know a minimum of 65% of what was expected of me. It was the perfect blend of having enough skill and room to grow. Well, that wasn’t valid anymore.
It was time to go out in the wild. I applied for a job I had no business getting. They used Ansible, Python, Perforce, and it was 100% Linux (which I was not comfortable with at all). As I was interviewing for the job, I told the hiring manager straight up, “I don’t know your entire stack, but I can do this job.” After all, if there is one thing I learned in the last year was that I can do whatever it takes — that is the DevOps mentality. The challenges of learning the new world order of IT were not so impossible any more.
I got the job. Less than a month in I was more comfortable cruising around in Linux, writing Ansible playbooks, and creating scripts in Python. There was a point where even I couldn’t believe that I pulled off my bluff. Then I suddenly became the unicorn on my team. I was in the strange place where exactly half the team were operations (admins) and the other half were developers. There I was in the middle, figuratively somewhere in the middle, of both teams.
I found myself asking the ops folks why they wouldn’t automate a lengthy configuration, or asking the developers why don’t get the tweaks and tricks of the old systems being managed before they started designing the new systems. One daily standup, I asked the question:
“Why don’t we write any tests?” It wasn’t a question just for the ops folks.
I received a flippant answer about Ansible does everything for you. While it’s kinda true, and it does a great job of orchestration, how do we know that when we push a playbook that the servers were configured how we intended?How do we know the code is correct end-to-end when we make a change … without manually testing it? I realized at this point I was starting to think like a DevOps engineer. It was still new to me, but I realized it nonetheless.
My next gig wasn’t very “DevOpsy,” at all. Strangely enough, it was a culmination of my experience up until that point (and will further my final notes of this article). The job was to migrate a customer’s old data centers to AWS. They had every piece of technology under the sun. You name it in the last 30 years, they had it, and they wanted all of it moved to the cloud.
Migrating these data centers felt strangely comfortable — I was working on so many old systems I was accustomed to prior to starting my DevOps journey. On the other hand, I wanted no part of it. There was nothing cutting edge about “lifting and shifting” old workloads to a cloud. I found myself constantly scrutinizing the ways in which those systems were managed, how crufty they were, how extremely manual everything was — I was really scrutinizing my old ways. This migration was a huge feat and a story for another time.
In comes the BIG test.
I interviewed at a startup that was walking distance from my home. Beer in the fridge, bike racks, dart boards, it was amazing. My potential teammates were extremely sharp. They were doing all the cool stuff, 100% in AWS, CI/CD, immutable blue/green deployments, all Linux, using Puppet, ELK, containers, MongoDB, and the list goes on. It was a real DevOps shop. Again, I didn’t know about 85% of the solutions but I used the same line,
“I don’t know your entire stack, but I can do this job.”
Well, I got the job and man was I excited. I’d finally made it into that unicorn stable and was going to do all the cool stuff the cool ways. I was going to learn this new set of technologies from really smart folks, but the difference now was I was able to understand the DevOps practices.
First day I show up, I find out one of two DevOps engineers just had his last day. The other guy put in his notice, that day.
Then the CTO left a few days later. He managed several pieces of technology that only he knew in the company.
Then my boss stepped down. She wasn’t getting the support she needed from the business.
Now, it was all mine to manage. There was no real turnover, no real documentation, no guidance, just code.
Without going into the real specifics that were this perfect sh*tstorm, I will tell you that I was left with a disgruntled platform development staff, a mess of technology I didn’t know, and end of year deadlines looming just ahead of me.
I could have run. Less than two years prior, I definitely would have run straight back to my old company and begged for my job back. That was the old skillset, the old fears, and it was no longer me in this career.
After some brief soul searching, I dug in to all the technology as fast as possible. Some solutions were great, some were massive rabbit holes, some were dragons that were really hard to tame. Having gained the confidence and skills from the past few gigs, I was able to dig through code, follow deployment processes, improve test coverage: all the right things.
Six months later, after some very scary moments, a few close calls, some good teamwork, and a little luck, the entire stack was under my control.
Now, I tell you all this to get to the learnings of my transition. It’s not about coding, or using cool tools. What becoming a DevOps Engineer meant to me are these things:
All trains go through me
This doesn’t mean that I’m the bottleneck, or that my ego is that inflated. It means I have stake in every part of the business, infrastructure, the SDLC. If it is on me to automate a process end-to-end, I need to be part of all the conversations.
There is no wall
It’s common in the IT industry to throw the problem over the wall. “It’s the developer’s job to do that.” Or the infamous, “It worked on my laptop, sounds like an ops problem.” The bottom line is, it’s my problem, all of it.
Tests are great, results are better
It’s easy to get wrapped up in the perfectionism of DevOps: ensure 100% test coverage, including all edge-cases (yeah right). Fully DRY code, putting your workloads in the next cool things like containers, or Kubernetes. I can keep going, but what is most important is trying to not only improve the end-to-end process, but ensuring that it all works in a very stable and supportable way. As a former ops guy, I think it’s more important that everything works well, than everything being executed in the coolest way possibly by following some golden DevOps handbook.
Learn what you need to know for TODAY
Don’t worry about the rest. The old me would have worried about all the things I didn’t know and try to learn everything. There is so much I still don’t know, I often feel like Socrates:
“The ancient Oracle said that I was the wisest of all the Greeks. It is because I alone, of all the Greeks, know that I know nothing.”
Worrying and trying to learn everything is a huge waste of time and energy. Tackle today’s problems and move on.
Be the solution for your business
It’s no longer about creating a server or a database that runs in the shadows of your data center. It’s not about asking for a ticket in the correct format so you can do your job. Be the problem solver for the challenge. Research, test new technologies and don’t be afraid to fail.
Keep things simple
This should go without saying, but I’ve seen far too many “clever” DevOps people who only make the next guy/gal’s job so much harder by their automation wizardry. Document your ****** code.
If it is a problem or you’re going to do it twice, automate it
If you enjoy sadomasicism, then keep fixing that alarm after it wakes you at 2am. Automate the problem away so you you can clock out at 5pm and not worry about the night calls.
Maybe you’re new to IT. Maybe you are an administrator of 15 years like I used to be and looking to make the change. You could be a developer who wants to understand more about infrastructure as code. Whatever your reasons, know that you can absolutely make the leap. If you’re an old schooler, like I was, it just takes a ton of grit, but it’s totally worth it.
If you enjoyed this story and want to know about that nightmare project where I had my “ah ha” moment, here is that story: