How To Become a DevOps Engineer In Six Months or Less, Part 5: Deploy
Quick Recap
Let’s quickly take stock on where we are in our DevOps journey.
In Intro, we talked about the DevOps culture and the foundations required:
In Part 1, we discussed how to properly lay the foundation for future code deployment.
In Part 2, we talked about how to keep your deployed code organized.
In Part 3, we talked about how to package your organized code for easy deployment.
For reference, we are in Deploy stage of our map:
Therefore, if you spend about a month on each section, we are in month 4 at this point.
So, we know how to provision infrastructure that will run our software, we know how to version it appropriately, and we know how to package it for deployment.
Finally, we will discuss how to actually deploy your code!
Code Deployment
Have you noticed how right above I didn’t actually say, “how to deploy your code easily”? That’s for a reason.
Unfortunately, proper code deployment from a dev environment to prod is still a painful process, fraught with errors and failures.
Why is that?
Reasons are numerous, of course, but in my opinion it mostly comes down to differences.
Specifically, differences between environments where code is created and where it actually runs.
In fact, I would argue minimizing those differences is the single biggest improvement you can make not just in your overall code deployment but also post deployment run-time.
So, how do we go about reducing and/or eliminating the differences between our prod and non-prod environments?
It Worked On My Machine!
If your dev infrastructure looks like this
But your prod infrastructure looks like this
then you have a problem.
If you are utilizing infrastructure-as-code instead of configuring things by hand, then you are 90% of the way there.
If you are not, don’t despair — you are not alone. Take an afternoon, identify what gaps you have (training, culture, people, processes, etc.) and methodically eliminate them one by one.
Bottom line is that you are not going to be successful in managing a modern tech stack if you are still configuring things by hand. It’s as simple as that.
So, first thing you need to do is make sure everything that touches prod is a versioned artifact, deployed by your deployment server.
Assuming that’s done, I will argue the best way to deploy code is not to deploy it at all.
Modern Approaches to Code Deployment
That’s right — deploying code to prod machines is very 1990s.
The biggest problem with deploying code to a set of fixed production machines is that by definition, your prod servers (where code runs) are different from your dev servers (where code is written).
So, it’s no wonder a ton of issues arise immediately post deployment that were never seen before — everything is different!
Therefore, you need to do all you can to make sure your deployment artifact is the entire runtime, not a piece of code.
In other words, deploy your code once to your dev environment, clone the entire machine your code runs on and then copy it everywhere it needs to go.
This is known as “immutable deployment” and is a very powerful pattern that will save you hours of post-deployment headaches.
Of course, if you run containers, same idea applies: you deploy the same container everywhere.
“But wait! My prod is different from dev!” you might say. Database usernames/passwords, connection strings, S3 bucket locations, etc. These are all different!
Yes, they are different.
The way to solve that problem is with the 12 factor app config principle. All your configuration needs to be externalized and passed as environment variables to your machine.
For example, if you are in AWS, use SSM as the external parameter store — it integrates nicely with CloudFormation. It is also super easy to set environment variables directly from the aws ssm cli commands. Of course, other cloud providers have similar mechanisms.
Moreover, resist the urge to “fix” your prod machines when things go wrong. The machines are immutable and that means whatever fixes you make must come from dev.
In fact, your goal should be NO access allowed to prod machines at all. No ssh, no scp, no prod access to anybody. Not you, not aspiring hackers.
But what if I need logs to troubleshoot the problem?
You guessed it — your logs should also be externalized, ideally shipped elsewhere either with an ElasticSearch/Logstash/Kibana (ELK) stack or commercial software like SumoLogic or Datadog.
Whatever you do, your prod machines are “cattle” — they are replaced at the slightest sign of being unhealthy. They are not “pets”, to be nursed back to health by spending hours on troubleshooting efforts.
NOTE: Yes, I know this analogy is overused and I hear from people who actually take care of cattle that this is not how this works, exactly, but the point stands. Don’t “fix” your prod machines, fix your dev and redeploy.
Mechanics of Code Deployment
OK, so you know what to do but how to do it?
GitLab CI is a good place to start.
It’s a solid foundation upon which to build your CICD skills. GitLab is well respected in the industry and learning how their pipelines run will position you well for the future.
Remember, you want your CI/CD pipeline to live right inside your GitHub/GitLab repo codebase. That way pipeline itself is a versioned piece of code!
In fact, this is so important that it bears repeating again.
Everything is code.
Your application, how it is deployed, how it is monitored, how it is configured, etc. — all pieces of code, stored in GitHub/GitLab/Whatever, properly versioned.
The goal here is to create a truly frictionless environment for the core developers (software engineers who write feature code.)
For example, I should be able to write my little microservice, add whatever tests I deem necessary, add a Jenkinsfile, add monitoring-as-code configuration, specify my parameters in some “env.yaml” file, store it all in one repo, have Jenkins auto-discover said repo, build it, test it, deploy it (as canary or blue/green) and send me an email when it’s done!
Whew.
That is the goal! In fact, this is the very essence of DevOps engineers’ core mission.
Alternatives to GitLab
One is AWS’ own CodeDeploy service. It has limitations but the developers behind CodeDeploy have made significant improvements over the past year and if you are in AWS, I strongly urge you to give this a try.
Finally, GitHub announced Actions, which is their own pipeline-as-code automation.
Really, I don’t think tools matter all that much here. What matters is knowing that everything, including your code deployment pipelines, are versioned artifacts and nothing goes to prod unless it comes from dev first.
In the next section, we’ll talk about how to actually Run your code.
