DevOps Launch Control
Let’s face it, we all yearn for greenfield projects. Is there anybody that doesn’t want a blank slate for a project, a chance to start something so pure, so thought out, the code itself reads out like a piece from the Struga Poetry Evenings, the architecture diagram worthy of a spot on your bedroom’s wall right next to the Lamborghini (or favorite boyband) poster?!
The problems will start to pile up pretty fast, pretty soon after the first building block is in place if you don't set some ground rules. Here are my 5 rules for a perfect project launch control:
Security! Security! Security!
I’ll say it again, SECURITY! I cannot stress enough how important this is. Do not cut corners here. If you weave your security guardrails from the start it will be much easier for the evolution of the project to happen within the confinement of these security measures. It is monumentally harder to later refactor and trace back all loose ends so you can strengthen your project. Here are some of the most common mistakes made with new projects:
All the engineers work from home and they have dynamic IPs so let’s just leave the database public and open to 0.0.0.0/0 so they can connect to it easily and ship features fast. Don’t worry we’ll protect it with a very complex password.
Guess what?! You should worry. Your “very complex” password is quite literally the last line of defense and to be honest it’s a very thin line. There’s a ton of other ways to cause mayhem to your system if you expose your database to the world. Your system suddenly becomes that much more vulnerable to social engineering (just one password is needed to get fully pwned), DDoS attacks on your public server IP (the cloud providers are not impervious to this, you know?!), Slowloris (the most awesome name for a cyber attack) and much more. It’s a hacker’s dream playground.
Why would anyone want to attack us?! We are an insignificant player in the market, just starting, we have no enemies. We can afford to be a little less strict around security.
Let me tell you a short story, I once attended a company event (a comedy improv show) where immediately after entering the theater, a friend of mine who’s been known to dabble in the black arts (hats), stands at the entrance, scans the room for weak points just like Robocop’s threat analysis feature and sees the theater projector. Pulls out his phone and using the infrared port takes over control of it in less than a minute. When I asked him why, that time, and every other time he pulled a stunt like that, the answer was always the same: “Because I can!”. Sometimes, you/your project/your company will become a target just because you left yourself vulnerable. Another example is the TV provider store fiasco back in my home country. Because the TVs in the store were always on and nobody bothered to change the PIN on the protected channels, somebody, presumably just for gags, got a remote from the same TV box from home and changed all the TVs in the store to the “adult” channels. I would’ve paid money to see the look on the first employee to open the store the next morning.
I could go on and on talking about other issues like:
- Give everyone Administrator permissions because you trust that they know what they are doing
- Skip that sanity check because you are really sure that your change is so small it couldn’t possibly break anything, etc.
The point is you are better off safe than sorry! Agree on standard security measures, put them in place first, and only then start building your project.
My standard set of rules usually include:
- Mandatory MFA
- Least Privilege Access
- No wide-open security groups
- Strong password policy (18+ characters, 3-month rotations)
- No password reuse
- Encrypt all data at rest
Stock Your Toolbox
Most companies nowadays whenever they get a new hire they ship him (back in the wild days we used to do things in person, I know right) a package full of goodies, like a laptop, a notebook, a coffee mug with an inspirational quote that has never inspired anyone, a little colorful puffball that you can put on your pencil top, a business card to a partner therapist, etc. This represents the standard set of things that everyone is using. So, why should projects not have that? If you outline the tools your project will use everyone will be on the same page and there will never be any unnecessary chatter and clash of things. Here are some examples of tools you should define beforehand:
- Version control: Gitlab, Github, Bitbucket, etc. (format commit messages)
- Infrastructure as Code: Cloudformation, Terraform, Pulumi, etc. (along with exact version)
- Documentation: GDocs, Confluence, Readme, etc. (define a template)
- Issue tracking: JIRA, Trello, Redmine, etc. (Definition of Ready, Definition of Done)
- Diagraming: Visio, Draw.io, etc.
This way, you set yourself up for success, so that when your project growth explodes and you need to scale fast on people, the onboarding time is reduced to a minimum, since the way of working is standardized and documented.
Rise Sir Production-Server-Work-a-Lot
If you can count the amount of time IT people have spent most on during their careers I think you’ll find pretty much all of them have one top-ranking item: naming stuff. I am one of those people that names everything around them. My gaming laptop is called “Legionnaire” (it’s a Lenovo Legion, duh) and my work one is “The Thinker” (obviously, a Lenovo ThinkPad). But also, every wild bird outside is called Bim (don’t ask me why), every new dog is Jackie and my car’s name is Hyu (I bet you won’t guess the make :D ). That gives me a sense of ownership, a way to communicate more effectively when I need to reference something. The same goes for DevOps projects. A strong naming convention is like a GPS through an infrastructure labyrinth. This is essential that you set it up from the start. It can be however you like it, we can argue pros and cons of each naming convention template out there, but eventually, it comes to down personal preference. But HAVE IT and stick to it. Don’t ever deviate from it.
The way I personally think about naming stuff in my DevOps world is like this:
If I do an export of the name of every single thing I have created in my project (e.g. AWS resources and services) I want to find what I need by a process of elimination or cutting out chunks of the list.
My go-to naming convention for AWS resources for example is:
<environment>-<region>-<product>-<name/uid>-<type>
This way I can sort of “drill down” through my resources until I get to the level I need. In one “iteration” I can see everything that I have running in PRD (production). I can see everything that is needed to run a certain part of my project, be it a product or a service or however else I have organized my project. Not saying this is the naming convention to rule them all, but it works for me and my projects.
You’re IT
How is it that when most of you post a photo on your Instagram feed you can off the top of your head come up with at least 10 hashtags to append to it, yet if I ask you to come up for tags for a specific database or a server instance, you get stuck on anything past the name (which we already agreed on :) ), or maybe one or two more?
There’s not a person I know that didn’t grow up playing the most universal language-barrier-breaking timeless game of “Tag”. Well, I guess you can think of this as the “Tag 2.0, Digital Edition”. In Tag 2.0, when you touch something it’s not just “IT”, it is also “This” and “That” and maybe “Also that”. Tags on your infrastructure are like metadata. They say so much with so few words and coupled with the strong naming convention it is a match made in heaven. Some tags to consider adding on ALL of your resources are:
- CostCentre: Core, Dev, QA, PRD, API, Web, etc. Up to you, how you want to organize your billing overview.
- Environment: Dev, Acc, Prd. etc… Despite this being also in the name, programmatically querying for resources from a certain environment is much easier (read “better”) with a tag than regex parsing the name string.
- Service: API, WEB, ADMIN, CORE, MOBILE, etc. It can be any logical separation of services you have on your project. Maybe even each microservice if you use that architecture.
- Autogenerated: True/False. If a resource has this tag set to true and you make a manual change to it, you deserve any and all repercussions that will follow after the next release of your IaC.
- Repository: if you are using multiple repositories to organize your code the name or even the whole URL of the repo stored in this tag can have the same effect as using a cheat on “Where’s Waldo” to immediately find him.
Happy Little Tree
Last but not least in this launch control setup is the repository folder tree structure. This is especially important if you are using a multiple repository strategy. Defining where things are in your repository is much the same as organizing your house. If you want to cook eggs, you go to the kitchen; if you want to take your dog for a potty break you take him/her to the garden. The same should be with your repository code. If you want to make a certain change, you should immediately know where to do it. This varies highly based on the project context and there really is no wrong way to do it as long as it is clear for everyone what goes where. There are template repositories for pretty much any hello world project out there, and you can also use tools like cookie-cutter.
Conclusion
I know we are all eager to get our hands dirty in new projects, especially greenfield projects, but taking the time to define these few things upfront could save you countless hours of your time and maybe a few hairs on your head down the line. We already live in a chaotic enough world as it is, especially the world of IT, so adding this bit of structure and standardization to an itsy-bitsy teeny piece of it, will do no harm but it might make a Developer or DevOps somewhere smile for just a moment and the skies will open up and a ray of sun will shine on the keyboard and a masterpiece of software will come out of it. Software that just might become the Ph.D. research topic of a future IT historian. Isn’t that what we all strive towards?!