Scaling Environments Up & Down
Don’t forget to scale down for software development
Build software for scale is a common adage among software professionals. Conventional wisdom suggests that a software system should be constructed in such a way that it is highly available, resilient and flexible to change. In short, it should support both immediate needs and future growth. However, somehow we commonly forget about supporting the software developers who are responsible for the future development of the software product. We get so wrapped up in making sure the software can scale up that we forget it needs to scale down, too.
The People Who Build It Also Need To Run It
There are few things that suck worse as a developer than not being able to run the software you built in an isolated environment — you know, somewhere you can experiment and break stuff without getting chastised. It is well known that there are multiple levels of testing. Shouldn’t developers be able to verify that the software they created works at multiple levels, too? That’s a rhetorical question. Of course they should!
Software development and software testing are two sides of the same coin. Developers write tests and code. They shouldn’t write code having no idea as to whether or not it works and then pitch it over the fence for someone else to test (sadly, a practice that happens all too often). The goal is to write working code — if you don’t know if it’s working, then it’s not working.
Now, that we’ve [hopefully] established that developers are responsible for making sure the code they wrote works, they need somewhere to do that. Unit tests can generally be written and executed on the developer workstation. However, higher level developer testing needs to happen somewhere. Maybe it’s on the developer workstation? Maybe it’s on a local server cluster? Maybe it’s in a cloud environment? Regardless of where it is, developers need access to run it without fear or inhibition. Additionally, the creation of the environment and the running of the system needs to be streamlined. If there are significant barriers for developers to test the stuff they built then the message being communicated is, “we don’t want you to test your software.”
Note: “We don’t want you to test your software” is not a great message to communicate to the people who are expected to build working software.
Environments
The Development, Test and Production environment paradigm of yore is just is not good enough. Multiple developers stepping on each other in a common development environment and stark differences between the environments that require different deployment steps/mechanisms mean that testing is not “safe.” It’s not safe for developers, it’s not safe for testers and it’s not safe for operations. Different deployments for different environments mean that Production deployments are only really ever verified in Production and it’s anyone’s guess how well that’s going to turn out (probably not good). Different environment configurations mean that any kind of system testing is not comparing apples to apples.
Therefore, the following environments are proposed as general guidance.
Local
Local is where developers can run a complete working system in isolation. This is a per-developer playground. They can see how their changes impact the system, spin it up & down on command, etc. It may not be partitioned like a live system, but it has all the working pieces running together. With tools like Kubernetes, this could be on a local developer workstation. Of course, it could also be in the cloud using something like Terraform to provision.
Development
The Development environment is a reflection of the changes on deck. It should be a reasonable facsimile of the Production environment, but perhaps scaled down. It could also be internally facing instead of external. It is deployed in much the same way that Production is deployed and it is kept up to date with the latest merged changes (via automated deployments?). The Development environment is owned by the development team.
Test
There can be various types of Test environments: Functional Test, Load Test, etc. But, a Test environment is an environment that software testers own. They control what is deployed, when it is deployed and how it is scaled.
Stage
The Stage environment is as near an exact copy of Production as possible. It is deployed exactly the same way, it is scaled the same way. In short, it looks, feels and behaves exactly like the Production environment. Stage can also be one side of a green-blue deployment that is continuously made available to the team for validating stuff without actually being in Production.
I knew of one organization, in particular, who always kept a Stage environment running. Deployments were done using green-blue. First, the Stage environment was updated. Once the Stage environment was working, Stage would then become Production and Production would become Stage.
Production
The Production environment is the live system.
How To You Know If You Are Doing It Right
Can new developers check out code from a source repository, build it, test it and deploy it without going on a multi-hour (or multi-day) scavenger hunt for hidden tools, other repositories or tribal secrets? If you answered yes, then you are probably doing it right.
So, how do you ensure that you are doing it right and that you keep doing it right? Here are a few tips:
Maintain a consistent deployment between environments
Consistent deployments between environments not only reduce the amount of deployment script/code and documentation that needs to be maintained, they also make it easy for all technical stakeholders to understand what is involved in a deployment. Additionally, consistent deployments mean that relevant deployments are executed more frequently — because every deployment is a relevant deployment. As a bonus, the more frequently deployments are executed, the less likely they are to fail and the more quickly failures can be identified and resolved when they do happen.
Use common build, test, deploy patterns and practices
Q: What’s easier than training people on how to build, test and deploy software?
A: Not having to train people on how to build, test and deploy software because you are using common patterns and practices.
Put everything you need in one place
I worked on a project once where you needed to check out code from 5 or more repositories just to complete what should be a trivial task.
You want to deploy to your own isolated environment? You need to also check out the IaC repository.
Don’t require people to be master cartographers just to map out the stuff they need to write, build, test and deploy their code. Put all the stuff they need next to each other: check out, build, test, deploy.
Resist the need to make up the next great bespoke software tool
The tooling required to build, test, or deploy software should be very obvious in its use. That generally means people have used it before, it’s super well-documented or it’s so simple to use that it doesn’t matter. If you can make it all 3, that’s a best case scenario. Developing software is hard enough without having to learn some one-off thing to do your daily tasks that you can’t Google and that only makes sense to the person who wrote it.
Some Final Words
Software systems need to scale up for Production workloads and scale down for software development. It’s not just a good idea to enable developers to easily build, test, deploy and run the software systems they work on in an isolated environment; to do otherwise is an impediment to delivering working software.
Thank you for reading. If you liked this post, please check out the Blu Flame Technologies Blog.
