Continuous Integration: Fighting the Fear

This is an older post published on my blog on Feb 07 2015, I’m moving a few old posts over here.

The Fear

If you’re anything like me, and judging by the brilliant people I work with there must be more of us out there, then every now and then you get the fear. The fear is that feeling you get when complexity starts to creep in. When processes you have to go through start to get hard to repeat, or begin eating more and more time. The gut feeling that there must be a better way.

When it comes to getting your projects into user’s hands, continuous integration is that better way.

Continuous what?

Recently “devops” has become a popular talking point so I doubt you’ll have read too far without hearing the term continuous integration, but just incase I’m your first…

Continuous integration in it’s simplest form is an automated process to make your builds and deployments safe and repeatable. There are many tools out there too help make this process as easy as possible, aiming to provide the flexibility to build a pipeline from your code to your production environment. Recently when a colleague mentioned how easy deployments were, even though we were adding more and more services I thought I should share the system I’ve been setting up and polishing over the last few months with the world.

Hopefully the best way to show the power of CI is to tell you how we use it at work to make our teams and our code better, faster, stronger. I’ll take you through it stage by stage:

GIT

It all starts with git flow for us, we’ve adapted it slightly but it’s the same idea. We have three permanent branches:

• master is our current production release. No one has permission to push into this branch, it can only be merged via Stash. Stash is a bit like github but aimed at the enterprise. Stash gives us some extra restrictions as well, we restrict any history changes and we also refuse any pull requests without passing builds (more on this later).
• release is our current staging or UAT branch. Again no one has permission to push into this branch and it has the same restrictions around passing builds. Once we have finished testing a pull request can be made and merged when the project lead is ready.
• develop is our current integrated build. All other feature branches need to be merged into this branch to move towards a release. Depending on the project size, pull requests are sometimes required so that code can be reviewed by one of the senior team members. Stash gives us the ability to restrict how many approvals are required for a merge and by which group of people.

Beyond this we just have the standard bugfix and feature branches. On it’s own this is a good way of working to make sure the code bases stay clean and focused. Whoever is given responsibility of leading the project is able to control the flow of code and give direction where needed. This isn’t the end though, we use these branches as triggers for the next stage of our pipeline.

TeamCity

TeamCity is a “Continuous Integration Server” from JetBrains that really is the the hub of our toolchain. You can configure multiple projects, with each one containing multiple build configurations. A build configuration is essentially a series of steps that can be triggered in different way, for us they’re going to be triggered by our branches.

We have a build configuration for each branch defined above, but they are all variations on the same theme. Mostly the changes are which parts of the version number that are going to be incremented, the visual studio target (debug, release, etc.) and finally where the deployment should be triggered.

The parts that stay the same are the steps that make sure that the projects build and execute any unit tests that can be found in the solutions. If any tests fail then our builds fail, notifications are sent into the project rooms on hipchat and the TeamCity dashboard glows red. When I mentioned in the GIT section about requiring passing builds, this is what I meant. Pull requests can’t be merged unless you have passed all of the tests (no cheating either, TeamCity will fail if the quantity of tests drop below a certain level).

Each build configuration finishes with a deployment step. Deployment is completely possible with TeamCity but we find Octopus Deploy to be even easier so we use that. Always striving to remove friction and remove the fear.

Octopus Deploy

Octopus Deploy is a brilliant application, pretty much built on top of powershell with one purpose, deploy your software. It’s pretty much aimed at .NET developers, but that’s ok, we’re .NET developers. Your milage may vary.

Octopus is configured with steps like in TeamCity, but here we start with the nuget package that comes from that last stage of our process. Steps can be added to send notifications, pause for manual intervention or start modifying the files it’s received. You can specify variables that transform configuration files in your code depending on which environment you want to deploy to (develop, release or production).

When you setup octopus you install what they refer to as “tentacles” on your servers. They are basically a client that receive your files and configure their environment. In your deployment steps you can configure which environments use which server.

The big picture

Here’s the situation you have to imagine: Deployments are now run of the mill, your main focus is on the quality of the project. A request come in, you review it with the developer and when you’re happy you approve the merge. Now you go about your work, a message will arrive to let you know it’s deployed in a few minutes so you can go and experiment with the most up to date version.

This has happened a few times and you feel you’re ready for a release. Make a pull request into the release branch and the team can review it, when you’re happy the merge is approved. In a few minutes a message comes through and you’re ready for your testing process.

Finally when testing is complete the testers submit a pull request into master and the project lead can accept the merge when they’re ready to have production updated. A few minutes later their message comes through. No fear and completely repeatable.

What does this mean for your team?

Most importantly, they can get on with what you pay them to do. Build things! Personally I’ve found that as we’ve started to decompose our systems into smaller components it started to become a bit difficult to make changes to a service that I’ve not looked at in a while. Did I remember all of the steps required to deploy? Even if it was written down, is it up to date? All of that is gone now, I know the CI server is going to take care of it all for me.

The fear is gone, and I’m left with increased productivity.