Tale of Two CIs
A follow on from my unit testing article from a few months back and loosely based on real life experience…with apologies to all who think I might be referring to them!
So, you work on a project, you write code, test it (I hope!) and deploy to production. Not so hard was it? So what’s all this noise about CI? We’re doing just fine, aren’t we?
Now fast forward a few years. Your predecessor (but two) wrote some program that the business now relies on. But it’s broken. The libraries and OS that it relied on have moved on, security standards have updated and the Computer Says: NO! NO! NO!
Surely someone must know how this POS works (after all its your job on the line now!). They’ve all left. And they’ve all wiped any reference to your current employer from their LinkedIn profile. Just in case you came looking for them.
So where’s the source code. Is there any? Is there too much? (10 different versions in some obsolete source control system — any guesses which one was deployed to production?) Let’s take the most recent version and hope it wasn’t a work in progress, abandoned when your predecessor (twice removed) left for pastures new a few years back.
Now how do I build it? It looks like they used Visual Studio, but which version? Probably a couple of generations back (at least). So let’s go and download it and then try and persuade the IT department to let me install what they consider a dangerously out of date piece of software that should only be handled from the safe confines of a hazmat suit.
So a week passes, and you finally manage to install an ancient and unsupported version of the development environment — much to the distaste of your shiny new (and very up to date) laptop. Just open the solution and build it. Not a problem is it?
How many errors? Surely shome mistake. I can’t believe anyone checked in a version of the software which was so completely remote from anything compilable. Or is it me? Am I just building it wrong?
OK lets dig into the errors. Hmm, they seem to reference assemblies on the hard drive (of my predecessor but two). Very useful. Google is just laughing at me when I try to search for them. What now? Wait! The production system must have a copy of those assemblies (whatever they do???) Whew! Day saved.
Try to build again. More unknown assemblies. Had this guy (or girl) never heard of nuget? *Sigh*.
So a week of tracking down obscure and obsolete assembly references. (At least I’m checking my work into a modern source control system, so it’s not wasted effort, is it? ;-/ Now I have something that more or less builds (as long as I keep a firm grip on my lucky signed copy of The Mythical Man Month)
So how do I fix the problem? (my manager isn’t standing by my desk looking impatient. Not really!) And if I fix it, how do I make sure nothing else is broken? And how do I deploy it?
Ok the fix doesn’t seem too hard. While I’m at it I might as well update it to later version of VS and start using nuget. Now we’re cooking on gas!
So where are the unit tests? (did I hear laughter?) Fine, I think I know this tune. Let’s at least write some tests for the bit I’ve fixed as it should save me some pain later.
Later, much later, why are my tests failing? It makes no sense! Did the users never notice that this didn’t work properly?!! :-O
Let’s ask the users: “Oh yes, that was rather annoying, but we just thought it was a feature!”. Grrr. Hard stare. “Do you want me to fix it?” — “Yes please!”
Another week passes. It seems to be building and it probably does the right thing. Now how do I set up an environment for testing? (As my reality distortion field is probably not sufficiently Jobsian to support a fingers crossed direct-to-production deployment) It has to be something like production, so let’s set up a test environment and give it a go. Can’t be that hard can it?
Much later — and I’m sure my hair was not grey when I started it — I have a test environment. It’s not quite like the production environment. I’m not sure what half the config values do, but it seems like it might work, although I don’t have test instances for half of the stuff that production taps into.
Then the PHB (sorry Dilbert) says “you’re sure this is going to fix the problem the CEO has been bugging me about for the last month?”. I cross my fingers (behind my back) and try to sound casual but confident: “Yes, course”.
So the day (or more likely Friday evening) comes. Deployment will consist of stopping production, taking a backup of the old files then copying in the new. (At least there are no databases involved, this time). Restart production and see what happens…..nothing. There goes my weekend!
Come Monday morning the PHB drops by as I clear the empty pizza boxes and soft drink cans off my desk.
“Great job, the CEO is really pleased with me for finally fixing that issue, sorry, with you of course”.
I gaze at him with bleary eyes, trying to remember what a soft bed feels like.
“While you’re at it, there’s this other system that’s not working that your predecessor but two wrote. It won’t take you more than five minutes to fix, will it?”
History doesn’t record if the PHB survived the “workplace accident” on the emergency stairs.
Meanwhile, in a nearby parallel universe..
My alter-ego sits polishing his rose-tinted spectacles as his very own PHB drops in to tell him that the production order system is acting up.
“But don’t worry your predecessor but two was really up keen on ALM, CI/CD and devops, so it should be a cinch”
I know better, but I’m new here, so I might get lucky (for once). Let’s have a trawl through devops and see what I can find.
Bingo! There’s a deployment pipeline for the production order system right here. Says it was last successfully deployed two years ago. Looks like it was built from the develop branch and nothing much seems to have changed since then. Looks like the build pipeline ran OK, so thank your deity of preference that this guy was so OCD about writing unit tests. This could save my day.
Let’s check out the code and see what we can do. Does it build? Out of the box! This guy knew what he was doing. Let’s just run the test suite before I try to break/fix anything. Looks good. Let’s just copy one of his tests to reproduce this problem. Run the test, and what do you know? It breaks just like production.
I’ll just branch the code, update it to make my unit test work. Check in, pull request. No go? OK he’s really OCD — he’s running the build and test suite to check the PR. But I like it, and it was my bad not running the test suite before check-in.
Build looks good now, main branch updated. Let’s use that deployment pipeline to push it out to UAT. Looks like UAT is still there, but this guy’s such a star he even had IaC scripts in bicep to create the environment in Azure. Don’t need to go there today though.
Run the integration tests. Still looks good, lets drop the PHB a call. “Is this really urgent? You’re sure you want me to deploy mid-afternoon?”. Oh you do trust the ability of the deployment process that much to revert to the previous version? OK then. On you head…
I’m cautious by nature, so I think I’ll just check back from the release to see what has changed — devops seems to be showing me exactly the changes I expected to see and nothing else. So let’s press the button and go for it.
5 minutes later — deployment complete, no sirens have gone off. Has anyone even noticed the change? Talk to the helpful guy in accounting. “You know that problem the PHB was talking about earlier?”. Click click. “Looks good now, thanks mate, can’t believe how quickly you fixed that. This sort of thing used to take months the last place I worked at”.
I retreat to my desk where the PHB is waiting. “Great job that. Managed to catch it before it got to the CEO, so thanks for that.” Checks his watch. “I think it might be time to call it a day, fancy a pint on the way home?”
