How slow deploy is killing your productivity
This is just a story of my day. It was an incredibly boring day. Why do I write about it? I analyzed why it was so dull and uncovered a great insight. What is the most critical factor in software engineering productivity? I think it is the time it takes to deploy your app.
Beautiful start of the day
I woke up at around 10 am. Not too bad, given that I’ve stayed late the night before — I have been trying to solve an engineering problem at work. The problem was adding a new feature to the gigantic project with a not so scalable architecture. The project was born out of the hackathon, so it’s nobody’s fault… Anyway, the engineering problem. It wasn’t even that hard because as soon as I woke up, I had it all figured out in my head. I just needed my morning coffee first.
I started coding right after my first sip. Something really happened in my head during sleep, because it felt so easy to write code. The solution was pretty obvious, and it did not introduce any unneeded complexity to an already chaotic code base. It looked good, so I ran the code locally. Oh, yes, I forgot to import the module here. Oops, this variable is NULL now, let’s fix that. Okay, it seems to work now. So I committed changes to my branch and created a PR. It’s 11 am on the clock. Pretty sweet as a start of the day, ha? Now what? Even if I solved the problem, it’s not going to do much good without getting into production. Now it’s time to get my code to the finish line.
My open PR had automatically triggered tests. We have all kinds of tests: unit, integration, code style, you name it. I knew I had some time because this process is not usually fast. However, I didn’t have enough time to shift my focus to another task or fully relax. So I kept on checking my messengers, social apps, and news feed. Then a message arrived in Slack. “Lint-check failed.” Oh, I hate those! I thought I had everything right! Okay, it looks like I forgot to remove this import, and it’s not used anymore. I removed one line from the code and committed my changes. 5 minutes wait, again. What’s going on on Twitter?
We all hate waiting
I did not notice how 15 minutes passed. Then I got another slack message — integration tests failed. At least my code style is right. So, I spent another 5 minutes fixing the issue (integration test actually did an excellent job catching a tricky bug, so I’m not mad at it). Then I committed my changes and played the waiting game again.
I understand the importance of tests, but 20 minutes, come on! Of course, I could’ve made some progress on another task, meanwhile. No, not really. I don’t know about you, but I hate switching focus. It takes so much time to get into a really focused state. So I’ve checked all my apps on the phone a couple times and found the bottom of the twitter feed before I saw a Slack notification about the successful deployment of my code.
Just to be safe, I need to test the app on production to see if the new feature works. From first glance, it did, but there was a small glitch. I had to update the code in the UI of a web application, nothing fancy, just changing a string. This turned out to be a header on the page, and I added redundant whitespace at the beginning, so now it looks weird. Okay, here we go again.
Now all tests passed, but the deployment took forever. I’m not sure what kind of computation is involved in moving a folder with files around 10 megabytes from my laptop to the server, but it sure feels like it could’ve trained a conscious AI by now. I don’t remember how much time it took for me to refresh the page and finally see the whitespace disappear, but it definitely took another cup of coffee. I managed to check my emails during this time, so I was productive, I guess. Anyway, it’s 2 pm, and it’s time for lunch.
How much time we waste
As you may have noticed, I’m complaining. That day I really spent around 1 hour actually solving an engineering problem, including bug-fixing time. But it took me 4 hours until I saw my changes in production. There are many days like this, so time adds up. I feel righteous anger because this thing is common in the industry, and many people experience the same thing.
So, what’s the solution? Not writing tests? Of course, not, tests are freaking great. The issue is more generic — it’s all those inefficiencies that are easy to introduce to the deploy time (I bundle tests and all CI/CD stuff in the deploy time). It’s easy to write an inefficient CI/CD job. Its intent could be positive, and at the time, it may seem like a great idea. But a little delay is multiplied by the number of people working on the project and frequency of changes they make. Because deploy time affects EVERY contributor and EVERY code change.
Slow deploy means slow feedback loops. Imagine you’re an artist. Now imagine each time you move your brush on the canvas, you need to wait 5 minutes before the paint appears. That would be the most annoying thing ever.
The feedback loop (deploy) has to be fast
I believe that each developer who likes to solve creative problems needs to have a fast feedback loop. You change the code — “snap” — it’s updated on the server — “snap” — you see the result.
Please think about this next time you want to add something to the deploy time. And also, think about improving deployment speed next time you work on technical debts. If you cut just 1 minute for a project with 5 contributors, each of them contributing once a day (only on workdays), it saves more than 100 hours a year. And I don’t know how to count improved productivity because of not having to switch the focus, but for me, it would multiply this time by 2 at least. To create better software, we need to improve how we write software.
PS. There is a great talk that inspired me https://vimeo.com/36579366. You should check it out, it’s mind-blowing.
You can read more of our blog here.
Originally published at https://blog.seamlesscloud.io on June 26, 2020.
