10 Things They Won’t Tell You About Software Engineering
To preface, I be not wise, nor old, nor an expert, but I’ve come to gain quite a bit of insight into the world of software engineering, and more specifically, how it has the ability to crush you if you aren’t careful. Furthermore, I hope to impart some of these lessons that I’ve learned the hard way, upon you, in hopes you don’t have to repeat my mistakes. Throughout the course of this guide, I will share my experiences of the dilemmas and truths software engineers face each and every day.
As an overview, here are the 10 things they won’t tell you about software engineering:
- The impromptu meeting is not your friend
- Your life spec requires sleep
- Everyone has time for fixing bugs but no time for writing tests
- Do not build something you plan to destroy
- Doing a task more than twice, automate it
- Explaining something more than twice, document it
- Tech debt. It’s a chess game.
- There’s a thin line between complaining and innovating
- Formula for completion time: x = 2x
- School ended, learning didn’t
Lesson 1: The impromptu meeting is not your friend
Them: Hey, do you have a minute?
Me: Ya sure, zoom?
***3 hours later***
Them: Oh, that’s all I needed to do?
I once spent a full week and a half, scoped entirely for implementation, not writing a single line of code. Time and time again, someone needed help from me, or I needed help from them. This stems, in large, from lesson #6: Explaining something more than twice, document it, but, nonetheless, is bound to occur to some degree. I’d hop on to quickly show someone how to use our new state management engine, help with cutting releases and deployments, and even help others with debugging new pieces of infrastructure.
You must reserve what I prefer to call “get shit done” time, or gsd for short — time slotted specifically to get heads down on your work. Schedule it. Abide by it. Only break it if you absolutely must. To clarify, I’m not saying don’t help people, I’m merely stating to block off time for yourself, as intermittent interruptions keep you from getting “in the zone”. Protect and respect the time you reserve to get heads down. After all, time is the most limited resource.
When I realized how much time I spent in meetings, I started logging my work meticulously. I tracked it down to the minute I started a meeting to the time I spent eating lunch, I wrote down everything. Every week going forward, the goal was to bring that number down just a bit, and I did reduce it to nearly half, but not without the lessons learned from a few other of these.
There are some other caveats here. The first is new hires, who may need more help getting off the ground. In addition to having your gsd time, I recommend a few things:
- Give them requirements that are very well-documented
- Introduce them to other people on the team and tell them what each person is good at
- Never solve problems for them, but rather guide them in the process you might take
The second caveat is root problems that may be out of your control. For example, if your team has grown from 5 to 20 people, like mine did, you’ll have more stakeholders, more opinions in meetings, and a harder time finding meeting slots. While I’m a big fan of Amazon’s two pizza rule, I can only influence these decisions to an extent and such a transition takes time and meticulous planning. To which I suggest to stay small and move fast. Create a workgroup with the key players, ideally two others at most, and run with it.
Lesson 2: Your life spec requires sleep
Them: Unfortunately, the project started late because of reasons. So, we need you to complete it yesterday.
Project deadlines are a b****. If I could rid of them entirely I would, but that’s the software engineer in me speaking, not the business man in me. When I jumped on board my team, we were working on a data lake project. The concept being ingest data from sources, land it in a centralized place, and deliver the data from there. Easy enough, right? How long will it take? Eh, 2 months. That’s what we said the first time. 3 more months? That’s what we said 2 months in. 8 months? That’s what we said 5 months in. A year? Well, take out this and that and sure, a year sounds pretty close to on the mark.
If you think that a project will get done faster because you set an earlier deadline, you’re dead wrong. Unrealistic deadlines cause people to rush, forget testing, build tech debt, and slow themselves down to a halt. In the end, you end up with a project that won’t survive another year, and if it does, no one will touch it with a ten foot pole. More on this in lesson #4: Do not build something you plan to destroy. Additionally, the unrealistic deadlines are partially self-induced, as we’ll talk about in lesson #9: Formula for completion time: x = 2x.
Now, how does this have anything to do with sleep? Well, during the insane periods where we were rounding off that 2 or 5 or 8 months, I figured the only way to deliver is by working round the clock. There was an internal drive to deliver; to solve the puzzle; to conquer the challenge that had thwarted me for weeks. I was so determined to get my feature done in time, I spent every day for 3–4 weeks straight with the following schedule:
8am: wake up. eat
9am-2am: work
2am-8am: sleep
However, on the last week of it, 2am turned into 3am which turned into 4 or even 5am. On Thursday of that week, I was feeling ready to go! Ready to conquer the world! Ready to bring my feature home! When 3pm rolled around, my head started to feel very fuzzy, my eyes were being held open, and I had the worst migraine I’ve ever had in my life. Around 7pm, matters had only gotten worse. I was in such unparalleled pain that I became extremely nauseous. I couldn’t work any longer, no matter how hard I tried. In fact, I was so nauseous I spent the next 45 minutes in front of a toilet bowl dry-heaving. I apologize for the grueling details, but I cannot emphasize enough how much this sucked.
Mental strength can be fought, physical strength, unfortunately, cannot. Had my body not entirely shut down on me, I most likely would have kept going through the weekend. However, being 90+ hours into the workweek coupled with a stupid amount of sleep deprivation, I had no choice but to stop. At around 8pm, I finally was able to get some sleep and I’ve never passed out as hard or as quick as I did then. Don’t get me wrong, I still woke up an hour later for my 9pm 1-on-1 meeting that I had consistently pushed back, but I’ll never make that mistake again.
To this day, I argue that if you absolutely need to get something done by a certain date, spread your time out over a larger window. Rather than spending 3–4 80+ hour weeks, I could have just as easily worked until 10 or 11pm each day had I started earlier and been consistent. In fact, I’m positive I would’ve be much more alert and focused each day as well. Everything in moderation. Just because a bartender will serve you more drinks doesn’t mean you have to keep drinking. The same is true for the endless array of problems that will challenge you as a software engineer. Staying up until 4am one day may not hit you right away, but the day after you feel like an absolute zombie. Net benefit is not positive in these scenarios, so you’re better off taking the time to not physically exhaust yourself. Not everyone comes back from that.
Lesson 3: Everyone has time for fixing bugs but no time for writing tests
Them: Well can you merge the feature in now and get the unit tests in after the release?
Yes, this is a question I have been asked. Furthermore, it’s a question I’ve been asked by a once developer turned manager. Also, if you think the business is going to keep those unit tests a priority after the feature is in, think again.
Testing is a fine balance though. In fact, you can think of testing as an optimization problem. From a high level, the question is how can I minimize overall development time?
What’s interesting though, is that while development time is a function of these four other factors, each of these four other factors are a function of the exhaustiveness of your testing.
Thus to optimize overall development time, it strongly depends on finding the optimal level of testing. In other words, there is a balance. Let’s look at an example.
Roy Osherove in his book The Art of Unit Testing, Second Edition in chapter 9 did a case study of implementing similarly sized projects with similar teams (skill wise) for two different clients where one team did testing while the other one did not. Here were his results:
In other words, all else constant, how does testing impact overall development time? It’s clear here that tests were indeed an overall benefit. Lower overall release time and lower number of bugs found in production, but let’s address the caveats:
- The sample size is too small to determine this to be statistically significant, however, the cost of repeating this case study to a large enough sample size is too expensive, having to develop the same thing twice and discard one.
- The case study does not focus on the level of testing. There can always be more tests, meaning that the team with tests could have spent a near-infinite time on implementation (coding).
That being said, let’s look at some of the underlying intuition of testing:
- The longer a bug lives, the more expensive it is to fix.
- The law of diminishing returns ensures you can test yourself into oblivion trying to ensure there are no bugs.
- Buddha taught the wisdom of the middle path. Tests are good. There is such a thing as too much of a good thing. The key is being able to tell when you are out of balance.
- Every line of code you write without tests will have significantly greater costs to adding tests later than if you had written the tests before writing the code.
- Every line of code without tests will be significantly more difficult to debug or refactor without regression.
- Every test you write will take time.
- Every bug will take time to fix.
That being said, tests don’t add features. Production code adds features. And features are what pay the bills. At the end of the day, I believe investing in some programatic tests at all levels of the testing pyramid will inevitably lower overall development time. Additionally, design for testing — it should not be an afterthought of your application.
Lesson 4: Do not build something you plan to destroy
Them: Hey, here’s a slew of documentation and repos for our legacy system. We’re planning on building a new system so familiarize yourself with the old system.”
***12 months later***
Them: Hey, our new system is taking too much time, we’re going to have to speed up development, cut out testing, and cut out some features.
***3 years later***
Them: Hey, our system has become legacy. It’s too hard to add new features and completely overhaul the tech so we plan on building a new system.
One of my biggest fears, if not my biggest fear, is getting stuck in a loop. Doing things but not actually going anywhere. Think Groundhog Day, Russian Doll, Edge of Tomorrow, Source Code, the list goes on.
In the context of software engineering, this fear is realized through the potential of constantly rebuilding the same thing over, and over, and over. Some of you may be thinking that agile development is constant iteration with marginal improvement and you’d be right. That is not this. This is different. This is the creation of a legacy system, designed to be killed.
In essence, I see development of projects occur in two ways:
Scenario 1: The project was built fast. The code is complex. Testing is minimal. As such, it no longer receives support and maintenance. Technology eventually becomes brittle.
Scenario 2: The project was built slower. The code is clean. Testing is optimal. As such, it receives continual support and improvements. The technology is constantly upgraded and is up to date.
Scenario 1 is the development of a soon-to-be legacy system.
While it’s up to the product manager to determine what exactly to build, it’s up to the developer to determine how long each feature or story will take. Give estimates that allow you to do the right thing and build things the right way. Developers are the final line of defense against falling into scenario 1 and developers face the fallout of scenario 1 the most, having to consistently code in the filth that they’ve created. It is not a place you want to be, trust me. I’ll speak more about this later when discussing lesson #7: Tech debt. It’s a chess game.
Lesson 5: Doing a task more than twice, automate it
I’m 19 years old. It’s the first day of my internship. The data team is walking me through a few of the things they do on a day to day basis.
Them: Ok, I want to walk you guys through a quick demo of our data cleaning process.
Them: ***Opens a folder of 150 excel files***
Them: ***Opens file 1***
Them: ***Creates a pivot table***
Them: ***Filters some values of columns***
Them: ***Runs an aggregate***
Them: ***Drags file 1 to a “done” directory***
Them: Then we do this for all the files and send it over to the client
Me: ***Jaw drops***
Me: What percent of the time do we spend doing this?
Them: Well we usually have about 2 people doing it for about 4 hours a day each.
Me: ***Jaw drops further***
Me: Why haven’t we automated this?
Them: We tried but couldn’t find a script.
Me: Can I make one?
Them: Well, if you find some free time in the next couple months, sure, but we have other things we want you to focus on.
Me: ***Spends the next 3 hours writing a quick Python script with Pandas***
Me: Okay, I’m going to send this script over and let’s get it running on your computer.
Them: ***One click***
Them: Wow, that did the trick.
Them: ***Leaves room. Was not seen for the rest of the day***
I later found out that after leaving the room, they broke down saying the interns were taking their jobs, and left for the rest of the day.
Yes, this is a true story. Yes, this was an early stage start up. There’s quite a few lessons here. For example, don’t upset the person who was supposed to give you a ride home on your first day. Okay all jokes aside, from an automation perspective, 3 hours to automate something saved a years worth of analyst time to do something else, pushing the business forward in other ways.
This was not the only time or place that I’ve had an experience like this. It was simply the most drastic. If you do something more than twice, automate it. DRY.
Lesson 6: Explaining something more than twice, document it
Ok, let’s walk through a similar but different scenario:
Data Analyst 1: Hey, can you explain why my data hasn’t landed as expected?
Me: Ya, let’s hop on a zoom?
***3 hours later***
Data Analyst 1: Oh, that makes sense.
***2 days later***
Data Analyst 2: Hey, can you explain why my data hasn’t landed as expected?
Me: Ok, this is the second time I’ve been asked so let me see if the other data analyst can do a KT, or knowledge transfer, with the rest of the data analysts.
***2 days later***
Data Analyst 3: Hey, can you explain why my data hasn’t landed as expected?
Me: Did the other data analyst do a KT with you guys?
Data Analyst 3: Yes, but I still don’t get it.
At this point, I decided to create the “Why hasn’t my data landed as expected” page. It followed the series of steps to confirm or perform to ensure that data landed as expected. In fact, it essentially just followed the conversation to a tee that I had with each analyst.
While documentation is something that we all admittedly ignore, it’s another essential piece of the puzzle of slowing down to speed up, very similar to proper testing.
Lesson 7: Tech debt. It’s a chess game.
Technical debt (also known as tech debt or code debt) describes what results when development teams take actions to expedite the delivery of a piece of functionality or a project which later needs to be refactored. In other words, according to Product Plan, it’s the result of prioritizing speed of delivery over optimal code.
From my own experience, I’ve had technical debt build to the extent where the engineers decided something must be done! We created what we called “dev huddles”, a place where the engineers could get together every few weeks to list out our tech debt, figure out which were the most pressing, and attempt to get the top couple prioritized by the business.
Developer 1: Ok, this is a solid list and I’m going to bring these items to the product owner and product manager for prioritization.
***2 weeks later***
Developer 2: Ok, let’s go over our tech debt list, find the highest priority item, and try to get it prioritized.
***Rinse and repeat for months***
Developer 3: Ok, have any of our items that we discussed as “priority” actually been prioritized, scoped, or gotten done?
Developer 1: No, but they acknowledged our concerns.
At this point, it was obvious that this strategy wasn’t working. But why? How could something we, as engineers, see as the single thing holding us back the most not get prioritized?
We were not putting ourselves in the mind of “the business”, nor was “the business” putting themselves in the mind of the developers. Lacking this mindset, from each person’s point of view, the conversation looked more like this:
Engineers: If we fix this thing, we will be able to speed up future development on it by 10x.
Business: Does fixing this provide any immediate business value?
Engineers: No.
Business: Here’s 5 features that will provide immediate business value. They will be higher priority.
Engineers: Ok.
***sometime in the near future***
Business: We are behind. Development is taking too long. We need to cut out some testing and documentation to get features out faster.
I think you can see where this is going as well as its recursive nature. Nowhere good for either party. Now, let’s discuss the concept of game theory, something we’ll help use to resolve the issue we see above.
Game theory is the study of how and why people make decisions. Game theory is not just about games, but how and why businesses make decisions, and just about any decision based on valuing likely outcomes. In game theory, all of these situations are “games” since the people involved make choices based on how they value the possible outcomes of the choices. The classical example is what’s known as the Prisoner’s Dilemma:
Now, let’s try to put this in the context of tech debt.
If I suggest fixing an item of tech debt, what is the benefit to the business of deciding to do it or not do it? In other words, what will the business do? And if the business makes the most optimal decision for them, how will that impact the engineer? Now given the outcome of how the engineer is impacted, how can that logic be used to provide the impact back to the business to find an equilibrium that benefits both parties?
In other words, as an engineer, you must think 1 move ahead, being able to clearly explain equilibriums that benefit both the engineer and the business. These equilibriums are often counterintuitive to the business. How does not creating new features help create new features? By not looking ahead, we, as engineers, chose only the option that benefit us the most, ending in a suboptimal outcome for both the business and the engineers.
Choose tech debt items that, when looking a few moves ahead from both perspectives, are in both parties best interest. Additionally, understand the business’s perspective to clearly define the long-run outcome as quantitatively as possible to explain the benefit to the business.
Tech debt is not sustainable, and in time, becomes insurmountable. One of the most classic examples is LinkedIn, who stopped all new features for 2 months to conquer their tech debt.
Lesson 8: There’s a thin line between complaining and innovating
Let’s start with a few examples and see if you can spot the difference.
Example 1:
Complaining Developer: Our schemas are duplicated all over the place.
Innovating Developer: Our schemas are duplicated all over the place. If we centralize schema management with tool X, taking approximately 2 weeks, we can reduce duplicate schemas that’ve cost us 3 prod fires in the past month, equating to 2 engineers 1 day each per prod fire.
Example 2:
Complaining Developer: Our testing process takes forever.
Innovating Developer: Our testing process takes forever. If we invest 2 weeks in setting up programmatic integration testing, our future development time is reduced by 25% by removing manual testing and our test coverage and regression improves each release.
There’s a thin line between complaining and innovating. Both point out the same flaws. The difference between complaining versus innovating is providing a solution.
Complaining is frowned upon and leads to an unproductive culture. If you have a series of issues, think critically about at least a single solution before bringing it to another team member, or worse, your manager.
Lesson 9: Formula for completion time: x = 2x
The first few months on the job looked a lot like this:
Manager: Hey we have this task for you. How long do you think it should take?
Me: Doesn’t seem too difficult. I’d say 3 days to be safe.
***6 days later***
Me: Okay it’s ready for release.
It took twice as long as expected, but what was I doing wrong? Why was this happening over and over again? What I found out was it wasn’t just me. Many engineers were giving estimates that were often far from reality in hindsight. In turn, managers were giving unrealistic estimates for projects up the chain, as noted in lesson #2: Your life spec requires sleep. So let’s discuss some underlying causes:
- I was accounting for implementation time. I was not accounting for everything else. As discussed earlier, development time is the combination of implementation (coding), testing and debugging, integration, documentation, etc. Implementation is only a fraction of what it actually takes.
- I could not actually account for all the hiccups along the way. Going into a task, there are unknown risks, typically due to unfamiliarity with a system, technology, or framework. These are what we call the “unknown unknowns”. These scenarios come up when we least expect them and take time to solve for.
- I don’t spend my 8 hours of the day heads down on implementation. I had intermittent meetings. I had others asking to hop on a zoom for help. I had some prod fires here and there. In fact, the minority of my day was actually spent heads down on a feature.
There are other factors at play here such as properly estimating a team’s velocity and how that’s calculated but the above were the largest factors in that order. As such, the time it took to develop a feature to completion was actually near twice that of what I was estimating.
Take time to think out everything that will go into development of a feature, breaking estimates for each down, padding for unknowns, before giving an estimate.
Lesson 10: School ended, learning didn’t
Yes! I’m finally done with school! Now I can get hands on, sit down, and just build things!
False. Well, you can, you just won’t be a very good developer.
Technology moves fast. If you don’t learn, your technology will remain constant. If technology remains constant, innovation remains constant. If innovation remains constant, your company will soon be behind the rest, attempting to hold together their legacy systems.
Senior developers are those that can bring new technology into a tech stack, know current technology in great depth, and can mentor others on ways to do the same.
Continuous learning is so crucial to software engineering that I would go so far as to say that if you do not like constantly learning new things, you should consider a new career.
Someone reading this: “But I know developers that have been coding in COBOL for 50 years?”
There are exceptions to every rule, but even those using COBOL, with it’s high demand and huge remaining codebase (that pose a systemic risk), have plenty to learn to optimize their development.
But how?
Okay, now that we’re on the same page about the importance of continuous learning, how do you even begin to keep up with technology nowadays?
- Learning by doing. Software engineering is deeply rooted in problem solving. To solve even individual problems in your day to day, you’ll find yourself googling, reading articles, etc. You will never know it all. Satya Nadella says that “The learn-it-all does better than the know-it-all.” Learning from 9 to 5 will only get you so far though and it will not be enough, especially for new developers.
- Certifications. Certifications do more than look good on a resume. They force you to not only learn a technology, but often are accompanied with great tooling for learning that technology in depth conceptually. Online courses are great, but it’s like sitting through lectures without taking an exam. It’s at exam time that you study hard and commit these concepts to long-term memory. That’s what certification exams force you to do as well.
- Be chatty. Whether it’s with a mentor, coworkers, or forums, don’t stop talking about tech. It’s the great developers that are constantly slacking about new tools they find intriguing. It’s the great developers that are constantly responding to StackOverflow and Quora posts about who knows what. Listen to them and become a part of the community.
- Watch talks. Tech is a strange industry, and thank goodness it is. People love to share and benefit from one another. I’m not just talking about Open Source, but in depth talks that people give about what they’ve built. In the past week, I can name several I’ve watched that have been nothing short of eye opening. Capital One’s use of H20 for Mobile Transaction Forecasting & Anomaly Detection, Spotify’s Automated Testing Strategies for Microservice Architectures, AWS re:Invent 200: Deep Dive on Best Practices for Amazon Redshift, etc. There is an unparalleled amount of relevant content no matter what your focus is.
- Do POC’s. Whether it’s for your own learning or for something work-related, always practice turning your conceptual knowledge to hands-on knowledge. One is worthless without the other. Doing so will keep you in a place where you are always ready and able to bring new concepts into your team, or just have in your back pocket.
This is also not an exhaustive list, but a good place to start. All in all, you must find a good balance of the things listed above. Being a good developer is an optimization problem where time is the most limited resource.
Now What?
Now it’s time to put these lessons into practice. I’d love to hear your thoughts about these 10 tips, so feel free to reach out to me on LinkedIn or respond in the comments below!
And if you enjoyed this article, here’s some more content. Enjoy!
