Import the world, harness it for your nefarious ends, profit
TL:DR
- We get paid to solve business problems, not to write code.
- If the problem is not directly related to your business, there is probably a library for that — use it.
- The time cost of writing a piece of code is not just time spent tapping on a keyboard.
- Things that you should definitely use a library for: Money / Time / Logging / Threads / Parsing
- Focus on solving your business problem, import the rest.
Why do we get paid?
Software engineering is a pretty fantastic job. Especially if you enjoy the act of writing software. We tend to get a bit more freedom about where and how we work. Depending on who you work for, you might end up with beanbags, 2 screens, unlimited coffee and often without having to clock in or out.
Contrast that to working in a more traditional role, you have a job to do, you do it, you get paid, this is the contract. Tell us where you are at all times so that we know how much to pay you / charge you out for.
Part of the difference is autonomy, software engineers tend to get more autonomy to decide how they work. Businesses worked out that treating building software like building things in a factory was a very ineffective way to produce good software.
Autonomy cannot be given without accountability, we are not held accountable for having written software. We are held accountable for having solved the problem the business was having.
We get paid to solve business problems, not to write code.
There are lots of problems to solve, there are also lots of people working in software that have already solved problems. Many of those solved problems exist as libraries that you can just reuse and save a whole load of time.
We get paid to solve business problems, there are plenty of other people who get paid to write libraries that solve common problems. Often, these library writers will work full time.
They have a lot more time to dedicate to solving a common problem than we have, because we are working to solve our specific business problem, we can only work, at best, part time to solve a common problem.
These library writers will do a better job of solving a common problem than we will, not because they are fundamentally better necessarily, simply because they are dedicating more time and energy to it than we are.
My advice is, let them, imagine being an individual developer with the coding power of a tech-giant like Microsoft / Amazon / Google / Facebook. That is what you get when you pull in libraries for these established entities. Pull in as many libraries as you can to solve any problem that is not directly related to the business problem you are trying to solve. You need to be laser focussed on the problem you are solving for your business.
If the problem is not directly related to your business, there is probably a library for that — use it.
The cost of writing a piece of code
What happens when we don’t use a library? We need to write a piece of code ourselves.
There is a cost associated to every piece of code we write. Most of that cost comes in the form of time, and an engineer’s time is very expensive. For every piece of code that gets written, time gets spent:
Writing the code — The most obvious time sink, it takes time to write a piece of code, sometimes days, sometimes months. Going back and forth between tech and product, clarifying requirements, ensuring that you have the right APIs. There is a lot of time that goes into writing a piece of code that is not just the time spent tapping on a keyboard.
Testing — Testing takes time, if it is done manually then waiting for a QA to be available, then explaining the feature, then getting sign off. Do the tests need to be tracked centrally? Do automated tests need to be written? Are they flakey? Are there multiple levels of testing that need to be carried out? Answering all of these questions takes time and needs to be done for everything we write.
Releasing — There are some places that can release to prod for every commit, getting to that state takes a lot of time and effort. In my experience the most businesses do not release anywhere near that frequently. Releasing is much more manual, it requires coordination, signoff, massaging through environments. Some businesses will also have to deal with regulatory issues every time they release. Releasing a piece of software is not zero effort.
Monitoring — Once a piece of code has successfully found itself in production, we are not done. How do you know it works? Ideally we’d like to know if our piece of code is buggy before our users find out. Setting up good monitoring in all environments can be tricky. Getting that monitoring to spit out meaningful alerts that are read by the right people at the right time is a lot more than tricky. This is an ongoing cost for running any piece of code in production, the more code there is, the more complexity that monitoring needs to track.
Fixing bugs — Everything that gets released has an ongoing cost in maintenance. Sometimes it will be bugs, sometimes it will be upgrading dependencies to newer versions. Every piece of code that gets written needs to be maintained by someone until it is deleted. They need to understand how it works in a broad sense and will need to update it if the business decides to do something different.
The time cost of writing a piece of code is not just time spent tapping on a keyboard.
Every piece of software we write represents a significant time investment, so we better make what we write count. If someone else has already invested the time to solve a problem, we should make use of that.
Things that you should definitely use a library for
Time
You’d think time would be simple. A day is 24 x 60 x 60 seconds, a year is 365 days and contains 12 calendar months.
Except when they don’t.
Adding 24 x 60 x 60 seconds to a point in time and saying that a day has been added works, until daylight savings. Then it fails, because the amount to add is 23 or 25 hours. I’ve worked at places where a load of tests failed because of this kind of assumption. The end result was we couldn’t release until they were fixed and had to drop what we had been working on to fix them.
A year is 365 days, until a leap year, which is not just every 4 years, it’s every 4 years, except every 100 years, except except every 400 years. That is just in the Gregorian calendar, which is the calendar generally used by the world, but it is not the only calendar. There is also the Iranian calendar, the Ethiopian calendar, the Somali calendar and Thai solar calendar.
Then there are also the weird things that happen to time because of the way people decide to count it, like the answer on this stackoverflow question.
Time is edge-cases on edge-cases on edge-cases.
The best chance we have of dealing with time in an accurate way is to use a good library and leverage the hard work of others. Using a time library means that even simple things like adding a day should not be done without the constructs provided by the library, time is hard.
Money
Money is a topic that gets businesses very interested. It is important to a business that money is handled correctly.
One of the classic mistakes that gets made when representing money is storing something like dollars as a floating point value. This stack overflow question explains why this is a bad idea in a much better way than I can. The summary of the issue being that floating point values cannot represent the base 10 numbers we use for money exactly, so as we do more and more calculations with them we land up with values that a visibly different from what we were expecting.
Often times money will be represented as an exact data type (like BigDecimal in Java) or by using an integer to represent the smallest fraction of currency that is going to be processed. This might mean representing $12.34 as the int 1234 where each increment is a dollar cent, it might also mean representing it as 1234000 where each increment is 1000th of a cent.
What about conversions? That’s not an easy problem to solve:
$1234 dollars converted to Euros now will yield a different amount than converting
…
now.
(unless you are reading this when the markets are closed — edge-cases)
How do we know which amount to use? How do we keep track of the conversions? Do we want to be writing this kind of stuff? — Probably not. These are the kinds of problems that should be farmed out to external companies that specialise in this kind of operation. When representing money in code we write, we should be using a well written money library to capture the complexity of dealing with this subject.
Threads
Don’t.
Threads often get asked about in interview questions, so one might think they get used frequently in software engineering roles. They are being used, but generally they are abstracted away from the engineer by things like a webserver or the environment in which the software is running.
Dealing with threads directly can be fiendishly complex and getting them right takes a significant time investment. Even when using libraries that make working with individual threads easier by adding pooling or the ability to run arbitrary pieces of code in parallel, it is very easy to introduce subtle and hard to reproduce bugs.
My advice would be to stay in a single threaded environment as much as possible because it makes software much easier to understand and we should prioritise writing clear software over fast software.
That single threaded environment might be something like javascript which for most intents and purposes can be thought of as single threaded. Alternatively, it might be looking at a single request inside a webserver which in something like Java can usually be thought of as a single thread of execution, even though there are multiple threads processing multiple requests at the same time.
If you have found yourself in a situation where dealing with threads is necessary, make extensive use of any available libraries for your platform.
For threads more than many other topics, the cost of writing this kind of code yourself is exceptionally high, reusing tried and tested libraries will save a lot of heartache.
Logging
Logging can be simple, we use it all the time, a sneaky print statement here or there to check that some part of the code is doing what you thought. There is a difference between just simply printing something to the screen and having a production ready logging system.
There are things expected from a production ready logging system like:
Non-blocking logging — if the app is doing a lot of logging it should not be slowing down the execution. Then what about making sure that all logs get written to the log file in the correct order? Is every log statement going to open the file, write and close the file? Is it going to batch writes? What happens if the app closes unexpectedly?
Logging levels — our logging system should be able to differentiate the levels at which we log. We should be able to turn on extra detailed logging using configuration or environment variables. Ideally we should be able to target individual files / classes / modules that we want to increase the logging levels on.
Configurable log formatting — the logging system should be able to update the format of the output based on configuration alone. In production we might want to output logs in a JSON format so that it can easily be read by another tool, whereas locally we may want to have human readable logs with pretty colours.
These things are available in most good logging frameworks, if we were writing this stuff from scratch every time we started a project we would land up writing a significant amount of duplicated code.
Parsing
So much of software is just moving data from A to B, from one format to another slightly different format. If possible, try to stick to well established formats that have widely used standards backing them. Things like JSON / XML are fairly standard.
For common formats there are a wealth of libraries available for parsing them. It might seem like a fun project to write an XML parser, but the standard is not as simple as one might think. Parsing HTML using regexes is a particularly common and ill-advised idea.
Use libraries to do the tedious job of parsing things, focus on what happens to the parsed thing afterwards.
& many more…
There are a lot of difficult common problems for which there are libraries / services available, such as:
- Monitoring — If done well it should be agnostic of the type of application it is monitoring, which means that many businesses solve monitoring in a similar way. There are businesses that specialise in monitoring tools, use them.
- Push notifications — Sending push notifications in an easy way to all mobile / web devices is difficult. Handling how those devices are targeted and how to deal with failures is not easy.
- Email — There are a lot of different email clients that receive emails in all sorts of interesting and slightly different ways. Dealing with all that variety when you just want to send some HTML to a load of email addresses is not something you should focus on.
- Payments — Unless the business you are in is certified as a payment processor you basically have to use an external payment system. There are many different payment processors though, and so, picking a service that abstracts that away from you can allow you to focus on solving your problem.
- User management — Every application with users lands up solving a very similar set of problems. These problems have been solved before, there is no need to reinvent the wheel here and write your own user management system.
So many problems have already been solved, 30 minutes of googling can save literal months of development time.
Focus on solving your business problem, import the rest.
Summary
- We get paid to solve business problems, not to write code.
- If the problem is not directly related to your business, there is probably a library for that — use it.
- The time cost of writing a piece of code is not just time spent tapping on a keyboard.
- Things that you should definitely use a library for: Money / Time / Logging / Threads / Parsing
- Focus on solving your business problem, import the rest.
About the author
Hi, I’m Doogal, I’m a Tech Lead that has spent a bunch of years learning software engineering from several very talented people. When I started mentoring engineers I found there was a lot of knowledge that I was taking for granted that I think should be passed on. These stories are my way of trying to pay it forward.
