A Simple Way To Become a Better Software Engineer
The world’s best experts explain how they solved the most difficult problems; for free.
There are quite a lot of resources that teach how to solve algorithmic problems, and in general, how to ace technical interviews. A lot of people and companies teach how to get hired. But it seems there are considerably fewer people and companies that focus on what happens after you get hired.
Doing work and delivering software that solves problems is the best way to grow, but oftentimes this growth has a limit. Not every project has to handle several thousand requests per second, process terabytes of data, or provide the best performance possible. But even in that case, there is a simple (though not easy) way to learn a thing or two about difficult and challenging computing problems: reading white papers.
The first time I saw a white paper was on my team lead’s desk. It was printed on A4 lists and stapled together in the top left corner. I gave it a brief look and thought, “Hmm, maybe I should try to read some white paper too.” Quick googling, and here it is — “The Google File System.” Fifteen pages, no complicated mathematical models, familiar terms such as “metadata,” “replica,” “high availability” — should be a good white paper to start with.
I started reading and was surprised to find that it was written in a normal language, which was easy to understand and follow. The text is dense, packed with terms that are not always familiar, but it’s not an “academic” text full of formal constructs that is hard to get through.
However, quite soon, I started to feel overwhelmed with details. In the middle of section 2, my mind was wandering; I had to constantly focus on the text to find out later that I completely forgot what the paragraph is about. I realized that I can’t read the white paper the same way as posts on social networks or technical articles; it requires a more serious approach.
So I printed out the white paper and stapled it in the top left corner, took pen and notebook, booked a meeting for half an hour with no guests invited, and started working with the paper. How many sections are there? There are nine sections: introduction, design overview, system interactions, …, conclusions. Ok, the first ten pages contain a definition of a problem, an overview of the architecture, and then details how various challenges are addressed. The remaining pages are about running the system in production, conclusion, and further work. Now I can start reading…
In allocated half an hour I was able to almost get to section 3. To avoid felling of “stack overflow” I was taking notes of every section of the document, highlighting the most important parts in the printout and separately writing down things to learn or revisit: POSIX API, Linux vnode layer, B-tree, data corruption detection with checksumming,
O_APPEND mode and so on.
How to read technical books
Personal experience of getting a real understanding, not an illusion of it
It took me four of such meetings to finish reading. And it was a very unusual experience of deep diving into a problem and approaching it systemically. It’s very different from the usual StackOverflow routine, where you google something specific like
BeanNotOfRequiredTypeException, click on the first link, and then scroll to the accepted answer without even looking at the question to see if there is some quick fix that could solve the problem. No, here the authors first describe what it is they are working on. It’s a distributed file system, and it has to handle X, Y, and Z well. Then they outline the assumptions, then propose an architecture and then go on explaining how it works… If it was possible to film it, then the Discovery channel could have created a documentary based on this whitepaper.
In addition to the systemic approach to problem-solving, the reading also gave me a few examples of technical patterns (we have a single master, but it only works with metadata and keeps all of it in memory), challenged me to look up certain things (
O_APPEND mode or copy-on-write techniques), and also a list of references if there is an interest to dig deeper into specific topics.
My next project was based on druid, and while reading its docs, I found references to white papers that inspired the design. Since I already knew the drill, I printed them, stapled in the top left corner, and started to read, one by one. It took quite a significant amount of time, but the benefits are very well worth it. Knowing the fundamentals of the system makes it way easier to use it. It allows designing solutions to problems instead of looking for hacks to make something work.
As a backend software engineer, working with distributed systems, I found these whitepapers to be very insightful:
- Google File System — distributed file system. IIUC GFS inspired HDFS — the file system for Hadoop — a framework used for big data solutions.
- BigTable — a distributed key-value storage, inspired HBase, also a component of Hadoop.
- MapReduce — a programming model for processing large datasets (i.e., big data)
- Dremel — a system for querying read-only nested data, “capable of running aggregation queries over trillion-row tables in seconds.” It also describes what is “columnar storage format.”
- Spanner — a distributed database that challenges CAP theorem
- Borg — cluster management system that inspired Kubernetes
- Chubby — a distributed lock service (i.e., distributed mutex)
- Goods — an approach to organizing structured datasets at scale
- Ubiq — streaming data processing system.
The list is by no means exhaustive. There are many other excellent papers (and there are also many white papers that look much more like marketing materials) where the world’s best experts explain the problem and how they have solved it. It’s a very good resource to get better at software engineering.