From Coding Bootcamp Graduate to Building Distributed Databases

Practical advice for early career software engineers.

I get a lot of questions from people who’ve completed a software engineering bootcamp and are looking for advice on finding a job or advancing their career. I think enough has been written about the former, but very little information is out there about the latter.

I’ve discussed how I made the jump from doing normal backend CRUD / API work to more interesting roles that involve building custom infrastructure and distributed storage from scratch before, but I wanted to sit down and lay out what I think are the most salient points.

My target audience is early career software engineers who are trying to figure out how to advance themselves and control the arc of their career trajectory.

Most of my advice falls under one of three broad categories:

1. Visibility, Reputation, and Social Capital
2. Continued Education
3. Taking Responsibility for your Career Trajectory

Visibility, Reputation, and Social Capital

One of the most important assets you have as a software engineer is your reputation and social capital. There are two aspects to your reputation:

1. Your reputation within the company you work for
2. Your reputation within your industry

Your Reputation Within the Company You Work For

The most important way in which you build your reputation within your company is by shipping successfully. Always be shipping. While this may seem like obvious advice, keep in mind that there are a lot of software engineers, teams, and even companies that struggle to ship changes on a consistent basis.

At least within the early portion of your career, you will be judged almost entirely by your ability to ship reliable and bug-free code in a consistent manner. Nothing else matters more than this. Building a strong reputation as someone who gets things done will open up all kinds of opportunities for you. Promotions will come faster and you’ll get preference for interesting and important projects. At some point you may find that your superiors stop assigning you work altogether as they realize that left to your own devices you can be trusted to ship the right things at the right time and communicate upwards any obstacles that you’re encountering. This is what you should strive towards. Your managers should feel like you’re so proactive that they don’t have to do any work to manage you. This will naturally lead them to micromanage you less and focus their energies elsewhere which will make your day to day much more enjoyable.

A core tenet of building your internal reputation is how you make people feel. If you ship code on a regular schedule, people will begin to feel confident that your part of the work will be delivered on time, and automatically begin turning to you for their most important work. Why wait to deliver to create that feeling? By sending daily or weekly updates, you can spark the feeling of safety in your manager and colleagues much more frequently than if you simply deliver on the big day.

One piece of advice you can incorporate immediately is to strive to structure your work so that you’re always demonstrating progress. Once a project has been assigned to you, try and imagine all of the stakeholders involved (your boss, the product/project manager, marketing, sales, etc) as being in a constant state of anxiety not only about when the task will be finished, but whether it will get finished at all. I know that this seems like an absurd mental model to work with, but it’s a useful one.

Send daily or weekly updates before you’re asked, communicate regularly and often, and strive to structure your work into milestones that are meaningful and useful. This will help with your own planning and deadline estimation, but will also reduce the anxiety that all of your stakeholders feel so that they leave you alone to get your job done. While this isn’t always possible, in an ideal world you want to break large projects down into smaller ones where each inherent milestone is intrinsically useful, even if it doesn’t fulfill all the requirements yet, while also not doing a bunch of throw-away work.

For example, I worked on a project recently that involved building a system that could rewrite large volumes of data in real time. I knew that the entire system would probably take me 6 weeks to build in a way that met all of our requirements for reliability, availability, and scale, but we shipped the first working version that did something useful in two weeks. The first version ran as a single instance on the largest EC2 we could provision, which is basically a worst case scenario for a production system, but that initial prototype proved that the fundamental design was sound, and I spent four weeks after that “productionizing” the system so that it could run on many different machines concurrently and scaled arbitrarily with the volume of input data.

Most of the code from the two-week prototype was re-used and all in all I lost less than a day’s worth of work to “throw away” code that was replaced. This approach had the benefit of demonstrating early on in the process to all of the stakeholders (including myself!) that the design was sound and that we could economically process the amount of data in question.

Second, adopt the mindset that shipping is the most important goal. You need to own every aspect of getting your work out the door. Sometimes you may need to do other people’s jobs to get there, and that’s fine. If your product manager is bad at coming up with specs then you need to write them yourself, lock yourself in a room with them, and not leave until you’ve reached consensus. If another team is in your critical path, but they have other priorities, then you need to jump into their code-base and get it done, even if it’s written in a language you don’t know and uses technologies you’re not familiar with.

Early in my career at Uber I was put in charge of doing all the backend work for a new cross-cutting feature that had multiple consumers: an external API for user-authorized 3rd parties as well as a mobile API for the Uber client app to integrate with. The feature was user-facing and involved integrations with myriad other companies via Uber’s developer platform which meant that I was dealing with an overwhelming and often contradictory amount of input from marketing, design, legal, product, and other engineering teams. On top of that, the product manager in charge would reply to any detail oriented question with: “I don’t care, whatever you think is best”, but once the engineering work was done he would develop an opinion that often required going back and redoing work that was supposed to already be finished.

One day after growing frustrated from being asked to reimplement the same feature for a third time, I sat down and wrote a 13 page doc with every outstanding issue that I needed a decision on in order to be able to progress with the engineering work. I then refused to write any more code until all of the issues were resolved if not unambiguously, at least with some high degree of confidence.

It took me about a week to chase down all of the relevant stakeholders and get them to commit to something, but once that was done I was able to spend the next few months building out all the backend work more or less uninterrupted and with very few major changes. Was this my responsibility as junior engineer on the team? No, not really, but that project solidified my reputation as an engineer who got things done which meant that I was usually assigned the most “important” projects to work on.

Another good way to build your reputation within your company is to be willing to jump into things you don’t understand and get your hands dirty with things no one else wants to touch. Every company has a problematic service that no one understands anymore, a part of the code base that no one wants to touch. Issues like this should be seen as an opportunity to roll your sleeves up, dive deep into a thorny and hairy problem that no one else wants to touch, and come out the other side with the reputation as someone who is willing and capable of solving the hard problems.

Six months into my tenure on the Developer Platform team at Uber we needed to migrate an older service. The old service was simply called “Login” and while it did power Uber’s login functionality at the time, it had become a dumping ground for all kinds of loosely related functionality. For example, it also contained all of Uber’s developer platform data that included things like which 3rd party applications were allowed to use which external API scopes, OAuth bearer tokens, API configuration, etc.

This service had various limitations, like the fact that it was backed by MySQL which with Uber’s infrastructure at the time meant that it would become “read only” whenever we performed a datacenter failover, and these limitations were limiting our team’s ability to grow and extend Uber’s developer platform. In addition, Uber’s security and identity teams were planning on blowing this service apart into multiple component services which meant that all the aspects of the Login service that powered my teams functionality needed to be moved somewhere else.

No one else on the team wanted to be responsible for this project because migrations aren’t sexy and the Login service’s code was a bit of a mess. I volunteered for the project, however, because I knew it would be a good learning experience. I’d get to learn a new programming language (while many legacy services at Uber were written in Python, anything new was written in Go), as well as a new storage technology (Cassandra was our storage system of choice for services that needed to be “active-active”).

I spent a week creating a migration plan that included things like: estimating storage requirements, outlining how we’d do a zero downtime migration by dual writing, etc, and got it approved. After that, I disappeared down a rabbit hole for 4 months knocking the migration out piece by piece. The work was challenging and often frustrating, but I learned a lot and when I came out the other side I had developed a ton of good will for myself within the broader org. For months many new features and lucrative deals had been blocked by this migration, and having my name tied to unlocking all of that work boosted my reputation significantly.

Just remember that it’s on you to communicate the impact and details of your work. Always summarize the impact and details of your work, especially if it’s an “unsexy” project, in an email or presentation to ensure it gets the visibility it deserves because it is easy for “maintenance” work like this to go unnoticed unless you go out of your way to draw attention to it.

Don’t be afraid to share your work. Get over any insecurities or uncomfortableness you have about putting your work out there. Your work won’t highlight itself. If you accomplish something interesting or impactful, especially if it was unsexy but important maintenance work, share it in an email. Include graphs to drive home the impact.

Your Reputation Within Your Industry

All of the points above apply to your reputation outside of your company as well, although the way you build up your reputation is different. The easiest way to get started with building your “industry” reputation is by writing blog posts. Similarly, speaking at conferences can be equally valuable and often anything that would make a valuable blog post would also make for an interesting conference talk.

Try and keep these interesting, one good blog post with an interesting story or non-obvious insight is more valuable than 10 run of the mill “here’s a tutorial on how to use this new framework” posts.

As a rule of thumb, any topic that passes any of the following litmus tests is a good candidate for a blog post or conference talk:

1. The most senior engineer you know would find the information and subject matter compelling.
2. People who read the post think: “Wow, I can’t believe they managed to <something really hard or difficult to figure out>. I really wish we had someone like that who could help us with <thorny problem>”.
3. The subject matter is deep in a way that readers are unlikely to be able to obtain the same information anywhere else.

For example, this blog post I wrote about debugging a painful performance regression in production passes tests #1 and #2.

Continued Education

The software engineering industry changes at a blistering pace. This is both a blessing and curse. It’s a blessing because there are always new and exciting things to learn, but it’s also a curse because it’s easy to fall behind and feel overwhelmed.

My advice for this section centers around two key insights:

1. Recognize that not all knowledge is created equal. Learning C.S fundamentals is timeless and will pay dividends for the rest of your career, whereas knowledge about the latest Javascript framework comes bundled with a ticking time-bomb the moment it settles into your brain. Before you know it the knowledge will be obsolete and the time you spent cultivating it wasted. For this reason it’s important that you orient your self-education around “deep” source of knowledge like books and papers, not blog posts.
2. The second insight is that some things can be learned from self-study, but other things can only be learned from doing. For that reason, especially early in your career, it’s important that you “do” as much as possible and the best mechanism for achieving this is by working through a large number of high leverage side projects.

Read Books (and papers), not Blog Posts

This point is especially important for engineers coming from non-traditional backgrounds like bootcamps. The best way to learn C.S fundamentals or to gain deep expertise in a specific topic is to read a textbook, not a blog post.

Blog posts are usually shallow and often the authors don’t have a deep understanding of the topic they’re writing about themselves. When I had just finished my programming bootcamp I was confused about the concepts of “processes” and “threads” and how they related to the concepts of “concurrency” and “parallelism”. If you try and learn about these topics by reading random articles on the internet, you’re likely to walk away with the shallow understanding that threads (like processes) are “units of execution,” but that threads run in a “shared memory space.” This is enough information to answer lightning round C.S trivia questions in an interview setting, but it won’t help you solve any concrete problems.

On the other hand, if you buy an operating systems text book you’ll learn that one of the primary purposes of your operating system is to virtualize the underlying hardware that it’s running on and that processes are the primary abstraction for virtualizing the CPU. You’ll learn what a process is, what operating system APIs are used to create them, what data structures the O.S uses to track them, and what algorithms are used to schedule them. You’ll learn that threads are important not only because they enable a single program to parallelize work across multiple CPUs and execution contexts, but because they also provide a mechanism for dealing with asynchronous APIs like those of the file system and network and how those concepts are related to the topics of concurrency and parallelism, synchronization, and much more. This rich context provides you with a useful mental framework that you can leverage to solve real problems.

If you’re coming from a non-traditional background (i.e. you don’t have a C.S degree) https://teachyourselfcs.com/ is a good place to start for a list of books to read. Learning the fundamentals is the best place to start because they’re timeless and will pay dividends for the duration of your entire career.

In terms of my own continuing education, I recognized early in my career that most software engineering jobs are boring. You can make good money “plumbing” APIs and databases together with whatever the current flavor of the month framework is, but that gets old really fast. I was working at Uber at the time and when I looked around the company for more interesting work, the teams that were doing the most compelling work were only interested in engineers with strong infrastructure and computer science fundamentals.

I realized that if I was going to get where I wanted to be in my career I’d have to fill in all the gaps in my computer science education. I accomplished this through a variety of projects and classes:

1. nand2tetris is a great place to start. The book walks you through the process of building a virtual computer, starting with nand gates and moving up the abstraction layer by building an ALU, then a CPU, then an assembler, a compiler, and finally an “operating system”. While the book drops off in quality and details in the later third or so, all of the chapters up to and including the compiler are an invaluable way to learn and understand how computers really work.
2. I took every class offered by Bradfield School of Computer Science.

I still focus on my own continuing education. Sometimes that means reading research papers in my areas of interest: databases and storage. Other times it means digging into the open source code of interesting projects, for example try spelunking through the source code of Redis or PostGres, you’ll be surprised at how readable they are. Finally, I still indulge myself in learning things the old fashioned way by just reading a textbook cover to cover.

The text book I read most recently was: “The Garbage Collection Handbook: The Art of Automatic Memory Management“. While I had learned a little about garbage collection algorithms from writing high-performance software in garbage collected languages like Go, studying the fundamental algorithms in a thorough manner has paid large dividends already in terms of anticipating how the garbage collector will interact with my applications.

For example, while I was very familiar with how to write software that minimized garbage collection overhead in Go I had no experience doing so in Java. However, after having read that text book, understanding the garbage collection behavior of any programming language is just a matter of pattern matching which garbage collection techniques and algorithms are employed by that particular language.

That’s a bit of a gross oversimplification, but you’ll be surprised how big of a difference even just a little intuition about what’s going on under the hood can make in your day to day development experience.

Lastly, whenever you learn something new, do your best to understand it as deeply as possible. For example, the first time I built a service that was backed by Cassandra, I spent a few days reading the documentation, learning about the storage engine, and reading detailed blog posts of other engineers sharing their experiences migrating existing systems to Cassandra. This allowed me to avoid a lot of the common pitfalls people usually run into when they first start using Cassandra, and to structure my application in a way that played to Cassandra’s strengths instead of its weaknesses.

Similarly, the first time I had to use Protobuf in a project, I didn’t just learn the basic syntax and APIs for marshaling/unmarshalling. I read the docs thoroughly, including the encoding / wire protocol until I was confident that if necessary I could implement my own protobuf library. This only took a few extra hours of my time and has already paid dividends multiple times in the last two years alone. While working on M3DB I developed a custom compression format for delta-encoding streams of protobuf messages, and more recently I wrote a streaming, zero-allocation library for Protobuf decoding that I’m currently using for a variety of projects at work.

It is far too tempting in this profession to reach for things that are easy, to use technologies that we don’t understand, and to be sheltered from complexity by a mountain of abstractions. It is easy to throw your hands up in the air and declare: “I don’t have time to understand all of this, I have things I need to get done today! Besides, I shouldn’t have to understand how…”

Fight this urge whenever possible. Know your tools. Accept the abstractions, but only once you’ve studied their implementation and understand their limitations. You’ll never have enough time to do this for every tool that you use, but if you do it for even a small fraction of them you’ll reap massive benefits.

For example, as a freshly-minted bootcamp grad, you probably don’t have the luxury of going and obtaining a comprehensive C.S education before you get your first job, but once you have that first job you should invest in a comprehensive C.S education. Similarly, if you joined a new team at work that uses Java, you don’t need to spend two weeks studying the JVM before you write any Java code, but at some point you should go learn a little bit about the JVM.

Do lots of side-projects

Side projects are like practice. The more you practice, the better you will get. It doesn’t matter what, as long as you’re learning either a new technical skill (technology, technique, principal) or soft skill (design, product development, etc).

The side projects I’ve worked on over the years have ranged from things as simple as writing a regular expression engine to learn how they work, to as advanced as spending 6 months working with two of my friends to develop and ship a usable product.

The goal of these side projects is to condense years of experience and skills into the shortest amount of time. If you compare two software engineers who both have 2 years of experience in industry, but one has worked on several medium to large complexity side projects in that time and the other has not, chances are the engineer with the side projects will be a lot better simply by the nature of having done more work, and a much wider variety of it.

The side projects also serve as an opportunity to develop skill sets that you wish to acquire, but that you wouldn’t otherwise get to develop at your day job. My first software engineering job was mostly API/CRUD work, but I spent a lot of my free time studying C.S fundamentals (primarily by taking evening classes at Bradfield), and that prepared me to make the transition into infrastructure work.

I interviewed with a different team at Uber that was building an open source metrics database and as part of my interview process they asked me to investigate a performance issue they were having with their commitlog reading performance. Even though I only had 3 days to work on the problem, I was able to jump in and get working immediately because I already had all the prerequisite knowledge. I knew what kind of internal APIs to expect from the classes I’d taken and textbooks I’d read about database internals, I already knew how to profile the application to see why it was slow, and I knew enough from my computer architecture classes that this operation should be about 10x faster than it was which helped me narrow down the list of potential issues dramatically. If I hadn’t spent the last 18 months studying and preparing, I wouldn’t have had the slightest clue on how to solve that problem because none of the work at my day job would have prepared me for it.

Take responsibility for your career trajectory

At some point in your career you’ll have to start taking responsibility for your career trajectory. The key insight is that you should make decisions about what projects to work on and which teams and companies to join strategically, not tactically. Think long term.

In the grand scheme of things, a 10% to 30% difference in compensation for a few years is small in the grand scheme of your career. After my first two years at Uber I could have switched companies for an easy 30% to 40% boost in my compensation, but instead I made a strategic decision to position myself on a team where I was paid “below market”, but where I was developing skills that I would leverage to achieve a 3x jump in my compensation later.

In particular, there are two areas that you should pay particular attention to:

1. Finding good mentors
2. Developing a valuable and unique skill set

Do whatever it takes to find good mentors

Nothing will accelerate your growth faster than spending all day working with other very good engineers. The moment you start to feel like you’re not learning from the engineers around you is the moment you should start looking for a new team.

Keep in mind that when I use the term “mentor” I mean engineers that you work with everyday, participate in design discussions with, and who review your code on a regular basis. This is very different from a “mentor” that you solicit career advice from once a month over coffee.

There is no exact formula for finding good mentors. The key is having the humility to recognize those people when you meet them, and to be willing to make sacrifices to be around them.

If you work at a large company, this process shouldn’t be too difficult. Start asking around and try to identify a high performing team with very good engineers doing impactful work, and then do whatever you can to start working with them. Offer to help out on their projects on your own time, or if you’ve been at the company long enough and have built up some social capital, try and transfer to that team. Your manager may not be happy about it, but hiring is hard and the manager of the team that you want to join will likely extend the necessary political cover to make the transfer happen if they think they can get a coachable and motivated junior engineer with a proven track record “for cheap”.

In my case, I was working at Uber and had been working on the same team for 18 months and was stagnating technically. I told my manager that I was going to start looking for a new team and then I spoke with the managers of 15 different teams at the company. The team that seemed to have the most skilled engineers doing the most technically interesting and impactful work was located in New York, a city I had no interest in moving to, but I flew to NY on my own dime and spent 3 days working with their team on a small task. After meeting the team and seeing how skilled they were, I knew I had to move to New York even though I didn’t know anyone there. In hindsight, that decision completely altered the trajectory of my career and I have no doubt that I wouldn’t be where I am today if I hadn’t prioritized technical mentorship over everything else that was going on in my life at the time.

Obviously dropping everything and moving across the country isn’t an option for everyone, but if your goal is to get dramatically better then finding mentors should be one of your top priorities.

It’s also worth emphasizing that I interviewed with 15 different teams. That’s a lot of opportunities to investigate! For context, after I graduated from my bootcamp I had 3 onsite interviews before landing at Uber. If you assume that opportunities follow some kind of power law distribution such that some opportunities are significantly more valuable than others, then going through this exercise whenever possible will likely lead to much better outcomes. If you then factor into the equation that these types of decisions compound over the lifetime of your career it starts to seem grossly negligent to not take this exercise seriously everytime you’re gearing up for a major change.

Develop a Valuable and Unique Skill Set

At some point in your career you’ll want to start “specializing” in some way to set yourself apart from others. This is important because having a valuable and unique skill set is the key to making more money (or better work life balance if that’s what you value more), job security, autonomy, and more impactful and satisfying work.

Money isn’t everything, but it’s a proxy for value, and having a valuable skill set gives you the freedom to live your life the way you want. If someone is willing to pay you half a million dollars a year for your skillset, then you’ll find someone willing to pay you less in exchange for a 20–30 hour work week. Similarly if your skill set is valuable you’ll be able to find a company that will let you work 100% remote if that’s something you desire.

In my opinion, the skillsets worth optimizing for are those that are highly valued by the large tech firms. The reason I think this is because the gap between what the Googles and Facebooks of the world can pay their engineers, and what everybody else can pay has grown into a large chasm. For context, Facebook generates $634,694 in profit per employee and that’s an average! Imagine how much value the most skilled and critical employees are adding to Facebook’s bottom line, and how much Facebook would be willing to pay to obtain and retain them. It is simply not possible for other companies to match the compensation provided by the large tech firms because they can’t leverage their employees to the same degree.

Unfortunately, I think that the compensation gap that I just described is going to continue to grow. Over the last 15 years the industry has seen massive consolidation into a small number of larger players and while it’s possible this process will be disrupted in the future, I wouldn’t bet against it.

If you need even more evidence that I’m right, consider this: Even if you don’t want to work for a big tech firm, if you have a skillset that Facebook or Google is willing to pay a lot of money for, you’ll be able to find an interesting startup that wants to hire you, but the inverse is not necessarily true.

To drive this point home, spend some time on levels.fyi and contemplate how much some of the larger firms are playing their senior (achievable within 5 years of experience, often less) engineers. Annual compensation in the $300–500k range is very attainable for engineers who develop the prerequisite skill sets.

Consider this, let’s say you have two job offers on the table:

1. $500,000 / year total compensation from Facebook
2. $150,000 / year total compensation from a startup + 1% equity and an estimated 10% chance of making it to a billion dollar valuation (in practice this probability is much lower!)

Assuming that 8 years is a fairly reasonable time horizon for a no-name startup to reach a billion dollar valuation (something that almost never happens) then in that time period the Facebook job would have generated 4 million dollars in income whereas the startup generated just shy of 11 million dollars. 10x the risk for double the return, does that seem like a good deal to you?

Now consider the fact that the actual risk is an order of magnitude higher, and I haven’t factored in dilution, promotions, additional Facebook stock grants, and the fact that in the real world large tech firms stock tends to appreciate massively.

I’ve already beat this horse to death completely, but I’m going to make one final point about why working for a large tech firm is the best choice you can make career-wise regardless of how you look at it. Imagine you spent 8 years bouncing around the Google’s, Facebook, and Amazon’s of the world working on highly scalable distributed systems. One day an upstart company that has been quietly disrupting the industry in the background for the last 8 years comes along and is getting ready to IPO. They’ve done an amazing job so far but they’re really starting to hit their stride in terms of growth and scale and they’re starting to nip at the heels of the bigger players in the space. How much do you think they’ll pay you to steal you away from Facebook 6 months before their IPO? The answer is a lot. Based on my experiences from talking to various people who’ve had these conversations, they’ll likely pay you enough that post-IPO you’ll be making equivalent or more than a mid-level engineer who had the good fortune of joining them 2–3 years earlier in their trajectory when the company’s trajectory was much more uncertain.

The point is, the gap between what the large tech companies pay and what everyone else pays is so large (and growing!) that it makes sense to develop a skill set those companies will find particularly valuable.

I personally ended up settling on distributed storage for two reasons:

1. Unrelated to the value of the skill set, I find it extremely interesting. Building and maintaining large distributed databases requires a fairly deep understanding of all the computer science fundamentals: computer architecture, operating systems, networking, algorithms, data structures, etc. It’s all there in one problem space.
2. The skillset is highly in demand among the large tech companies because the number of people capable of solving distributed storage problems at their scale is limited, the number of people who have a proven track record of doing so is even smaller, and the skill-set is inherently high leverage which means that the large tech firms can leverage a small number of individual’s skill-sets to generate huge per-employee profits that make even highly compensated FAANG engineer’s compensation seem reasonable by comparison.

Distributed storage is just one example (and the one I’m most familiar with), but generally speaking I think that most of the interesting work right now is in the space of large scale systems of various forms. The reason for this is because it is the one common thread that connects all of the large tech firms together. They’re all operating at a massive scale that is completely disproportionate to the number of employees they have and they have a strong need for highly-leverageable employees that can build the distributed systems that will sustain their business.

Whatever niche you settle on, try and spend at least two years working on a meaty problem in that space. Spending at least two years on the same “problem” is important because it’s how you develop depth. During that two year period the system you’re developing will undergo major revisions and you’ll be forced to grapple with the consequences of decisions you made a long time ago. You’ll build depth that other companies will pay a premium for, and you’ll be immersed in the problem space for long enough to begin to grapple with not only the localized version of the problem that you’re solving, but also the macro version of the problem that the industry is striving to solve in general. If you’re lucky, you might even develop a valuable enough insight and depth of expertise to strike out on your own.

In summary, take responsibility for your own education and your own career. Don’t day trade with your career. Invest in it strategically and you might surprise yourself with how far you’re able to go in a relatively short period of time.