Building With Node.js At LinkedIn
Deepank Gupta is a Senior Software Engineer at LinkedIn, a platform that connects the world’s professionals to make them more productive and successful. His short answer to the question “What do you do?” is: “Writing code to solve difficult problems.”
Can you please tell us a little bit about what you do now and the path you took to get here?
I am the technical lead for our frontend API servers written in Node.js and also lead the feed for the Linkedin Mobile application.
I got introduced to programming by chance during a summer camp in middle school. I had signed up for another class but somehow ended up building games in C. Since then I knew I wanted to be a software engineer. After my undergrad in Computer Science, I started working at Google, I worked on the News, Mobile and Search Ads teams.
I joined Linkedin at a time when mobile was a small quirky team within a large company. The rules for doing mobile were different and it has had a rapid growth over the last few years that I have been here. I got introduced to Node.js at LinkedIn. My first project at Linkedin was to work on sponsored updates in the feed; which was really interesting since nobody at that time knew anything about making money on mobile.
Previously LinkedIn Mobile was powered by Ruby on Rails, but now it is one of the larger Node.js apps in production. Can you please share with us the reasons why you made the switch?
Our Ruby on Rails app was a synchronous app in which clients used to make several calls for a single page. All the calls happened sequentially with every thread handling a single request. Consequently the system was bursting at its seams.
With node.js we wanted to move to an asynchronous evented system. It also enabled us to move to a model where the client makes a single request for a page. The code was simplified (in fact the first iteration of the app in Node.js was codenamed “Simplify”) and we moved to stateless servers.
What were the performance gains, in numbers, after migrating to Node.js?
Since the migration happened before my time, I do not have numbers. From what I have heard from senior folks in the company, the reduction in the number of machines used to host our service was greater than 10:1.
What are the biggest challenges a company should expect from developing and running a large scale Node app? How did you overcome them?
For any large scale app, monitoring is critical. We have built monitoring and alerting at every layer:
- Operating system level stats such as socket counts, cpu usage, memory usage
- Process level stats
- QPS, latency, HTTP status codes for various outbound routes and downstream services
- Application level counters
Another problem with Node is the lack of good profiling tools. Getting setup with flame-graphs and analyzing performance problems is not straight-forward and requires lot of time and energy. We are building some automated tools to catch performance regressions but there is a lot of work required in this area.
Besides this, debugging is harder as well since the original stack trace is lost. At Linkedin, one of the engineers came up with trycatch which he later published into npm which can generate long stack traces.
Lastly, we have also published some other tools like Sepia which can be used as a record/playback mechanism for integration tests in Node.js
Did you get any important benefits from Node.js that you didn’t consider initially?
Writing code in Node.js also encourages you to think in certain ways. Node provides an evented system with a single process which executes multiple requests. It has no concept of thread-locals forcing you to make your code stateless and modular. Although Node has added support for domains; it is mostly untested for high qps services.
Another unexpected benefit with Node is its package manager. Npm is light, straightforward and easy to use. The node community has focussed on building small utilities which come together to provide a vibrant ecosystem for developers.
What performance tips can you share with us that we can all use when building Node apps?
One of the single most important thing to do is to never keep your event loop busy. We employed a monitoring trick to keep track of this called “pacemaker”. What pacemaker does is that it puts an event into the queue with the timestamp at which it was created. When the callback is executed, another timestamp is taken. The difference between the two timestamps is the time on average for a request to be processed off the event queue. Here is the sample code to generate a time series of data:
Pacemaker delay is the additional penalty your service is paying for every request. If you start seeing spikes on graphs with increase in max pacemaker delays, that generally points to some requests doing cpu-intensive work in your Node app.
What are some secrets that make you highly productive with Node.js? Can you share some personal examples?
We embrace functional programming and use utility libraries like lodash. I discovered very early on that if I find myself writing nested for-if statements, generally I am doing it wrong. A lot of code can be written in simple map/filter/reduce type statements. Using a functional style of coding makes the code concise and readable.
To give you some examples: here is the code to display the profile insight based on the common occupations between your and the profile you are viewing:
and getting the participants except the viewer in a group discussion is like this:
I got introduced to functional programming with Node.js fairly early. It took me some time to wrap my head around functions being passed in as parameters to functions and higher order functions returning functions. But once you start thinking of functions as first class citizen, it becomes very easy.
How do you avoid the “callback hell” in Node.js?
We employed a bunch of strategies when we started to deal with “callback hell”. First we used Async library, later we moved onto Step. But now we have settled down onto promises. We use promises heavily in our codebase. Even though it has a steeper learning curve compared to step, the payoff is simpler and easy to read codebase. The power of using promises is especially evident with a complicated workflow of downstream requests to serve a response back to the user. For unit-testing promises, we use the popular chai-as-promised module.
What were the most important things you did as a beginner developer that had the highest impact on your career?
The first time I submitted a 200 line code change for review, I received back 150 comments on it. While most of them were related to coding conventions, some other comments were more fundamental. After discussing with my mentor, he said to me: “You do not understand what you have written. First go and understand it.”
I think really understanding each and every word in your code makes a big difference. For instance, in any project, lets assume you need to add a new call to another service. For this, you would go and look up an example inside your codebase on how to call a service. Now you have two choices: either copy paste and modify the code to suit your needs OR really understand the example code. If you do the latter, you would notice that instead of calling native http module it is using the Request module. At this time, you should ask yourself: Why does it need to use that, what are the benefits; and what do various parameters mean? Really understanding your code helps you become a better developer.
What do you think a mentor can bring to the table that no other learning experience can? Do you have any personal examples?
I often looked up to people whose work I admired and asked them to pair-program with me. Observing and emulating habits of good programmers can help you succeed.
What little habits make you a better software engineer?
One of my mentors once said to me: “A great engineer is a like a good boy scout. They will leave the camp-site cleaner than when they came.” I try to follow that mantra by refactoring and adding tests anytime I go around changing an existing piece of code.
Besides coding, having a bigger picture about the product, getting to know user numbers instills a product sense which is immensely useful for an engineer.
What distinguishes a good software engineer from a great one?
A great software engineer has these two qualities:
- Given the same problem, a great software engineer will be able to write it in simpler and lesser lines of code than a good engineer.
- Secondly they are detail oriented. These engineers are never afraid of debugging or bug fixing. Often times they will create small scripts to automate common workflows after having tested their code hundreds of times manually. They will consider most edge cases and have healthy skepticism towards assumptions inside frameworks and libraries they use.
How do you learn new things? Do you attend conferences, learn through building products at your job or work on your own projects outside your job?
Going to conferences certainly helps. Besides, I love to read books to get new ideas.
To learn a new language/framework, the only way to really learn is to build something. It could either be something completely new or fixing bugs in existing systems. Without getting your hands dirty with some project, it is hard to keep yourself motivated and grok the new concepts.
Do you have any advice to share with full stack developers that are early in their career?
I would reiterate what I said earlier about understanding what you are writing. Building a deep fundamental understanding is essential to your career. Secondly, build good habits. Bringing the discipline of following coding conventions, writing README files, making your code testable and writing unit tests makes you an effective engineer.
Lastly, but most importantly, seek out a mentor. Find someone whose work you admire and reach out to them.
And one last piece of advice: if you are looking for interesting problems to solve and grow your career, Linkedin is a great place to be and we are looking for people. Please visit LinkedIn’s careers page.