Building With Node.js
Plus The Journey From Arts to Engineering And More With Jeff Harrell, Director of Engineering At PayPal
Jeff Harrell is the Director of Engineering of the Business and Payments division at PayPal. PayPal’s web applications were spread across two technology stacks: their original stack, C++ using XSL for templating, and its successor, Java with JSP templates. Jeff helped migrate away from them and over to a Node.js based stack.
Can you please describe your path to what you’re doing now?
My career path happened in a roundabout way. I started out in fine art – specifically with pencil, charcoal, and oil paint – and studied that for years. Outside of class, I was always the “computer guy”, but since I was raised in a tech-capable household I never thought much of it.
As I progressed, my interests started to converge and I found myself fascinated by the graphic design side of art. This lead to working for a company where I designed their print catalog, website, and interactive applications on CDs.
My designs began outpacing what could be created with WYSIWYG tools, so I often found myself diving into the code to further enable my creations. Often, this meant I was spending more time in code than I was on the design, but I was always up for the challenge and taught myself about programming, algorithms, networks, and databases.
It wasn’t long before I found myself really enjoying the engineering side of things, so I doubled down and joined PayPal as a UI engineer. At first my involvement was limited to building our payments applications, but it soon branched out as I became involved with mobile products and our overall technical strategy. Around the three year mark, I felt I had accomplished all I could at PayPal, so I left and joined the TVUI team at Netflix.
The TVUI team was a very small and densely talented team focused on building their streaming applications. This taught me a lot about experimentation and building resilient applications and I was there for just about two years when PayPal contacted me about rejoining to rebuild their engineering teams and technology.
We heard you’re running some of the largest Node apps in production at PayPal. Can you please give us a bit of background about the old Paypal technology stack and why you decided to migrate everything to JavaScript and Node.js?
PayPal’s web applications were spread across two technology stacks: Our original stack, C++ using XSL for templating, and its successor, Java with JSP templates. When the time came, there were multiple reasons why we decided to migrate away from them and over to a node-based, open source stack:
- Our new approach was all about rapid iteration. We wanted to be able to experiment and bring those experiences to our customers as fast as possible. Neither the C++ or Java stacks were built for this, both having large build times for even the smallest changes. Node was one of the many environments out there which catered to this type of fast development.
- JavaScript is the programming language of the internet. We faced an interesting problem in our engineering teams. Since they were built around C++ and Java skillsets, we found they often didn’t have a strong understanding of the modern web’s capabilities. Client-side rendering, web sockets, asynchronous programming, and even XHR didn’t mean much, thus they weren’t used and our customer facing applications began to get dated. JavaScript for the server is still early in large scale adoption, but we found a strong correlation between people who were familiar with JavaScript in any form, and those familiar with the web and how it works.
- Open source is our new mantra. Our predecessor technology stacks were built to provide good value, but were done in a proprietary manner. This resulted in weeks of training for a new hire to learn our technologies. We were very intentional when building our Node.js stack and from day one intended for it to be as open source friendly as possible. If the industry used a technology or technique, we wanted to use it. A good example here is our decision to use express. Both Node.js and its community are still young though, so there were gaps we faced where we felt there was a better way to do it. In this case, we were excited to contribute that code back to the community and help grow it instead of writing something just for ourselves. Krakenjs and a few of our other modules came from this.
PayPal has 13,000 employees and you wanted to change the technology stack. That’s crazy, but you did it! What approach would you recommend to other companies that are thinking to do the same?
Start small. Seriously. I’m a big believer in leading by example. We have millions of lines of code running across three different technology stacks. Tackling all of that at once and moving everything to Node.js would have been a massive undertaking and likely result in failure.
We opted to start small, solving for only a single application and a single development team. This team would either gloriously succeed, in which case everyone would want to do what they were doing, or it would fail fast, in which case it would have been a good attempt that we learned from. Luckily it ended up being the former.
When I tell people this they tend to think of building an internal application in a new technology to “prove it out”. This comes to the second part of my advice: choose something that will be a challenge and cause to you flex your muscles to prove it will work.
In our case, we chose one of our most trafficked customer applications. It was high risk, but high reward for both our engineers and our customers. Solving the use cases in this application proved that the rest of the applications could be built too, and helped us gain a lot of adoption very quickly afterwards.
Considering most of the PayPal engineers were Java devs, how did everyone get up to speed with JavaScript and Node?
When it came time for mass adoption we invested in a training program. This was an intensive class composed of three days of functional JavaScript training by Doug Crockford followed by three days of server side JavaScript and node.js. The people going into the class were a mixed bag: we seeded each class with some of our UI engineering team, who were highly familiar with JavaScript, and those from our application teams, who often hadn’t coded a line of JavaScript. After the class was over, I personally asked each team to work on their app for a sprint or two, have it reviewed, and – after they learned from that review’s feedback – toss the code and start over. The expectation here was that the class would only give them a foundation in JavaScript, but to really understand they had to build an app and make some mistakes. I wanted to make sure our new applications were high quality, so I was willing to have them take a few weeks to polish their skills using a real application and then start over rather than building our next generation of applications on the developers’ first official lines of JavaScript.
What are the biggest challenges a company should expect from developing and running a large scale Node app? How did you overcome them?
As it turns out our biggest problem wasn’t technical, but cultural. We already knew how to build, deploy, monitor, and scale applications, so it wasn’t really that different with Node.js. We had our missteps along the way, e.g. our monitoring infrastructure being tied to our JVM implementation, but those were easy to find and fix. What we didn’t know as a company was how to look outside of what we were doing internally and embrace modern technology.
One thing I value in myself is the ability to identify and challenge things which don’t feel right. This happened a lot along our journey and we often found ourselves using an internal solution instead of its equivalent open source alternative. In some cases you win, others you don’t. The key part is knowing which battles to stick with and Node.js was one of these.
Javascript itself has an old stigma attached to it which slotted it in the realm of a “toy language”. To consider using it and Node.js for our server application was appalling to many at first, but we were strong in our belief that it was the right technology to invest in. I’m a firm believer in taking action rather than talking about it, so while I was having all of the conversations required to get buy in, we were also working on building out our first application to back our words with actual numbers. The decision to use a new technology instead of an established one is subjective, but data is not, and when you can show the application can be built faster, have a smaller server footprint, and deliver results to our customers faster it’s hard to disagree it’s not the right direction to take.
Did you get any important benefits from Javascript and Node that you didn’t consider initially?
Our initial reason for moving to a new technology stack was rapid iterations. We wanted a technology stack that would help us bring better experiences to our customers faster, and Node.js provided great developer agility to our teams. There were other benefits which were realized, of course, e.g. needing less server capacity, faster response times, involvement in the open source node community; but the one which we hadn’t considered was developer joy.
Let me explain: Early on in the process of moving to Node.js we had a lot of resistance from our engineering teams. I didn’t blame them; we were changing how they worked. After the training started, their attitudes started to change little by little. People came out of their classes eager to try something new.
As teams started building their applications, I would bump into them in the hallway and ask how it was going. This typically resulted in excited answers and the occasional question or two, but in a few cases, I actually had engineers hug me and thank me for bringing a modern technology to the company. No joke.
For once, PayPal was using a near cutting-edge technology and we were ahead of the curve. Our engineering teams were not the only ones energized by this though; the external engineering community was also interested in learning from what PayPal was doing and this helped us gain even more traction internally.
Developer joy wasn’t the initial reason we set out to go with JavaScript and node, but in retrospect I think it’s the most important. Our developers are now extremely passionate and excited about what they are working on. This is resulting in better products and experiences for our customers.
How did the dynamics between front-end developers and back-end developers change after adopting JavaScript?
Previously, we had separate UI (HTML, CSS, and JavaScript) and app (Java) engineering teams. There was crossover between the teams, but it was not regular and usually resulted in two engineers doing the job of one. With Node.js, this artificial boundary goes away and those two teams are now combined into a single application engineering team. Our UI engineers, while proficient in JavaScript, still needed to learn how to develop in a large scale, multi-user server environment. Conversely, our app engineers needed to learn to code for the browser, an often times scarier environment than the server given its fragmentation. Both have made great strides in this direction.
There are exceptions, of course, where some people specialize in an area like accessibility, but the average team is fully end to end at this point.
How do you learn new things? Do you attend conferences, learn through building products at your job or work on your own projects outside your job?
I try to always be some level of hands on. I authored some of the original lines of our Krakenjs code, have touched nearly every early application to help guide our teams, and always have some type of personal side project going on. Typically these involve areas I’d like to explore like OpenStack, deployments with Docker, and machine learning. This helps me keep in touch with both my engineering teams and the latest technology.
What are the things you did in your early days as a developer that were most beneficial for your career?
Having a job while I was in college which used what I was learning was huge for getting me off on the right step. Not only did I get to practice what I learned, but it often put me ahead of the curve in what I knew. When it came time to graduate, I already had professional experience, which was extremely helpful.
In what types of environments do full stack devs thrive and in what environments do they not?
Full stack developers thrive in environments which allow them to be autonomous and in control. This allows them to choose the right tool for the job and make the decisions required to scale their app the way it needs to be. We’re working to further empower our engineers at PayPal exactly for these reasons.
Full stack developers aren’t going to thrive in a rigid environment which controls how they work. Organization separation of your UI, application, and Operations teams is a good example of this.
Do you have any advice to share with full stack developers that are early in their career?
If you’re early on the absolutely best thing you can do is start coding, understand what you don’t know, and research the answers. Pick a library, e. g. jQuery, Express, etc., and read the code to understand really how it works. Build an application and put it on Github. If you have little professional experience apply for an internship with a company that inspires you – we’re always accepting interns at PayPal! If you’re interested you can find out more on our careers site.
Perhaps most important, look at every opportunity you can to learn something new as a challenge you’ll accept.
Written by Octav Druta. A version of this post first appeared on the Talentbuddy blog.
Talentbuddy is the simplest way to learn full stack web development with Javascript. The program is focused on teaching the unique challenges of building real-time web applications with fast-growing technologies (Javascript, Ember.js, Node.js, Express.js, MongoDB) Find out more.
