Plus The Journey From Arts to Engineering And More With Jeff Harrell, Director of Engineering At PayPal
Jeff Harrell is the Director of Engineering of the Business and Payments division at PayPal. PayPal’s web applications were spread across two technology stacks: their original stack, C++ using XSL for templating, and its successor, Java with JSP templates. Jeff helped migrate away from them and over to a Node.js based stack.
Can you please describe your path to what you’re doing now?
My career path happened in a roundabout way. I started out in fine art – specifically with pencil, charcoal, and oil paint – and studied that for years. Outside of class, I was always the “computer guy”, but since I was raised in a tech-capable household I never thought much of it.
As I progressed, my interests started to converge and I found myself fascinated by the graphic design side of art. This lead to working for a company where I designed their print catalog, website, and interactive applications on CDs.
My designs began outpacing what could be created with WYSIWYG tools, so I often found myself diving into the code to further enable my creations. Often, this meant I was spending more time in code than I was on the design, but I was always up for the challenge and taught myself about programming, algorithms, networks, and databases.
It wasn’t long before I found myself really enjoying the engineering side of things, so I doubled down and joined PayPal as a UI engineer. At first my involvement was limited to building our payments applications, but it soon branched out as I became involved with mobile products and our overall technical strategy. Around the three year mark, I felt I had accomplished all I could at PayPal, so I left and joined the TVUI team at Netflix.
The TVUI team was a very small and densely talented team focused on building their streaming applications. This taught me a lot about experimentation and building resilient applications and I was there for just about two years when PayPal contacted me about rejoining to rebuild their engineering teams and technology.
PayPal’s web applications were spread across two technology stacks: Our original stack, C++ using XSL for templating, and its successor, Java with JSP templates. When the time came, there were multiple reasons why we decided to migrate away from them and over to a node-based, open source stack:
- Our new approach was all about rapid iteration. We wanted to be able to experiment and bring those experiences to our customers as fast as possible. Neither the C++ or Java stacks were built for this, both having large build times for even the smallest changes. Node was one of the many environments out there which catered to this type of fast development.
- Open source is our new mantra. Our predecessor technology stacks were built to provide good value, but were done in a proprietary manner. This resulted in weeks of training for a new hire to learn our technologies. We were very intentional when building our Node.js stack and from day one intended for it to be as open source friendly as possible. If the industry used a technology or technique, we wanted to use it. A good example here is our decision to use express. Both Node.js and its community are still young though, so there were gaps we faced where we felt there was a better way to do it. In this case, we were excited to contribute that code back to the community and help grow it instead of writing something just for ourselves. Krakenjs and a few of our other modules came from this.
PayPal has 13,000 employees and you wanted to change the technology stack. That’s crazy, but you did it! What approach would you recommend to other companies that are thinking to do the same?
Start small. Seriously. I’m a big believer in leading by example. We have millions of lines of code running across three different technology stacks. Tackling all of that at once and moving everything to Node.js would have been a massive undertaking and likely result in failure.
We opted to start small, solving for only a single application and a single development team. This team would either gloriously succeed, in which case everyone would want to do what they were doing, or it would fail fast, in which case it would have been a good attempt that we learned from. Luckily it ended up being the former.
When I tell people this they tend to think of building an internal application in a new technology to “prove it out”. This comes to the second part of my advice: choose something that will be a challenge and cause to you flex your muscles to prove it will work.
In our case, we chose one of our most trafficked customer applications. It was high risk, but high reward for both our engineers and our customers. Solving the use cases in this application proved that the rest of the applications could be built too, and helped us gain a lot of adoption very quickly afterwards.
What are the biggest challenges a company should expect from developing and running a large scale Node app? How did you overcome them?
As it turns out our biggest problem wasn’t technical, but cultural. We already knew how to build, deploy, monitor, and scale applications, so it wasn’t really that different with Node.js. We had our missteps along the way, e.g. our monitoring infrastructure being tied to our JVM implementation, but those were easy to find and fix. What we didn’t know as a company was how to look outside of what we were doing internally and embrace modern technology.
One thing I value in myself is the ability to identify and challenge things which don’t feel right. This happened a lot along our journey and we often found ourselves using an internal solution instead of its equivalent open source alternative. In some cases you win, others you don’t. The key part is knowing which battles to stick with and Node.js was one of these.
Our initial reason for moving to a new technology stack was rapid iterations. We wanted a technology stack that would help us bring better experiences to our customers faster, and Node.js provided great developer agility to our teams. There were other benefits which were realized, of course, e.g. needing less server capacity, faster response times, involvement in the open source node community; but the one which we hadn’t considered was developer joy.
Let me explain: Early on in the process of moving to Node.js we had a lot of resistance from our engineering teams. I didn’t blame them; we were changing how they worked. After the training started, their attitudes started to change little by little. People came out of their classes eager to try something new.
As teams started building their applications, I would bump into them in the hallway and ask how it was going. This typically resulted in excited answers and the occasional question or two, but in a few cases, I actually had engineers hug me and thank me for bringing a modern technology to the company. No joke.
For once, PayPal was using a near cutting-edge technology and we were ahead of the curve. Our engineering teams were not the only ones energized by this though; the external engineering community was also interested in learning from what PayPal was doing and this helped us gain even more traction internally.
There are exceptions, of course, where some people specialize in an area like accessibility, but the average team is fully end to end at this point.
How do you learn new things? Do you attend conferences, learn through building products at your job or work on your own projects outside your job?
I try to always be some level of hands on. I authored some of the original lines of our Krakenjs code, have touched nearly every early application to help guide our teams, and always have some type of personal side project going on. Typically these involve areas I’d like to explore like OpenStack, deployments with Docker, and machine learning. This helps me keep in touch with both my engineering teams and the latest technology.
What are the things you did in your early days as a developer that were most beneficial for your career?
Having a job while I was in college which used what I was learning was huge for getting me off on the right step. Not only did I get to practice what I learned, but it often put me ahead of the curve in what I knew. When it came time to graduate, I already had professional experience, which was extremely helpful.
In what types of environments do full stack devs thrive and in what environments do they not?
Full stack developers thrive in environments which allow them to be autonomous and in control. This allows them to choose the right tool for the job and make the decisions required to scale their app the way it needs to be. We’re working to further empower our engineers at PayPal exactly for these reasons.
Full stack developers aren’t going to thrive in a rigid environment which controls how they work. Organization separation of your UI, application, and Operations teams is a good example of this.
Do you have any advice to share with full stack developers that are early in their career?
If you’re early on the absolutely best thing you can do is start coding, understand what you don’t know, and research the answers. Pick a library, e. g. jQuery, Express, etc., and read the code to understand really how it works. Build an application and put it on Github. If you have little professional experience apply for an internship with a company that inspires you – we’re always accepting interns at PayPal! If you’re interested you can find out more on our careers site.
Perhaps most important, look at every opportunity you can to learn something new as a challenge you’ll accept.