Meet the Adobe I/O Team: Senior Computer Scientist Dan McWeeney on Serverless, Auto-Scaling, and Laziness
Senior computer scientist Dan McWeeney, at Adobe for more than 10 years, has just joined the team that develops Adobe’s serverless platform, Adobe I/O Runtime. His specialities include software architecture, community management, and open source projects. In fact, he wrote the first open source project built on the SAP ecosystem, one of the world’s largest and most important markets for software providers.
We caught up with Dan to find out how his experience at Colgate Palmolive paved the path to the projects he works on at Adobe, why he believes in the power of abstractions, and what’s behind the three axes of laziness, impatience, and hubris.
What do you do for Adobe I/O, and what’s your career path at Adobe been like?
I work on the core code that underpins our serverless system, called Apache OpenWhisk. I’ve been working on some of our logging integration, getting OpenWhisk running on Mesos, and now I’ve started work on our auto-scaling solution to make sure we are always running and running quickly.
How did you get started in the industry?
I took a weird route to get to Adobe, I suppose. I graduated university with a degree in computer engineering, which is a mix of electrical engineering and computer science. But about three quarters of the way through I realized I was more interested in the higher-level software abstractions than the gates and adders I was studying.
After graduation I got a job at Colgate Palmolive building financial planning systems, which — for someone that took no finance or accounting classes in college — was ‘different.’ However, an engineering degree gives you a good foundation in math, and as long as someone from finance could show me the math, I basically didn’t care if we were selling toothpaste in Tijuana or dog food in Detroit.
How does your previous experience help you in your new role?
While at Colgate Palmolive, I wrote the first version of what became the first real open source project in the SAP ecosystem, Saplink. The tool allowed SAP developers to easily share code across systems. I know it sounds nuts but at that time code for an SAP system was represented in many database tables spread across the system. Saplink allowed developers a way to gather all those things and install them in another system. Think of it like package management instead of just source code. I got to present this solution in a huge ‘demo competition’ in front of a few thousand developers in both Las Vegas and Amsterdam.
The Saplink project gave me a lot of useful experience running an open source project and figuring out how to weigh all the contributions we were getting, while also keeping the central code stable and safe. The way the system was originally designed really allowed for contributions to come in as plugins, and the core development team was able to focus on keeping those abstractions stable, while the community writ large was able to work on the edges. It really showed me how powerful solid abstractions are for working with developers.
What’s so awesome about serverless?
A totally unrelated system from a decade ago taught me how powerful abstractions allow people to build amazing things. Serverless is no different. The promise of serverless is to allow developers to focus on where they can add value to their companies and products and get out of the business of having to worry about every detail from the OS up. In college I focused on the design of chips and overall computer architecture — something no developer usually worries about today. It’s a given that you have a stable way to allocate and read from memory. It’s a given that floating point math happens in a consistent way.
In the near future, developers will have a consistent way to reason about very high scale, and they will have a stable way to get their code running in the cloud without all the hassle of standing up VMs and worrying about security groups.
What are you currently most excited about in your work?
I’m kind of hoping the most interesting thing I’m going to work on at Adobe is whatever is next. Right now it’s our auto-scaling subsystem. A lifetime of work could be spent on getting a perfect auto-scaling solution built and deployed. However, no one has a lifetime to spare, so as with nearly all engineering exercises, it’s a question of first building just enough, then learning from that and moving forward to build something better. It’s very often more art than science, and I’m really excited to be working on it with such a great team. Getting great input from other engineers really keeps the process moving forward and can be a really great check on trying to boil the ocean.
Why do you often refer to yourself as a ‘lazy developer’?
The quote basically goes, ‘There are three great virtues of a programmer: laziness, impatience and hubris’. It’s from the second edition of Larry Wall’s book Programming Perl. I really love this quote. I always strive to work along these three axes.
It has most recently come up in the highly pedantic area of build systems. I hate doing things more than once and basically said, ‘I’m too stupid to be trusted to always do all these steps right, so I’m just going to automate the whole thing.’ Which is probably true, but also, I’m just too lazy to even consider looking at the nice documentation someone wrote on how to do those things. I’d rather spend all that time just building a system, so I never have to go and do any of it again. This also dovetails with the second virtue: I’m much too impatient to have to sit and watch something to make sure it works. Some magic code in the sky should just handle all that for me and roll things back and keep everything moving forward as fast as possible if nothing is wrong. Finally, this also touches on never ever ever wanting to be the person that breaks something. I want a system that prevents anyone from realizing that I made a silly error. I will do a huge amount of work testing and building guardrails to ensure everyone thinks the code I write is perfect.
What’s the best piece of career advice you’ve ever received?
I think the best piece of advice I ever got was this: when a problem seems totally intractable, get up and do something else for a while.
At work I usually try to have two or three ‘important’ things happening at once. This allows me to switch from one task to another when I get really bogged down. I work remotely from most of the team, as I’m based in NYC, so I can sometimes spiral into the weeds as there isn’t another developer to chat with at 9am Eastern Time, so having some other task to hop to really helps me out. Very often taking a break from something will give you much greater insight when you get back to it.