How Pinterest is preparing for its future in the cloud — or on its own
Pinterest tends to fly under the radar of the tech industry, likely because the male-dominated tech sector doesn’t spend a whole lot of time pinning. But an awful lot of people — just over 76 million monthly users in July, according to Comscore — send roughly 120,000 requests per second to Pinterest’s infrastructure and have pinned over 50 billion things on the site, which even if you’re a unicorn is a pretty big undertaking.
Raj Patel is the man responsible for keeping Pinterest up and running, although he insists that’s really everybody’s job in the infrastructure engineering department at the five-year-old company. Patel, head of cloud engineering for Pinterest, is someone we’re really excited about having on stage at Structure 2015 this November in San Francisco.
I recently got a chance to sit down with him for a wide-ranging interview on how a modern consumer web startup plans, builds, and scales its computing needs. Patel has been through a few generations of infrastructure evolution while working for tech giants such as Yahoo and Cisco, and is now taking on a new challenge in the form of a consumer-facing social networking company run entirely on the public cloud. That experience provided some unique perspective on scale and engineering focus that, assuming Pinterest continues to grow, could mean a move into owning and operating its own infrastructure.
Here are a few topics we explored in our conversation, but expect to hear much more at Structure:
Public cloud forever? — Right now, Pinterest relies entirely on AWS for its needs. But when Patel arrived at Pinterest in March 2014, and he said Pinterest CEO Ben Silbermann asked him to build a two-to-three year vision of Pinterest’s engineering needs.
Patel said he’s looking at a range of different options when planning for the company’s future, including private clouds, hybrid approaches, and even Pinterest-controlled datacenters. He didn’t want to get into the specific factors that would cause Pinterest to move some workloads away from the public cloud, and was pretty clear that nothing was imminent, but he drew an analogy to Home Depot: if you’re a small-to-medium-sized building contractor, you can probably get away with buying most or all of your supplies at a one-size-fits-all retailer like Home Depot. But if your business gets big enough, at some point you have to start to control your own supply chain.
Even cloud-native companies need focus — Although Pinterest has been an Amazon Web Services customer since day one, as it expanded its engineering team (around 200 total, with around 30 in infrastructure engineering) and started to hire more and more people with different backgrounds, Patel realized he had to develop internal practices around operating entirely in the cloud; yes, even a company like Pinterest had to go through the process of training its engineers to think cloud first.
How has the company done this? Pinterest’s engineering organization is made up of small teams with a lot of autonomy, for one. And Patel said its engineers are challenged to build applications with two things in mind: understanding what they can’t control, and assuming that the infrastructure underlying that app is ephemeral.
Open wins — Patel acknowledged the debt that the startup community owes to open-source software and the Open Compute Project, movements that have made it so much cheaper and easier to build large web businesses from scratch with a fraction of the investment that was required around the time the web started to become a force. Later this year, Pinterest plans to release a few internally developed projects to the open source community, something we’ll definitely make sure to touch on at Structure.
“We have a humble culture at Pinterest,” Patel said. “We have gotten to where we are by leveraging open source.”
Hold the containers — Despite the surge in interest behind container technologies as the Next Big Thing in software development, Pinterest is going to watch from the sidelines for a while longer as the technology develops.
“Containers need more maturing before they are relevant at Pinterest,” he said. However, Patel is hopeful that containers will force public cloud software providers to work more closely together for customers that are running on different platforms.
Catch Patel’s appearance at Structure 2015 on November 18th at the Julia Morgan Ballroom in San Francisco. You can find more information here, and you can buy your tickets here.
