Discover Pinterest Tech Talk: Big Data and Apache Mesos

Last week we hosted a Tech Talk on Pinterest’s logging infrastructure and Apache Mesos, with engineers Roger Wang of Pinterest, Connor Doyle of Mesosphere, and Bernardo Gomez Palacio of Guavus. You can find a recap of the discussion below, and slides on SlideShare.

Logging Infrastructure at Pinterest (Roger Wang)

Roger joined Pinterest as a software engineer on the data engineering team last year, after working in engineering at Amazon, Google, and Microsoft. Over the years, he’s had the opportunity to work on persistence services, search features and infrastructure. For the talk, Roger shared insights into Pinterest’s new high performance logging agent, Singer, which plays a crucial role in our data architecture. Singer is currently in production to ship logs from application hosts to Kafka, and we plan to open-source it soon.

Deploying Docker Containers at Scale with Mesos & Marathon (Connor Doyle)

Connor joined Mesosphere as an engineer after most recently working at Gensler and One Orange Software. His current work focuses on Marathon — an Apache Mesos framework, which was the topic of his presentation. Connor started with a high level overview of Mesos before diving into new features of Marathon, including deploying Docker containers at scale.

Scaling Big Data with Hadoop & Mesos (Bernardo Gomez Palacio)

Bernardo joined Guavus from HashGo and IGN Entertainment. With a previous focus on backend APIs and infrastructure, he’s now working closer to big data. His presentation focused on using Mesos as a resource management layer for Spark and Hadoop clusters. He also shared case studies from personal experiences managing the Mesos clusters for these two frameworks, as well as comparisons between Mesos with Yarn.

Interested in keeping an eye out for future events? Follow us on Facebook and LinkedIn .

Krishna Gade is a software engineer at Pinterest.