Building Thumbtack’s Data Infrastructure

By: Nate Kupp

Image for post
Image for post
Photo by Joshua Sortino on Unsplash

Part 1: Organizing Chaos

Image for post
Image for post

Legacy Event Data Pipeline

@events.capture('type = "home page/view" AND browser_engine')
def on_homepage_view(event):
on_view(event)
@events.capture(type='category group page/close lightbox')
@events.capture(type='landing page/close lightbox')
@events.capture(type='service/close lightbox')
@events.capture(type='request form/close lightbox')
def on_interaction_event(event):
on_view(event)

Planning: Our Roadmap for Success

Scalability

Decoupling

Ease of access

Image for post
Image for post

Batch Processing

AWS Deployment

Image for post
Image for post

Distributed SQL Infrastructure

Dashboarding & BI Tools

Data Ingest Pipelines

Relational ETL

Event ETL

Time to Put Our Plan Into Play

Thumbtack Engineering

From the Engineering team at Thumbtack

Thumbtack Engineering

Written by

We're the builders behind Thumbtack - an online marketplace that matches customers with local professionals to accomplish their projects.

Thumbtack Engineering

Stories from the Engineering team at Thumbtack

Thumbtack Engineering

Written by

We're the builders behind Thumbtack - an online marketplace that matches customers with local professionals to accomplish their projects.

Thumbtack Engineering

Stories from the Engineering team at Thumbtack

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store