RAD — How We Replicate Terabytes of Data Around the World Every Day

Indeed Engineering
Apr 12, 2016 · 1 min read

This talk was held on Wednesday, March 30, 2016.

IndeedEng is back! After a hiatus in 2015, we’re back in a brand new office ready to tell some new stories. Indeed is growing and solving compelling problems with interesting technology, and we want to engage with the Austin tech community to share what we’ve learned.

At Indeed, we use massive amounts of data to build our products and services. At first, we relied on rsync to distribute these data to our servers. This rsync system lasted for ten years before we started to encounter scaling challenges. So we built a new system on top of BitTorrent to improve latency, reliability, and throughput. Today, terabytes of data flow around the world every day between our servers. In this talk we describe what we needed, what we created, and the lessons we learned building a system at this scale.

We’re updating the accompanying video and slide deck. Please check back later.

Speakers

Jason Koppe is a senior software engineer at Indeed.

Julie Scully is an engineering director at Indeed.

Originally published at Indeed Engineering Blog.