How to Get a Job 35 Million Times a Day Using RabbitMQ

This talk was held on Wednesday, March 27, 2013.

The goal of Indeed’s aggregation engine is to find and retrieve every job in the world, as quickly and accurately as possible. As we described in our previous tech talk, we strive to build products that are simple, fast, comprehensive, and relevant. The world’s most comprehensive job search site is fueled by the more than 35 million job postings we process every day, which we deliver to jobseekers within minutes of discovery.

Our original aggregation architecture was implemented using standard patterns. Our growth required levels of scalability, performance, and resilience this architecture simply could not handle. In a case study of scaling for the web, we discuss how we tackled this problem. We cover the issues we saw with our original architecture, how we analyzed our options to guide a solution, how we used RabbitMQ as a key component in the new architecture, and benchmarks to evaluate how successful we were.

Audio Description

The following video includes a descriptive audio track for this talk.

Transcripts

• Basic transcript (includes audio information only)
• Descriptive transcript (includes audio and visual information)

Speakers

Cameron Davison is a software engineer.

Ketan Gangatirkar is a VP of engineering at Indeed.

Originally published at Indeed Engineering Blog.