Learn about the background, challenges, and future of Airbnb’s distributed scheduling and queueing system.
Introduction
Asynchronous background jobs can often dramatically improve the scalability of web applications by moving time-consuming, resource-intensive tasks to the background. These tasks are often prone to failures, and retrying mechanisms often make it even more expensive to operate applications with such jobs. Having a background queue helps the web servers handle incoming web requests promptly, and reduces the likelihood of performance issues that occur when requests become backlogged.
At Airbnb, we built a job scheduling system called Dynein for very critical use cases. Since its introduction…
When designing services and the interactions among them, consistency and scalability are two of the most important qualities that we aim for. An important aspect of achieving those qualities is the serialization protocol used to communicate between services. Here at Airbnb, we use Apache Thrift for this very purpose. Thrift is an excellent serialization protocol that’s simple yet powerful. Think JSON, but binary and much faster, with schemas and generated model classes that are helpful for productivity. Thrift has been an essential tool for scaling Airbnb’s architecture to achieve our mission: creating a world where everyone belongs. As more and…
Introduction
Page 3
… information only becomes valuable when it is understood, not just when it’s made available.
Page 3
A heavy bank of figures is grievously wearisome to the eye, and the popular mind is incapable of drawing any useful lessons from it as of extracting sunbeams from cucumbers.
Page 7
We analyze information not for its own sake but so we can make informed decisions.
Andy Fang
Cloud Infrastructure @Airbnb. Ex @PoloDataClub, @GeorgiaTech. I make AWS wrappers. https://andyfang.me.
