Open-sourcing PinLater: An asynchronous job execution system

Zhiyuan Zhang | Pinterest engineer, Discovery

PinLater is an asynchronous job execution system we built as a replacement for our original solution based on Pyres. We’ve been using PinLater in production for over two years, and it’s one of our most mission critical systems. Usage of PinLater spans from core actions like Pinning, image thumbnail generation and when Pinners follow other Pinners, boards or topics, to larger batch operations like email delivery, push notifications and visual discovery workflows. We now operate more than 10 different clusters all on Amazon EC2. In aggregate, there are more than 500 job queues processing north of six million jobs per minute. Today we’re excited to announce PinLater as our latest open-source release.

Before building PinLater, we evaluated a number of existing open-source options, but none met all the requirements we had, including:

• Fault tolerant reliable execution with at least once guarantee.
• Simple service/API that abstracts storage details away from clients and supports pluggable storage backends and clients/jobs written in any programming language.
• Separation of job execution from scheduling/queuing so the two can scale independently, and the system can support different types of jobs.
• Fully horizontally scalable and built on top of reliable and operationally well understood open-source components such as MySQL.
• Priorities, delayed execution and flexible retry policies that can be specified per job at enqueue time.
• Fine grained visibility into job processing, and the ability to query job state or trace actions on individual jobs.
• A fundamentally simple design that makes the system easy to reason about and debug.

Under the hood

PinLater has three components:

1. A stateless Thrift service to manage job submission and scheduling
2. A storage backend to store the jobs and state
3. Worker pools to execute the jobs

The thrift service provides three core actions: enqueue, dequeue and ACK. When a job is enqueued, it is sent to the storage backend to keep track. Worker pools continuously dequeue jobs and execute them, replying with a positive or negative ACK depending on whether the execution succeeded or failed. The service layer is responsible for all aspects of job scheduling including priorities, retries and execution at a future time. For more details on all of this, please read our previous blog post.

Recent improvements

Over the past year, we’ve made a number of improvements to the system. We’ve made tweaks to achieve 5X more throughput on the MySQL backend, which allows us to do over 2,000 enqueues per second with a single i2.2xl MySQL EC2 instance. Since MySQL offers higher durability with reliable replication, this change allows us to move all our workloads to MySQL and deprecate the use of the Redis backend. We’ve also built a checkpointing feature to support long running jobs and complex workflows. We added a new dashboard that’s a one stop shop to view job status, debug job failures, lookup any job by ID or body text and a number of other features that delight our users.

This open-source release includes the service itself as well as a reference Java worker framework implementation. We hope PinLater will be helpful to others who have similar requirements, and can’t wait to see what contributions and improvements the community can bring to the system. In the future, we’ll continue to release other parts of our overall PinLater stack. Give it a whirl and reach out to pinlater-users@googlegroups.com if you have any questions.