EvalAI: Processing submissions using RabbitMQ

Taranjeet Singh
Jul 23, 2017 · 2 min read

EvalAI is a platform which helps AI researchers, students and data scientists to collaborate and participate in various AI challenges. The platform hosts various challenges, corresponding to which participants make their submission.

Submission

A submission is a prediction made against a Challenge Phase. It is processed using the evaluation script of a challenge, sometimes involving heavy computations.

Processing Submission

A submission can be processed either asynchronously or synchronously.

Processing a submission synchronously will defer the response for the request until the processing is not complete. The processing involves heavy computation, so it may be a bottleneck when the number of concurrent submissions is made.

This bottleneck can be easily handled if the submission messages are processed in an asynchronous way. All the submission made can be queued, which can be processed by workers in a round-robin fashion. This can be easily horizontally scaled by adding more workers if the number of submission messages increases.

RabbitMQ: The Saviour

To process messages in an asynchronous way, we at EvalAI are using RabbitMQ, a Queue-based system.

Whenever a submission is made, the response is sent immediately after saving a submission instance and a message is queued in an asynchronous way. This message is then picked up by the worker and processed further.

Situations wherein the number of submission increases and the workers are already processing at their maximum capacity can be handled easily as every message is eventually queued up. The message persists in the queue until it is not picked up and acknowledged by the worker. This way, we can easily handle and process a large number of submission even at peak times.

For processing messages, a python worker listens on the queue. The worker picks up the message from the queue, processes the submission and saves its status in the database.

This way, we are using RabbitMQ at EvalAI to process submissions made against a challenge.


Originally published at taranjeet.github.io on July 23, 2017.

Taranjeet Singh

Written by

Prev @paytm, @gradeupapp, @project_cloudcv. Open Source Developer. Quizzer. https://taranjeet.cc

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade