A to Z of Google Cloud Platform a personal selection — Q — Queues
It’s been a while since I’ve talked about App Engine and I just wanted to touch briefly on its Task queue feature .
Task queues are designed to allow your App Engine application to carry out small jobs or tasks as they are known in the background. Using Task Queues your application defines tasks, adds them to a queue, and the queue is then used to process them in aggregate.
App Engine’s standard environment provides two types of task queues Push & Pull Queues. Below are descriptions of these taken straight from the docs:
- Push queues process tasks based on the processing rate configured in the queue definition. App Engine Standard environment automatically scales processing capacity to match your queue configuration and processing volume, and also deletes tasks after processing. Push queues are the default.
- Pull queues allow a task consumer (either your application or code external to your application) to lease tasks at a specific time for processing within a specific timeframe. Pull queues give you more control over when tasks are processed, and also allow you to integrate your application with non-App-Engine code using the Task Queue REST API. When using pull queues, your application needs to handle scaling of instances based on processing volume, and also needs to delete tasks after processing.
A task as defined in the docs is a unit of work to be performed by the application. Each task is an object of the Task class. Each Task object contains an endpoint (with a request handler for the task and an optional data payload that parameterises the task).
An example application where we can use a push queue could be where your application accepts sign ups for various newsletters. The data payload for this task consists of the name, email address and newsletters to be subscribed to . The webhook might live at /app_worker/register_subscriber and contain a function that adds the subscription details to Datastore . The app can create a separate task for each subscriber it receives.
The table below provides a comparison of the queue types hopefully in an easily digestible format
I’d say the size of your payload and whether the queue needs to be accessible outside of App Engine would be the first two decision points to consider when trying to decide wether a pull or push queue is the one you need.