Polling using Task Queues on App Engine
Polling is the most traditional way to get data from a data source that produces a stream of events and updates. I’m not a big fan, it comes with numerous drawbacks - I’d rather have data being pushed via webhooks instead - but there are times it’s inevitable and the only option available.
This is a quick write up showing one way to implement polling using task queues on applications running on App Engine.
The idea is very simple: task chaining. The example below implements it using the deferred library from the Python runtime:
The code snippet is very straightforward, I will try to explain it in detail. Everything starts on start_polling (1–3). It basically gets the current state ( the resource/entity being watched) and enqueues a new task using the deferred.defer function. This function will create a new task which will be handled by poll_for_changes on a separate request. The state object will be serialized, it must be picklable.
The “polling loop” is implemented in the poll_for_changes (5–12) function. It gets the new state (using get_state again), compares it with the given state verifying whether it changed or not. If positive it notifies the interested parties. Then, after checking if it should keep polling, it enqueues a new task. The secret lies in the _countdown parameter which specifies the time in seconds into the future that the task should run. In this example we are checking for state changes every 15 seconds until should_keep_polling returns true.
And that’s it! Hopefully this example will be useful for someone out there.
IMPORTANT: Be aware that polling for long periods will increase your instance hours considerably and consequently your costs. Use it wisely.
