Background Tasks in Flask(SQS, Celery, Postgresql)
Your application may have a requirement to do long-running tasks which require more processing over some data such as sending marketing campaigns to all users in the system, sending bulk notifications to greet users, sending sales report to management at the end of the day, etc.
There will be two types of processes
- Application process where flask app runs
- Worker Process which executes background jobs assigned
Interprocess communication happens by means of a message queue. It holds required metadata like input parameters and methods to execute in a queue.
Celery is used to perform a complex task in the background for the flask app. It will be used with Amazon SQS which is a message queue to send necessary data to another process(workers) that will run the task in the background. However, there are many message queues available as service: Kafka, RabitMQ, SQS.
Furthermore, to execute workers or background jobs it may require database binding for getting information about records. We will bind PostgreSQL as a database for execution.
Installation
$ pip install celery[sqs]Configuration
