Monitor your Celery workers using Flower

Introduction to Celery:

Celery is a distributed task queue that lets you run tasks from your python script asynchronously. It is a task management system that you can use to distribute tasks across different machines or threads. Celery saves our waiting time for a process to get completed while running a program we can send the time-consuming processes or functions to celery which will wait till the process is done and you could continue with the rest of the task.

for more details about celery refer: https://medium.com/featurepreneur/get-to-know-celery-e930fe5cca10

What are Clusters..??

Celery allows you to have a task queue and can schedule and process tasks in real-time. This task queue is monitored by workers which constantly look for new work to perform. A Cluster is many Workers running in parallel. When several workers are working parallelly there are possibilities of getting confused about the tasks that are running and their results. To avoid such confusion these clusters must be monitored.

Flower

A flower is a web-based tool for monitoring and administrating Celery clusters. It has a UI that monitors all the workers on celery. It gives clear statistics on the active tasks, processed tasks showing whether the task was successful or not, and also tells the load average of the tasks. It also maintains the name of the task, the arguments, the result, and the time taken to complete the task.

Installing Flower

You can install flower using the command

pip install flower

The development version can be installed with

pip install https://github.com/mher/flower/zipball/master#egg=flower

Monitoring with Flower

Flower Usage

The actual way of accessing Flower is

celery [celery args] flower [flower args]

Various arguments are available that you could use depending on your requirement.

Celery args: https://docs.celeryq.dev/en/stable/reference/cli.html#celery

Flower args: https://flower.readthedocs.io/en/latest/config.html#options

Usage Example

You will be able to see all about the celery clusters once you launch the flower server. You can do it at any specified port other than the default port ‘5555’. You can launch the server using the command

celery flower --port= [port_number]

After running this command you can go to a browser and check the localhost with the initialized port number, you will be able to see the Flower monitoring tool. An example is given below.

command: celery flower --port=5432
url    : http://localhost:5432/

you can also launch it using the docker

docker run -p 5555:5555 mher/flower

Once you launch flower, you can now run your program. As you run you will get immediate responses corresponding to the task you pass to celery. In the dashboard, you will be able to see the connection status and the number of active activities, processed, failed, passed, and retrieved. If you move to tasks you will have all information about the job, its name, arguments, result, runtime, etc.

Conclusion

When there is a cluster of workers will be chaos if they are not monitored, with the help of flowers you can quickly know every action that is being done in celery.