Don’t mess with the celery while it running a task
Some time ago we decided to move part of our logic to celery. Several tasks was wrote and in one of them bash script was called, and this task never applied correct: first half of script was executed and then halted. I found that script running in processes, and can’t understand why it just stoped. Then I set some breakpoints in bash script and realised, that script stops at
service celeryd stop
Hm, interesting. And there was no task result in my broker. My colleague told me to look at task that called. I opened that task and found some code after calling a bash script.
Aha, thats it! Celery can’t stop while task isn’t finished.
I removed that code and it works!
P.S. Also there was a problem with calling a task. Don’t use subprocess.call, because it blocks process instead of subprocess.Popen and celery can’t just release it and finish a task.
UPD. I almost forgot — all of these also depends on celery task timeout. In our case, celery will kill task after time limit is reached and stop.