Two years with Celery in Production: Bug Fix Edition

Ketan Bhatt
Dec 17, 2017 · 7 min read
Image for post
Image for post
Photo by Martin Oslic on Unsplash

The Issues

Worker servers always had an unexplainably high RAM usage

USER        START   TIME COMMAND
ubuntu 03:38 0:57 \_ celery worker A
ubuntu 03:38 0:30 | \_ celery worker A
ubuntu 03:38 0:32 | \_ celery worker A
ubuntu 03:38 0:32 | \_ celery worker A
ubuntu 03:38 0:31 | \_ celery worker A
ubuntu 03:38 0:58 \_ celery worker B
ubuntu 03:38 0:31 \_ celery worker B
ubuntu 03:38 0:31 \_ celery worker B
ubuntu 03:38 0:31 \_ celery worker B
ubuntu 03:38 0:30 \_ celery worker B
ubuntu Sep09 1:59 celery worker C
ubuntu Sep10 3:04 celery worker D

Worker servers always had an unexplainably high CPU usage

ps aux -sort -%cpu

Workers stayed idle, not consuming any tasks

celery -A squadrun inspect active
sudo strace -p 27067 
Process 27067 attached
futex(0x33372e8, FUTEX_WAIT_PRIVATE, 2, NULL ^C
Process 27067 detached

Workers kept on restarting

2017-09-09 16:48:44,188 [CRITICAL] celery.worker: Unrecoverable error: TypeError("'NoneType' object is not callable",)
Traceback (most recent call last):
File "celery/worker/worker.py", line 203, in start
self.blueprint.start(self)

...
...
File "billiard/pool.py", line 1487, in apply_async
self._quick_put((TASK, (result._job, None, func, args, kwds)))
TypeError: 'NoneType' object is not callable

Publishers did not adhere to “task publish retry policy”

Takeaways:


Squad Engineering

Stories from the people who build Squad

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store