Alternatives to using Celery

I’ve found some candidates for replacing Celery in my company’s product. (My reasons for replacing it are elucidated here, here, and here.)

I got these from web trawling, blog comments, and some e-mail. At first blush, none of the candidates have any disqualifying attributes, except for lacking subtasks. Celery is the only Python-friendly asynchronous task technology with subtask support, so I’ll need to bend on that if I want any alternatives to consider. (If I’m wrong on this point, please let me know in the comments!)

I’m not saying that these candidates will definitely satisfy all (sans subtasks) of my requirements. Right now they’ve just passed my initial sniff test. The next step will be to read documentation in detail, assess the health/activity of its community and developers, and try some sample code.

The candidates

Huey: Web-accessible docs are here. (What is it with projects using Read the Docs? Read the Docs irks me. The stupid icon in the bottom right corner laughs at me.) Doesn’t have subtasks, and the management API amounts to reading the queue from Redis and pawing through it…

Gearman with python-gearman and/or django-gearman: Configuration seems more complex than Huey, but with little effort I found a few blog posts written about it so that’s fine. Seems like a mature technology. No subtasks. Admin API had some thought put into it. Not sure which of the bindings would be easiest to use.

Advanced Python Scheduler: No subtasks, but otherwise seems to fit my needs.

These didn’t make the cut

I disqualified two candidates because I could see, even this early in my evaluation process, that they wouldn’t fulfill my requirements. They may be fine for your needs. If my comments here are wrong, tell me!

django-background-task: The tasks run locally, so I can’t set up a farm. No subtasks. This would be suitable for low-overhead simple periodic tasks that can run locally on the Django server, e.g., a periodic task to purge an export queue.

django-ztask: No subtasks, no periodic tasks, no apparent ability to create a server farm, not much (any?) attention given to management and control.

More later!

11 thoughts on “Alternatives to using Celery

  1. Glad to see Huey on the shortlist! Just thought I’d let you know that sub tasks shouldn’t be an issue with Huey (that’s tasks running tasks, right). There’s a possible deadlock but only if you’re single process/single thread and try to block on the return value of a subtask. Sorry there’s no GUI or web admin but lots of redis tools exist for the adventurous.

    Also readthedocs was my djangodash project a couple years ago 😉

    1. *grin*. I don’t see the problem with readthedocs either. Finally some consistency between document versions, finally easily looking back at previous versions. I’m actually rather disappointed when a project doesn’t use RTD, and lives on their own little island.

  2. With django background task you can run tasks remotely, they’ll run anywhere the django app is installed. It’s also not for periodic tasks. It’s just a very simple db-backed queue, that was based on Ruby’s delayed job. Should be good for low-medium volume, but probably not much beyond that. I suspect thought it wouldn’t meet your needs – it’s very minimal.

    Nb. I wrote it.

  3. Just switched from Celery to Huey, because the combination Celery, Django and buildout is a no-go. Huey is excellent so far and has good documentation.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.