My requirements for replacing Celery

I’m ready to start looking at candidates to replace Celery in my company’s product. (The reasons are elucidated here, here, and here.)

Our SaaS product provides data mining and visualization for intellectual property. A 10-second elevator pitch is, it’s as though we attached Microsoft Excel’s chart wizard to US and international patent offices. (“As though” = “We didn’t do that, and in fact we go way beyond that, but I’m giving you a simple description.”) Our code is 100% Django and Python.

I looked at how we use Celery in our codebase. The reality of how we use it is much simpler than our ideas when we started two years ago. Combining our existing features with our product roadmap, I know with high confidence what features we need for our asynchronous tasks. And which ones are nice to have but not required, and which ones we’ll probably never need.

Here are my Celery replacement requirements. Except for the “I don’t need” or “Tasks don’t have to…” statements, they’re all about equally important.

1. The non-deprecated section of the API does not change often.
2. New code releases have one of two attributes: They’ll be 100% backwards compatible, or they’ll have clear instructions for migrating code.
3. There’s active current development, an easy way to interrogate and submit tickets, and a healthy user community. The main developer(s) is(are) reasonably communicative.
4. Our tasks are 100% Python code, and can easily access my project’s Python and Django resources. E.g., my project’s settings.py.
5. We can pass arguments into a task, just like calling a function.
6. Support for periodic and on-demand tasks. The periodic tasks’ frequency can be a simple cycle, like a timedelta().
7. For task servers:
   • Support for more than one. I.e., I can define a task server farm. It’s OK if they’re statically defined and homogeneous. (We now have just one dual hex-core Celery task server… I could let the multiple server requirement slide for six months.)
   • Support for defining the maximum number of executing tasks per server. It’s OK if this is homogeneous.
   • I don’t need different queues. I don’t need task priorities. I don’t need routing keys. I don’t need autoscaling. I don’t need remote shutdown. I don’t need any signal besides “abort.” I don’t need different tasks per server. All the tasks can be in one big FIFO queue. Submit a task, it runs, it finishes. Simplicity is powerful.
   • A built-in way for all task logging to go into one file per server would be nice, but not essential.
   • Task overhead equivalent to Unix cron jobs would be fine. We aren’t hitting the task scheduler with 18 billion task submissions/second.
8. For task management (these can be programmatic or via a web interface), a way to:
   • Know how many task servers are up.
   • Know what tasks are running, and when they were submitted. It would be nice but not essential to know on what task server they’re running.
   • Know what tasks are pending.
   • Send executing tasks an “abort” signal. It’s OK if the task code has to periodically interrogate it.
   • Cancel all pending tasks.
   • Cancel all pending tasks with a certain task name.
9. A way for a task to run subtasks, and for the parent task to wait until they’re all done. The parent just needs to know the “>= 1 subtask remains” and “0 subtasks remain” conditions.
10. Tasks don’t have to return values. All of a task’s effects can be side-effects.

I’m struck by how simple our requirements are, for such a complex and capable product!

My next step is to assemble a list of candidate technologies. I’ll report back what I come up with.