I’ve found some candidates for replacing Celery in my company’s product. (My reasons for replacing it are elucidated here, here, and here.)
I got these from web trawling, blog comments, and some e-mail. At first blush, none of the candidates have any disqualifying attributes, except for lacking subtasks. Celery is the only Python-friendly asynchronous task technology with subtask support, so I’ll need to bend on that if I want any alternatives to consider. (If I’m wrong on this point, please let me know in the comments!)
I’m not saying that these candidates will definitely satisfy all (sans subtasks) of my requirements. Right now they’ve just passed my initial sniff test. The next step will be to read documentation in detail, assess the health/activity of its community and developers, and try some sample code.
I’m ready to start looking at candidates to replace Celery in my company’s product. (The reasons are elucidated here, here, and here.)
Our SaaS product provides data mining and visualization for intellectual property. A 10-second elevator pitch is, it’s as though we attached Microsoft Excel’s chart wizard to US and international patent offices. (“As though” = “We didn’t do that, and in fact we go way beyond that, but I’m giving you a simple description.”) Our code is 100% Django and Python.
I looked at how we use Celery in our codebase. The reality of how we use it is much simpler than our ideas when we started two years ago. Combining our existing features with our product roadmap, I know with high confidence what features we need for our asynchronous tasks. And which ones are nice to have but not required, and which ones we’ll probably never need.
Commenting on my update to my Celery rant, @asksol asked me to post the Pylint results that made me question the claim of backwards compatibility.
(“@Asksol asked” — See what I did there? That’s alliteration. It’s a sign of a quality blog post. Ask for it by name.)
Again for the record, @asksol is a smart and friendly person. I know I wouldn’t last a day supporting a project the way he has supported Celery over multiple years. I’ve calmed down since yesterday, and I hope that something good results from my rant — if not for me, then for a future Celery user needing upgrade help. In his reply to my rant, @asksol describes some history and rationale for how he manages code change, and I encourage you to read it.
Here we go:
An update to my rant on Celery’s frequently-changing API: I’ve decided to stay with Django-celery 2.5.5 and Celery 2.5.3.
When I tried using Celery 3.0.4 with my existing code, Pylint threw about 60 warnings, many of which look real and all of which weren’t there when I used Celery 2.5.3.
“Backwards-compatible” my ass!
I shouldn’t have to chase my tail like this. Celery, you lost me. I’m now looking to replace you.
This is a rant.
Today, I dug into updating from Celery 2.5.3 to 3.0.4, and I popped my cork.
I am aggravated by the frequency and extent of Celery API changes. It’s easily changed more often than any other five technologies in our stack combined. I’ve been upgrading Celery and Django-celery every six months or so, which corresponds to upgrading every few minor versions. And the changes are similar in scope to what I see when upgrading any other technology across one or two major versions.
Sometimes you don’t write unit tests. Your reason for not doing so always falls into one of two categories.
The code you just wrote would be so much easier to test using system-level testing. For example…
- The setup and teardown would be 10x the test code.
- There’s too much interaction with multiple data stores or third-party vendors.
- Your dev boxes or CI server don’t all have the necessary technology installed.
These are rational reasons to not write unit tests for new code. You’re fine.
But sometimes you don’t write unit tests because the code you just wrote is so darn obvious.
It’s really simple. It’s straightforward. It’s nearly trivial. Why both writing unit tests for it?
Well, I’ll tell you why you should test it. In fact I’ll give you three reasons.
I was in some code I haven’t visited in a while. And I came upon something I coded months ago.
It used a list comprehension to test every element of a list. If the result was empty, it signaled an error. Otherwise, it used result.
Gah! That’s so retarded! Was I asleep when I wrote that?
I changed it to a generator expression with a .next(), within a try/except on StopIteration.
One fewer line of code, and it’s faster.
I felt good.