This is a rant.
My company’s code base is over 65K lines of Python and JavaScript code. We use Celery, Django-Celery, and RabbitMQ for our background asynchronous tasks. Ten different tasks.py files contain 30 task classes, split roughly 50-50 between periodic and on-demand. We use subtasks.
Today, I dug into updating from Celery 2.5.3 to 3.0.4, and I popped my cork.
I am aggravated by the frequency and extent of Celery API changes. It’s easily changed more often than any other five technologies in our stack combined. I’ve been upgrading Celery and Django-celery every six months or so, which corresponds to upgrading every few minor versions. And the changes are similar in scope to what I see when upgrading any other technology across one or two major versions.
Changes that may seem minor, like moving a class across modules (e.g., from celery.task.base to celery.app.task), are nontrivial when you have change lots of code. I might feel better if the edits gave my code some awesome new capabilities. But they don’t, so my internal reaction is, Why the <expletive> does this code have to change? Couldn’t the <unbelievably gross expletive involving a deformed farm animal> code have stayed where it <repeat the expletive> was?
The decorators change, the imports change, how to specify periodic tasks changes….FUCK. It’s maddening. The 2.5.3 -> 3.0.4 upgrade is enough to make me start thinking about replacing Celery.
I had a nice exchange with @asksol about this on #celery. He’s a smart, helpful, and friendly guy who’s always open to chatting and answering questions. (And he doesn’t deserve people like me using the fruits of his labor!) The exchange went something like this (I’m paraphrasing):
Me: Could the API not change so much?
Asksol: It doesn’t change so much.
Me: What about a, b, and c?
Asksol: There’s a compatibility API for that. You don’t have to change your code.
Me: But the compatibility API isn’t documented, and it will eventually be removed, so I either change my code now or later.
Asksol: It will be a very long time before the compat APIs are removed.
I came away sensing that we had a communications impedance mismatch. Although we spoke the same language, we didn’t see the problem with anywhere near the same degree of importance. He didn’t see it as anything notable, while I was thinking about killing small kittens.
From my perspective, deferring a code change doesn’t mean “I’ll never have to do it.” It means, “I’ll eventually have to do it, and my choice is, do it now or later.”
One could argue it’s always better to not change code unless you must, i.e., only when the deprecated code is actually removed. But the compatibility API isn’t documented, so someone looking for information before they edit this code will be massively confused. And I trust a maintainer when he/she warns that an API is deprecated, and I should use a replacement API; I’d rather make the necessary changes sooner than later because they’re, well, inevitable. It’s “good code hygiene.” Another reason to keep code current is so someone looking at it will (presumably) be better able to understand it. The reason why the maintainer deprecated something is because he/she doesn’t want it used anymore.
I don’t have the time for chasing after any more API changes. Or, more accurately, I choose to no longer do it. It’s time to catalog how many Celery features we use and look for an alternative.
End of rant.
I’m interested in hearing other opinions about when to update code based on a deprecated API. When do you do it?
Seek Nuance 
I know I feel your pain and probably to a greater degree.
Some will say that what we do is evil, but to achieve what we do as far as trying to monitor performance of tasks executed under Celery, we actually need to dynamically go in and monkey patch the internal Celery code at a couple of critical points. So, on every minor release of Celery our code breaks because they have decided to move or rename something, or introduced a completely new way of doing things such as like with billiard in Celery 3.0. In some cases between Celery 2.X and 3.0 they have changed the one thing three different times.
Unlike in your situation, we can’t just change things and ignore old versions as we need to support all versions of Celery 2.X and 3.X. We therefore end up with a bunch of checks to try and work out which way of doing things is being used in a specific version and instrument the right thing. Even then we generally can only guarantee that the mainstream case works.
The ideal situation would be if packages provided special APIs to allow hooking in and applying wrappers around things we need to instrument, but buy in for that tends to be hard to achieve if what we need is not generic and might be seen as specific to our product. It also doesn’t eliminate the need to support versions prior to when such an API is provided, so impetutus to push for such instrumentation APIs is low, at least when packages tend to have a stable code base internally as most do. Celery might well be the case though where we give up and will have to strongly push for the instrumentation hooking API we need being added to Celery, else it is going to become very hard to support it.
Hi John.
What if you write the documentation for the compatibility stuff and submit it as pull request to the celery repository. And/or maybe add tests for it (if there are none) so if anyone dare to break it, they will be aware of the problems …
Only a thought. Maybe too much work. 😦
Right now, I don’t have the time to do that. And I’m not confident it would solve the underlying problem of API changes treated as NBD.
BTW, given @asksol’s assertion that the code changes were backwards compatible, I tried backing out my code changes and running pylint. Lots of errors were flagged, so the changes were not backwards compatible.
We always appreciate docs and contribution changes, but all backward incompatible changes should be documented (if they aren’t then it’s a bug), we also strive towards 100% test coverage (though that is just a stupid metric, and won’t really protect us against anything). The best thing to do, and it shouldn’t require as much work is to test the development releases in a staging environment. I usually tell people to start testing on twitter, and on the mailing-list.
Though, I don’t think John complains so much about backward incompatible changes (there are only a few documented ones) as he does to the introduction of a new API.
I’ll share this: I’ve seriously been considering the cost/benefit to Celery in its current form, particularly since using it means taking on some overhead for basic system monitoring & management.
I’ve been meaning to look closer at http://python-rq.org/ and https://github.com/richardhenry/hotqueue, both of which seem to be very simple and – hopefully – less of a moving target.
Thanks for those two pointers! I’ll report back on what I find and decide re: a Celery replacement.
what overhead is there for monitoring and management?
Hey John,
The good news is that the API won’t change again for a very long time (years)
and the old Django-like API will be there for a very long time.
Celery 3.0 is still largely compatible with Celery 2.0 except for a few documented cases, and your code should work as long as you read the changelog.
These changes have been planned for a long time now, the two APIs have been coexisting for several versions. WIth the 3.0 release I’m trying to move people over to the new API by also changing the documentation.
The original API was severely Django inspired, which included using global settings and so on, and this makes it harder to use Celery in other frameworks. I can understand why you don’t see this as a very important change, and that’s why the old API still works and will continue to do so for a long time. The new API is also simpler to use, and may help us get rid of the ‘complex configuration’ myth.
I take backwards compatibility very seriously, and considered the changes done there have been no reports about undocumented incompatibility issues.
You forgot to mention that the examples that you said broke your code where the removal of the celery.ping task, which was deprecated for along time originally planned for removal in v2.3, because it was a naive implementation that shouldn’t be depended on (the replacement is actually useful). The removal was also documented in the changelog.
In the history of the Celery project the changes have been due to three things:
* The Django split where Celery no longer depended on Django.
* Removal of the magic keyword arguments where the worker automatically
passes certain arguments to the task if it supports it (how stupid
was that)
* The new API that enables multiple instantiation, making Celery more
into a library.
BTW: You mentioned moving celery.task.base to celery.app.task. Both of those are internal modules, the right location is `from celery import Task’.
I think that if you really take a look at the new API and the documentation
you will find that it’s a strong improvement. if not for your current project then at least for new users. And how could we have done this any differently? We’re providing a fully backwards compatible API, what else would make you happy? 🙂
@GrahamDumpleton: Monkey-patching and complaining about it? 🙂 I’ve told you we can find a way to do what you want, and have it part of the standard API. If you need compatibility with older versions then make me away of it so that I can keep you compatible.
Adding the billiard dependency was not something that i did lightly, it was necessary because it solved a very serious deadlock problem. The time you did reach out for help the problem was not due to a planned compatibility problem, but due to a bug in the new version which was fixed shortly thereafter.
I upgraded from celery 2.1.4 to 3.0.21. Can’t say I had the same issues as you. In terms of the API, I didn’t have to change anything but it says in celery that they support the old API (so I have to go with what your friend asksol said). I just had to fix the config and a few other things.