Archive

Tag Archives: Python


An update to an earlier post

We’ve had problems using the pyrax SDK, mostly in account authentication.

First, it wasn’t at all clear when, or under what conditions, we had to re-authenticate our pyrax token. As documented, after you initially authenticate your credentials, pyrax handles all subsequent re-authentication under the covers. I.e., it will automatically re-authenticate the token if it ever expires.

This is kind of odd. I don’t understand why a good token should need re-authentication.

We then discovered that pyrax sometimes can’t re-authenticate our token! Every 19 hours, we hit a period of about five hours when our token won’t automatically authenticate. Why? I still don’t have a clear answer. Some authentication server, somewhere, clearly gets confused. You won’t run into this bug if you don’t have long-running processes. But, we do.

Read More


We host IP Street’s SAAS product at Rackspace. We’re finally taking the plunge and upgrading from python-cloudfiles to pyrax. We didn’t have any big issues with python-cloudfiles, but I was tiring of getting the brush-off from Rackspace when we asked for help with an API failure.

The benefits of keeping a technology up-to-date far outweighs the costs, unless you’re in an extreme corner case with a very unreliable vendor. Better performance, bug fixes, better capabilities, better support… all good stuff.

 


I’ve found some candidates for replacing Celery in my company’s product. (My reasons for replacing it are elucidated here, here, and here.)

I got these from web trawling, blog comments, and some e-mail. At first blush, none of the candidates have any disqualifying attributes, except for lacking subtasks. Celery is the only Python-friendly asynchronous task technology with subtask support, so I’ll need to bend on that if I want any alternatives to consider. (If I’m wrong on this point, please let me know in the comments!)

I’m not saying that these candidates will definitely satisfy all (sans subtasks) of my requirements. Right now they’ve just passed my initial sniff test. The next step will be to read documentation in detail, assess the health/activity of its community and developers, and try some sample code.

Read More


I’m ready to start looking at candidates to replace Celery in my company’s product. (The reasons are elucidated here, here, and here.)

Our SaaS product provides data mining and visualization for intellectual property. A 10-second elevator pitch is, it’s as though we attached Microsoft Excel’s chart wizard to US and international patent offices. (“As though” = “We didn’t do that, and in fact we go way beyond that, but I’m giving you a simple description.”) Our code is 100% Django and Python.

I looked at how we use Celery in our codebase. The reality of how we use it is much simpler than our ideas when we started two years ago. Combining our existing features with our product roadmap, I know with high confidence what features we need for our asynchronous tasks. And which ones are nice to have but not required, and which ones we’ll probably never need.

Read More


Commenting on my update to my Celery rant, @asksol asked me to post the Pylint results that made me question the claim of backwards compatibility.

(“@Asksol asked” — See what I did there? That’s alliteration. It’s a sign of a quality blog post. Ask for it by name.)

Again for the record, @asksol is a smart and friendly person. I know I wouldn’t last a day supporting a project the way he has supported Celery over multiple years. I’ve calmed down since yesterday, and I hope that something good results from my rant — if not for me, then for a future Celery user needing upgrade help. In his reply to my rant, @asksol describes some history and rationale for how he manages code change, and I encourage you to read it.

Here we go:

Read More


An update to my rant on Celery’s frequently-changing API: I’ve decided to stay with Django-celery 2.5.5 and Celery 2.5.3.

When I tried using Celery 3.0.4 with my existing code, Pylint threw about 60 warnings, many of which look real and all of which weren’t there when I used Celery 2.5.3.

“Backwards-compatible” my ass!

I shouldn’t have to chase my tail like this. Celery, you lost me. I’m now looking to replace you.


This is a rant.

My company’s code base is over 65K lines of Python and JavaScript code. We use Celery, Django-Celery, and RabbitMQ for our background asynchronous tasks. Ten different tasks.py files contain 30 task classes, split roughly 50-50 between periodic and on-demand. We use subtasks.

Today, I dug into updating from Celery 2.5.3 to 3.0.4, and I popped my cork.

I am aggravated by the frequency and extent of Celery API changes. It’s easily changed more often than any other five technologies in our stack combined. I’ve been upgrading Celery and Django-celery every six months or so, which corresponds to upgrading every few minor versions. And the changes are similar in scope to what I see when upgrading any other technology across one or two major versions.

Read More

Follow

Get every new post delivered to your Inbox.

Join 9,493 other followers