Archive

Tag Archives: Python


An update to my rant on Celery’s frequently-changing API: I’ve decided to stay with Django-celery 2.5.5 and Celery 2.5.3.

When I tried using Celery 3.0.4 with my existing code, Pylint threw about 60 warnings, many of which look real and all of which weren’t there when I used Celery 2.5.3.

“Backwards-compatible” my ass!

I shouldn’t have to chase my tail like this. Celery, you lost me. I’m now looking to replace you.


This is a rant.

My company’s code base is over 65K lines of Python and JavaScript code. We use Celery, Django-Celery, and RabbitMQ for our background asynchronous tasks. Ten different tasks.py files contain 30 task classes, split roughly 50-50 between periodic and on-demand. We use subtasks.

Today, I dug into updating from Celery 2.5.3 to 3.0.4, and I popped my cork.

I am aggravated by the frequency and extent of Celery API changes. It’s easily changed more often than any other five technologies in our stack combined. I’ve been upgrading Celery and Django-celery every six months or so, which corresponds to upgrading every few minor versions. And the changes are similar in scope to what I see when upgrading any other technology across one or two major versions.

Read More


Sometimes you don’t write unit tests. Your reason for not doing so always falls into one of two categories.

Complexity

The code you just wrote would be so much easier to test using system-level testing. For example…

  • The setup and teardown would be 10x the test code.
  • There’s too much interaction with multiple data stores or third-party vendors.
  • Your dev boxes or CI server don’t all have the necessary technology installed.

These are rational reasons to not write unit tests for new code. You’re fine.

Simplicity

But sometimes you don’t write unit tests because the code you just wrote is so darn obvious.

It’s really simple. It’s straightforward. It’s nearly trivial. Why both writing unit tests for it?

Well, I’ll tell you why you should test it. In fact I’ll give you three reasons.
Read More


I was in some code I haven’t visited in a while. And I came upon something I coded months ago.

It used a list comprehension to test every element of a list. If the result was empty, it signaled an error. Otherwise, it used result[0].

Gah! That’s so retarded! Was I asleep when I wrote that?

I changed it to a generator expression with a .next(), within a try/except on StopIteration.

One fewer line of code, and it’s faster.

I felt good.


tl;dr

When writing tests, mock out a subsystem if and only if it’s prohibitive to test against the real thing.

!tl;dr

Our product uses Redis. It’s an awesome technology.

We’ve avoided needing Redis in our unit tests. But when I added a product feature that made deep use of Redis, I wrote its unit tests to use it, and changed our development fabfile to instantiate a test Redis server when running the unit tests locally.

(A QA purest might argue that unit tests should never touch major system components outside of the unit under test. I prefer to do as much testing as possible in unit tests, provided they don’t take too long to run, and setup and teardown aren’t too much of a PITA.)

This was a contributory reason for our builds now failing on our Hudson CI server. Redis wasn’t installed on it!

Why didn’t I immediately install Redis on our CI server?

  1. Our CI server had other problems
  2. I intended to nuke it and re-create it with the latest version of Jenkins. I just needed to first clear some things off my plate
  3. Our dev team had shrunk down to just two people
  4. We were both strict about running unit tests before checking code into the pool
  5. We were up to our necks in other alligators

From a test-quality perspective, if code uses X in production, it’s better for tests to run with X than with a simulation of X.

One of the many joys of working with Ryan is that he challenges my assumptions and makes me consider alternatives. Because of a perceived lack of elegance in needing Redis on our CI server, and because his work had been temporarily blocked by my code changes, he challenged me to replace my unit tests’ use of Redis with a mock.

I walked into work yesterday and it was quiet. All our critical bugs blocking Saturday’s release were closed. I thought, why not? I’ll give it a go. Today’s a good day to see what’s involved with replacing Redis with a mock!

Read More


After some job market feedback and chin-scratching, I’ve changed our Senior Developer opening’s job description. Now it’s less about Python or Django, and more about search technologies, specifically full-text and LSI search.

We hope candidates will have some experience with Python or Django, but search technology experience (e.g., tuning, tokenizers, parsers, relevancy rank tweaking, aggregates and pivots) in now more important, and emphasized, in the the job.

Here’s the new description:

———

Founded in 2009, IP Street develops and markets software to help corporations, law firms, financial research firms, and government agencies better analyze patent information.  Our goal is to make IP data easy to get, use, and understand, so everyone can have access to high quality and transparent information.

A significant facet of our application’s capabilities are derived from Solr and other search technologies. We’re seeking a great full-text Search developer with experience in:

  • Solr, Lucene, or other search engines
  • Full-text search schemas, tokenizers, parsers, and rules for returning statistics and meaningful analytics
  • Automated workflows that process millions of objects
  • Data quality metrics and repairs

You’ll be joining us at a great time! Revenue is coming in, and we’ve done two Angel funding rounds at increasing valuations.

Key Responsibilities.

  • Enhance our Solr engine to provide more statistics and meaningful analytics to the product
  • Enhance or tune our use of other search technologies, e.g., LSI
  • Enhance and extend the existing code base to add new product features. Our application is written in Django and Python, with an almost all open-source technology stack
  • Occasionally wear testing or devops “hats,” as the needs arise
  • Write unit tests for your code, and do performance analysis
  • Demonstrate technical leadership within the team
  • Communicate well with the team, in writing and orally

Qualifications.

  • Significant experience using and tuning Solr, Lucene, or other search engines with similar capabilities
  • 3+ years related experience in Python development
  • 1+ years experience in Django development, or a strong interest in learning
  • Experience using one or more of: MongoDB, CouchDB, or another NoSQL database; Celery; Redis; PostgreSQL or another SQL database
  • Experience using latent semantic indexing search technologies would be a plus
  • Experience integrating with open-source 3rd-party libraries
  • Experience creating customer-focused software to process data and generate statistics and analytics
  • Solid troubleshooting abilities, self-directed, and proactive
  • Enjoy all aspects of software product creation — design, implementation, and debugging
  • Familiarity with using OS X as a development environment, and Linux as a production environment
  • Bachelors Degree or equivalent in Computer Science or Software Engineering
  • Excellent communication skills

Salary is DOE.

Please send resume to johnd@ipstreet.com.


We’re looking to hire two lucky people who desire fame and fortune. Here’s the Senior Developer opening:

Founded in 2009, IP Street develops and markets software to help corporations, law firms, financial research firms, and government agencies better analyze patent-related information.  Our goal is to make IP data easy to get, use, and understand, so everyone can have access to high quality and transparent information.

We’re seeking a great Python developer with experience in: Automated workflows that process millions of objects; data quality metrics and repairs; search, particularly with Solr or Lucene; and/or general data mining. Our stack, and development & production environments, are almost all open-source. The key technologies are Python, Django, Celery, Solr, and PostgreSQL.

Read More


Pro tip: When you rely on Fabric to provision your servers, and your fabfile installs packages from the Cheeseshop, and package D (which you specify) depends on package C, and the Cheeseshop’s dependency rule is wrong, and it causes the wrong version of package C to be downloaded and installed, and the version number it should be is visually similar to the one it installed (say, 2.3.1 vs. 2.4.1), and it quickly scrolls up your terminal window, and package C version 2.4.1 is in fact incompatible with package D, and you’re tired, you can waste a lot of time chasing your tail.

 


Apple’s OS X Lion is a great product. I’m glad I upgraded. But it’s got some annoyances.

My first Mac was a version 1, back in 1985. After that, I used DEC operating systems for a few years, and then used Microsoft Windows exclusively throughout the 1990s and most of the 2000s. I switched back to Macs in 2008, when I bought a MacBook Pro with OS X Leopard.

OS X Leopard and Snow Leopard were fine, quintessential Apple products.

I’ve now upgraded to OS X Lion. Nothing in the earlier two versions made me clench my teeth, but some of Lion’s changes bother me. Someone surely had a good reason for each one, yet all but one are backward steps in usability. (I’m fence-sitting about one of them.)

I won’t prattle about why my opinion matters, but I’ll remind you that I’m really smart and what I say is always right.

Here’s my list of top OS X Lion annoyances.

Read More


I bought “Python Testing Cookbook” by Greg L. Turnquist, published by Packt.

The good. I liked the cookbook recipe approach. Each recipe has the same headers: Its name, “How to do it,” “How it works,” and “There’s more.” This may not sound fancy, and it isn’t, but it works.

The writing’s good (albeit sometimes elementary — see below) and the example code is well laid out (except when it isn’t — see below). The book starts with the simplest form of unit testing, and goes all the way to measuring test coverage, and load testing. That’s a lot for one book to cover, which is both good and bad. Your first line of defense can be one book, but unless the book is very well written, it may leave every reader scratching their head over one section or another.

I learned a couple of new tricks.

Read More

Follow

Get every new post delivered to your Inbox.

Join 9,318 other followers