Buildbot and py.test exit status


Buildbot is a great continuous integration tool, and PyTest, aka py.test, is a great unit testing tool.

But there’s an impedance mismatch between them.

In Buildbot, a ShellCommand is deemed to have failed if it returns a non-zero exit status, and succeeded if it returns a zero exit status.

Py.test always returns a zero status.  I guess their philosophy is: Py.test didn’t crash, so it was “successful,” and the exit status should be 0. So, when py.test returns a 0 status, Buildbot deems the ShellCommand successful — and therefore, the build “succeeded,” no build failure email is generated, and the buildbot waterfall remains green.

You’d think that py.test would have a “––returnnonzerostatusonfailure” switch.  But it doesn’t.

I found evidence on a mail archive that this problem has been around since 2005.

Am I missing something obvious, and if not, what’s the generally accepted way to work around this problem?

1 comment
  1. John said:

    I got an answer from Brian Dorsey on the SeaPIG mailing list:

    Hrm… interesting.

    I found this from the same year:
    http://codespeak.net/pipermail/py-dev/2005-May/000386.html

    Looks like Holger applied a fix which should exit with 1 when tests
    fail. However, when I test my local copy, it always exits with 0.

    I just chatted with Holger on IRC, and it looks like it works in 0.9.1
    and trunk but not 0.9.0. Also, the ‘dist’ branch on svn (which I
    follow for prod work) seems to be still on 0.9.0. Ironically, it looks
    like this is a regression that probably wouldn’t have happened if
    there was a test for it. 🙂

    I think Holger is planning on doing some work on this later today, but
    in the mean time, svn trunk or 0.9.1 should work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: