Buildbot and py.test exit status

Buildbot is a great continuous integration tool, and PyTest, aka py.test, is a great unit testing tool.

But there’s an impedance mismatch between them.

In Buildbot, a ShellCommand is deemed to have failed if it returns a non-zero exit status, and succeeded if it returns a zero exit status.

Py.test always returns a zero status.  I guess their philosophy is: Py.test didn’t crash, so it was “successful,” and the exit status should be 0. So, when py.test returns a 0 status, Buildbot deems the ShellCommand successful — and therefore, the build “succeeded,” no build failure email is generated, and the buildbot waterfall remains green.

You’d think that py.test would have a “––returnnonzerostatusonfailure” switch.  But it doesn’t.

I found evidence on a mail archive that this problem has been around since 2005.

Am I missing something obvious, and if not, what’s the generally accepted way to work around this problem?

One thought on “Buildbot and py.test exit status

  1. I got an answer from Brian Dorsey on the SeaPIG mailing list:

    Hrm… interesting.

    I found this from the same year:

    Looks like Holger applied a fix which should exit with 1 when tests
    fail. However, when I test my local copy, it always exits with 0.

    I just chatted with Holger on IRC, and it looks like it works in 0.9.1
    and trunk but not 0.9.0. Also, the ‘dist’ branch on svn (which I
    follow for prod work) seems to be still on 0.9.0. Ironically, it looks
    like this is a regression that probably wouldn’t have happened if
    there was a test for it. 🙂

    I think Holger is planning on doing some work on this later today, but
    in the mean time, svn trunk or 0.9.1 should work.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.