Richard Jones' Log

Thu, 28 Aug 2014
When testing goes bad

I've recently started working on a large, mature code base (some 65,000 lines of Python code). It has 1048 unit tests implemented in the standard unittest.TestCase fashion using the mox framework for mocking support (I'm not surprised you've not heard of it).

Recently I fixed a bug which was causing a user interface panel to display when it shouldn't have been. The fix basically amounts to a couple of lines of code added to the panel in question:

+    def can_access(self, context):
+        # extend basic permission-based check with a check to see whether 
+        # the Aggregates extension is even enabled in nova 
+        if not nova.extension_supported('Aggregates', context['request']):
+            return False
+        return super(Aggregates, self).can_access(context)

When I ran the unit test suite I discovered to my horror that 498 of the 1048 tests now failed. The reason for this is that the can_access() method here is called as a side-effect of those 498 tests and the nova.extension_supported (which is a REST call under the hood) needed to be mocked correctly to support it being called.

I quickly discovered that given the size of the test suite, and the testing tools used, each of those 498 tests must be fixed by hand, one at a time (if I'm lucky, some of them can be knocked off two at a time).

The main cause is mox's mocking of callables like the one above which enforces the order that those callables are invoked. It also enforces that the calls are made at all (uncalled mocks are treated as test failures).

This means there is no possibility to provide a blanket mock for the "nova.extension_supported". Tests with existing calls to that API need careful attention to ensure the ordering is correct. Tests which don't result in the side- effect call to the above method will raise an error, so even adding a mock setup in a TestCase.setUp() doesn't work in most cases.

It doesn't help that the codebase is so large, and has been developed by so many people over years. Mocking isn't consistently implemented; even the basic structure of tests in TestCases is inconsistent.

It's worth noting that the ordering check that mox provides is never used as far as I can tell in this codebase. I haven't sighted an example of multiple calls to the same mocked API without the additional use of the mox InAnyOrder() modifier. mox does not provide a mechanism to turn the ordering check off completely.

The pretend library (my go-to for stubbing) splits out the mocking step and the verification of calls so the ordering will only be enforced if you deem it absolutely necessary.

The choice to use unittest-style TestCase classes makes managing fixtures much more difficult (it becomes a nightmare of classes and mixins and setUp() super() calls or alternatively a nightmare of mixin classes and multiple explicit setup calls in test bodies). This is exacerbated by the test suite in question introducing its own mock-generating decorator which will generate a mock, but again leaves the implementation of the mocking to the test cases. py.test's fixtures are a far superior mechanism for managing mocking fixtures, allowing simpler, central creation of mocks and overriding of them through fixture dependencies.

The result is that I spent some time working through some of the test suite and discovered that in an afternoon I could fix about 10% of the failing tests. I have decided that spending a week fixing the tests for my 5 line bug fix is just not worth it, and I've withdrawn the patch.

category: Python | permanent link
Mon, 11 Aug 2014
devpi quick-start

(devpi is the caching proxy for PyPI which does a bunch of other things too but mostly just speeds up "pip install" and isolates you from network issues - or complete lack of connectivity)

I gave a 25-minute introduction to devpi at PyCon AU and since then have been requested to put together a quick-start guide:

devpi in 60 seconds

Step 1: create a virtualenv

mkvirtualenv devpi

Step 2: install devpi

pip install devpi

Step 3: run devpi

devpi-server --start

Step 4: use devpi (noting the URL from the previous command output)

devpi use --set-cfg [URL]/root/pypi

Step 5: profit!

pip install yarg

The first installation will call out to the Internet but subsequent installs will use the cached version.


category: Python | permanent link
Sat, 26 Apr 2014
Those neck-supporting pillows for flying? You're using them wrong!

OK folks, something I learned about those horseshoe-shaped pillows: they don't go around your neck. They're basically useless like that.

First up: make sure you get one that joins up at the open end - some sort of clip fastener to hold the ends together.

Then: it goes around the back of your head with the bottom of the horseshoe behind your neck and the open end clipped together but pointing up. If you're doing it right, that sucker will fit very snugly over the back of your head to the point that it'll stay stuck on when you move your head away from the headrest.

Doing this will mean that your head is held in place vertically and horizontally.

Note: I lost the pillow I had that joined together with a clip and bought another that didn't but was made of "memory foam" and it worked just as well.

Thu, 24 Apr 2014
PyCon AU 2014 CFP about to close!

The PyCon Australia 2014 CFP is about to close! Last chance to get your proposal in! Quoted:

The conference this year will be held on Saturday 2 and Sunday 3 August 2014 in Brisbane. We'll also be featuring a day of miniconfs on Friday 1 August.

The deadline for proposal submission is Friday 25 April, 2014.

PyCon Australia attracts professional developers from all walks of life, including industry, government, and science, as well as enthusiast and student developers. We’re looking for proposals for presentations and tutorials on any aspect of Python programming, at all skill levels from novice to advanced.

Presentation subjects may range from reports on open source, academic or commercial projects; or even tutorials and case studies. If a presentation is interesting and useful to the Python community, it will be considered for inclusion in the program.

We're especially interested in short presentations that will teach conference-goers something new and useful. Can you show attendees how to use a module? Explore a Python language feature? Package an application?

Proposals about the Django web framework are very strongly encouraged, and will also be considered for inclusion in DjangoCon AU, to be held on Friday 1 August.

There will also be a Science and Data Analysis miniconf and an OpenStack miniconf held alongside DjangoCon AU. Proposals on either of these topics will be considered for inclusion in these miniconfs.

We welcome first-time speakers; we are a community conference and we are eager to hear about your experience. If you have friends or colleagues who have something valuable to contribute, twist their arms to tell us about it! Please also forward this Call for Proposals to anyone that you feel may be interested.

See you in Brisbane in August!

category: Python | permanent link
Mon, 20 Jan 2014
Python 3.3 and virtualenv

We're kicking off some new projects using Python 3 (yay!) but had some issues getting virtualenvs working. Which is kinda ironic given that Python 3.3 included virtualenv in it, as pyvenv. Unfortunately, pyvenv isn't quite the same thing as virtualenv, and in particular it doesn't install/include pip and setuptools. There's also some additional issues introduced under Ubuntu.

First, you'll need to obtain Python 3.3. Some of the methods you could use will work and some are known to produce a non-viable environment. In particular:

  • OS X: get it from homebrew ("brew install python3"). I've not tried other avenues, but this works and is the easiest approach in my opinion.
  • Ubuntu: get it from source, building like so:
    sudo apt-get install build-essential libsqlite-dev sqlite3 bzip2 libbz2-dev
    tar jxf ./Python-3.3.3.tar.bz2
    cd ./Python-3.3.3
    ./configure --prefix=/opt/python3.3
    make && sudo make install
    Do not attempt to use any currently-available pre-built packages (eg. from a PPA) as they will create broken virtualenvs. See this discussion for some enlightenment, but note the lack of a reasonable solution.
  • Windows: no idea, sorry.

Now that you've got a Python 3.3 installation, you can create your virtual environment. You do this with this command combination:

. /bin/activate

Now you should have a viable, working Python 3.3 virtual environment.

Fortunately Python 3.4 is going to improve on this by installing pip alongside python.

Also, pip 1.5.1's "" will let you skip that extra setuptools install above when it's out (real soon).

category: Python | permanent link
Thu, 25 Jul 2013
Python Game Programming Challenge (PyWeek) #17 is coming!

The 17th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 1st to the 8th of September.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

category: Python | permanent link
Fri, 12 Jul 2013
PyCon AU 2013 and Things You Should Not Do In Python

So PyCon Australia is done for another year and it was a corker. I'm somewhat overwhelmed and Graeme Cross has an great summary of the event, but I really must repeat what an amazing job the organisers did in running a quality, smooth event packed with Python. See also Katie Miller's writeup about the 10 things that PyCon AU does particularly well.

Personally, I pleased as punch that my talk Don't Do This went as well as it did - I had a blast giving it and it was well-received.

Also, Ben Finney talked a bunch about PyCon AU in Wednesday's Byte Into It on 3RRR.

category: Python | permanent link
Thu, 21 Mar 2013
How awesome was PyCon?

PyCon was awesome.

It started with the Young Coders tutorial where I had the privilege of helping out as an assistant. The kids were amazing. Barbara Shaurette and Katie Cunningham were amazing. It was awesome.

The next day I popped in to help set up that room again but then spent the rest of the day focusing on my afternoon tutorial teaching people to write games in Python. It was packed - 35 or so attendees had signed up but then a bunch of the kids from the first day of Young Coders also came along to watch and learn. Including an 8 year old girl who told her dad she wanted to write video games. Awesome.

Then the conference proper started with an inspiring opening speech by the conference chair Jesse Noller about his vision of Change the future - education, outreach, politeness, respect, tenacity and vision. A broad, ambitious and awesome vision, and one I can fully get behind. Then Eben Upton took the stage to tell us about his journey with the Raspberry Pi project and a little of his vision. Lots of similarities. And then everyone got a Raspberry Pi. Awesome.

People wanting help figuring out how to teach kids to program their Raspberry Pi could try the free Raspberry Pi Education Manual PDF.

I spent a lot of time during the conference split between talking through various things in the hallway (I'm the Cheeseshop BDFL and there's many things afoot there and PyCon is a great time to move things forward very quickly), and spending time in the Raspberry Pi lab, chairing talk sessions for amazing speakers and generally having my mind awesomely expanded.

I taught many people the basics of pygame. I ran into the Kivy developers, and talked to them about the Pi. During the sprints they ported Kivy to the Pi and even developed some simple games using Kivy, the Pi and some simple hardware sensors (I'm playing the game with a tilt sensor).

I had an inspiring conversation with James Bennett during which a light went off in my head and the result is Python Enhancement Proposal (PEP) 439 for "Inclusion of pip bootstrap in Python installation". (apologies if you get a badly-formatted version; it should be fixed shortly.)

I'm exhausted. The people here - the community - is amazing and vibrant and brilliant and huge. 2500 people at the conference, 20% women. Awesome.

Seriously, look how many people were there...

category: Python | permanent link
Mon, 18 Feb 2013
Python Game Programming Challenge (PyWeek) #16 is coming!

The 16th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 14th to the 21st of April.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

category: Python | permanent link
Fri, 15 Feb 2013
PyPI password-related security changes
TL;DR: please log into PyPI and change your password.

Recently we have been auditing and improving security of the Python Package Index (PyPI) and other hosts.

You may be aware that the host was compromised. Since we must assume that all passwords stored in that system are also compromised, and we also assume that some users share passwords between systems, we are performing a password reset of all PyPI accounts in one week's time, at 2013-02-22 00:00 UTC.

If you log in before that deadline and change your password then you'll be fine, otherwise you'll need to use the password recovery form after the reset has occurred.

Additionally, we would ask you to begin to access PyPI using HTTPS through the web. We're in the process of installing a new SSL certificate so the current Big Red Certificate Warning should go away very soon.

We are in the process of updating the Python packaging toolset to use HTTPS.

These steps are but a couple of those we're intending to take to better secure PyPI. If you are interested in these matters I encourage you to participate in the discussion on the catalog SIG.

Finally, we apologise for any inconvenience these changes have caused.

category: Python | permanent link
Sat, 09 Feb 2013
Introducing - the new

We've just added a new domain to the stable,, to replace the existing domain. All accesses to the old domain are redirected to the new - all existing references will work. Google et al should catch up eventually.

This was done as part of an overall review of security of the PyPI website. Warning, that thread is longish :-)

category: Python | permanent link
Fri, 04 Jan 2013
Compiling cx_Oracle on OS X

Occasionally I need to compile cx_Oracle on OS X and the 32- and 64-bit worlds collide head-on in a mess of "implicit conversion shortens 64-bit value into a 32-bit value" and "file was built for unsupported file format which is not the architecture being linked (x86_64)" errors.

I keep forgetting the various steps needed to make this work correctly so here they are:

  1. Create a virtualenv with a python interpreter stripped to 32-bit only. In the virtualenv bin directory ($WORKON_HOME/name_of_virtualenv/bin):
    % mv python python.fat
    % lipo python.fat -remove x86_64 -output python
    Doing this saves all the hassles of that "arch", "VERSIONER_PYTHON_PREFER_32_BIT" and "defaults write blah blah" guff and doesn't affect any other virtualenvs.
  2. Now that we have a "thin" Python compatible with the Oracle library, we can build cx_Oracle. Unfortunately even though Python is thin its configuration is not so we need to force things using an obscure environment variable that distutils will pick up:
    ARCHFLAGS="-arch i386" pip install cx_Oracle

There, that was easy, wasn't it?

category: Python | permanent link
Thu, 09 Aug 2012
Python Game Programming Challenge (PyWeek) #15 is coming!

The 15th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 9th to the 16th of September.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

category: Python | permanent link
Thu, 23 Feb 2012
PyWeek #14 date change

The next PyWeek will be in May from the 6th to 13th. Not April.

category: Python | permanent link
Tue, 21 Feb 2012
Python Game Programming Challenge (PyWeek) #14 is coming!

The 14th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 6th to the 13th of May.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

If you're in the US and can make it I'm co-presenting a 3 hour pygame tutorial at PyCon in March.

category: Python | permanent link
Mon, 30 Jan 2012
PyPI is an OpenID provider

PyPI is now an OpenID provider.

To use this OpenID provider, enter into any form that expects an OpenID*. Should the service not support OpenID 2, you will have to enter instead (using your PyPI username.) Log into PyPI and visit your details page if you'd like to cut-n-paste the URL.

We follow the emerging approach that you have to sign into PyPI before signing into the actual services. This is intended to prevent phishing, as otherwise the relying party may fake PyPI's login page and collect your PyPI password (which they can still do if you fall for it.) It also avoids "nested" logins (i.e. where you need to log into PyPI with an OpenID while trying to login elsewhere with the PyPI id.)

If you find any problems with this service, please report them to the PyPI bug tracker.

*: of course for sites that extend PyPI this can be simplified to a simple button saying "link to my PyPI account".

category: Python | permanent link
Fri, 06 Jan 2012
Tips for Testing Twisted

This post contains some tips for testing Twisted applications. Note that I'm not using trial, the official Twisted test runner, for a variety of reasons.

Twisted Deferreds in Testing

You could use trial, but if you need to use another testing tool (nose, or behave) then you'll need to do a little more work.

To let your deferreds run (and make sure they do finish) wrap your test functions in the "@deferred" decorator from nose.twistedtools:


This will run the reactor for 1 second before declaring it hung.

If you want to see deferreds that are not behaving you will need to enable logging of Twisted events if they're not already enabled:

observer = log.PythonLoggingObserver()

and then patch nose.twistedtools as follows:

--- original/nose/
+++ installed/nose/
@@ -39,12 +39,19 @@

 _twisted_thread = None

+_deferreds = []
 def threaded_reactor():
     Start the Twisted reactor in a separate thread, if not already done.
     Returns the reactor.
     The thread will automatically be destroyed when all the tests are done.
+    from twisted.internet import defer
+    def __init__(self):
+        _deferreds.append(self)
+    defer.DebugInfo.__init__ = __init__
     global _twisted_thread
         from twisted.internet import reactor
@@ -135,6 +142,8 @@
             def errback(failure):
                 # Retrieve and save full exception info
+                    if issubclass(failure.type, StandardError):
+                        print failure
@@ -156,6 +165,9 @@
                 error = q.get(timeout=timeout)
             except Empty:
+                for d in _deferreds:
+                    if hasattr(d, 'creator') and not hasattr(d, 'invoker'):
+                        print d._getDebugTracebacks()
                 raise TimeExpired("timeout expired before end of test (%f s.)"
                                   % timeout)
             # Re-raise all exceptions

(I tried a couple of times to figure out how trial does the above but failed miserably)

This will result in a display of all the deferreds still pending when the deferred decorator timeout fires. Note that the deferred decorator introduces a deferred that will not have terminated.

One outstanding problem I've not solved is that inlineCallbacks aren't displayed usefully in the failure debug output - you get told that it is some inline callback that is still hanging, but not where it was created.

Twisted Web and Selenium

Testing Twisted web using Selenium is made more complicated by the need to allow the Twisted reactor to run at the same time as the selenium webdriver is trying to poke at it.

In other applications I've tested like this I just run up the server being tested in a separate thread but I can't do that in the case of Twisted.

To make this work any selenium operation that will cause a Twisted reactor event must be deferred to a thread, hence the liberal sprinkling of deferToThread throughout this code. Some sections do not touch the reactor (for example, filling out a form once the form page has been loaded doesn't), so it can be straight selenium calls.

Note also that the pages are generated in a server dynamically using deferreds so I use a "done" tag marker at the end to indicate to the tests that the page has been fully generated. Selenium's timeout on element access allows me to cause the test to fail if the done marker does not appear.

For example this step function directly from a behave feature implementation:

@then('the message should be sent')
def step(context):
    # wait until we're all done
    yield threads.deferToThread(lambda: get_element(context.browser, id="done"))

    # there should be a result container
    results = yield threads.deferToThread(lambda: find_elements(context.browser, id="result"))
    result = results[-1]
    status = yield threads.deferToThread(lambda: get_element(result, id="status"))

    # the status could be a number of values depending on synchronisiation of
    # the tests / runner
    assert status.text in 'created new accepted done'.split()

    message = context.db.messages.values()[0]
    if message.status != message.STATUS_DONE:
        yield esme._wait_until_done(context)

More tips as I happen to think of them or discover them :-)

category: Python | permanent link
Thu, 22 Dec 2011
New Year Python Meme

Thanks, Tarek, for this fun idea.

1. What's the coolest Python application, framework or library you have discovered in 2011?
I was pretty happy I discovered the awesomeness of bottle when researching my web micro-framework battle (video.)

I've started using Python 2.7 and 3.2 which is pretty cool (having been stuck in 2.3, gasp!)

I've been working with Twisted again after a couple of years' break and have discovered txpostgres and Twisted's own inlineCallbacks. Both are pretty cool. inlineCallbacks make Twisted programming quite bearable to think about :-)

2. What new programming technique did you learn in 2011?
I've been honing my testing skills and learned about Behaviour Driven Development. I've also learned to use both Mercurial and Git (and the latter still drives me insane sometimes) and their related websites bitbucket and github.

I also started using virtualenv and Fabric way more this year.

3. What's the name of the open source project you contributed the most in 2011? What did you do?
This year I created parse, overload and ooch and contributed to behave. I also helped run PyCon Australia.

4. What was the Python blog or website you read the most in 2011?
Planet Python and the New Packages feed from PyPI.

5. What are the three top things you want to learn in 2012?
I can never predict what I'll end up learning. Or what software I'll be writing.

I know I'll be learning a lot more about Mercuruial (and Git, I suppose) and the vagaries of modern software deployment.

I plan on working more on my test- and behaviour-driven development practices.

I also want to learn how to bake more things thanks to the Great British Bake Off :-)

6. What are the top software, app or lib you wish someone would write in 2012?
I'm hoping that someone will do something cool with the new PyPI OpenID provider ;-)

Want to do your own list? here's how:

  1. copy-paste the questions and answer to them in your blog
  2. tweet it with the #2012pythonmeme hashtag
category: Python | permanent link
Tue, 23 Aug 2011
Cheese Shop (PyPI) sprinting at PyCon AU

We've had a fun couple of days sprinting on the Cheese Shop at PyCon AU where a number of contributors have fixed bugs and improved or added features (though always with the goal of keeping the service simple of course.)

In particular:

  1. Andy Todd helped clean up some aspects of the underlying database and fix up some of the sql.
  2. Capel Brunker added some more XML-RPC functionality, performed some tracker triage and also addressed some bugs and security issues.
  3. Kaleb Ufton, in his first contribution to Open Source development, added a bug tracker URL field to packages (which persists across releases and you must enter by editing through the website.) He also helped me sort out some twisted Apache configuration issues.
  4. I finally got around to writing the "newest packages" RSS feed.

There's another secret project we kicked off that will hopefully appear in the next couple of days, and some additional work that will hopefully come to fruition within a week or so. Stay tuned :-)

Thanks to everyone who contributed!

category: Python | permanent link
Wed, 27 Jul 2011
Using Solaris "priv" with Fabric

I needed support for "priv" instead of Fabric's built-in "sudo" support. I went through a number of (sometimes quite horrific) iterations before I settled on this relatively simple solution:

import contextlib

def priv(user):
    '''Context manager to cause all run()'ed operations to be
    Replaces with the priv command for the duration of the
    save_shell = = 'priv su - %s -c' % user
    yield = save_shell

This is then used in a fabfile like so:

    with priv('remote_user'):
        run('do some remote command as remote_user')
        run('another remote command as remote_user')
category: Python | permanent link