I just discovered Selenium, which allows you to drive browers programmatically. This allows you to test the effects of javascript on the browser's internal page representation. Not very useful for the department website, since we don't use much javascript, but I like its python interface, so I'm replacing my traditional spider with a Selenium implementation.

Some bugs I've run into so far:

• start() doesn't take any args. Simple fix in selenium.py.
• captureNetworkTraffic("xml") returns requestHeaders as responseHeaders. Fixed in source, but you'll need to compile your selenium-server.jar on your own. There's a nice, detailed explaination of building Selenium with Maven here, but for the impatient, a simple mvn install in the svn root directory should do it.
• captureNetworkTraffic("xml") captures invalid XML in URL's (ampersand problem). Another simple patch, but this one's just posted by me, so you'll have to apply it by hand and recompile. You get the feeling that people don't use the captureNetworkTraffic side of Selenium very often ;). Ah well, only 3 open bugs in that area left to bump into ;).

To keep the test suite easy to maintain, I build it on the fly using nose generators, which are awesome :).

More details to come...