Richard Jones' Log

I've missed a couple of Fridays. Here's the side of the house that I really, some day, should actually get around to cleaning up...

First rule of any news you want people to take notice of is "Don't announce on Friday." This is why governments and corporations leave potentially nasty news releases until Friday - nobody pays any attention.

Add to this a new, second rule of announcing any Python-related news, "Don't announce anything during PyCon, unless you're actually there and giving a presentation" ... which probably seems obvious to everyone but me. Oh well, I'll make sure the Roundup 0.7 final release announcement goes out on a Monday :)

Beta testing of 0.7 is coming along swimmingly. While updating the documentation, I threw together a What's New in Roundup 0.7 page.

It's got all the big changes, with more of an explanation of what's been done (than what appears in the CHANGES file) and how to take advantage of the new features.

People probably don't realise that the Roundup hyperdb (which is Atomic, Transactional, Object-Oriented and Persistent) may be used outside of Roundup the issue tracker. Utterly contrived example:

from roundup.hyperdb import String, Number, Multilink
from roundup.backends.back_bsddb import Database, Class

class config:
    DATABASE='/tmp/hyperdb_example'

db = Database(config, 'admin')
# define a simple schema
spam = Class(db, 'spam', name=String(), size=Number())
widget = Class(db, 'widget', title=String(), spam=Multilink('spam'))

# add some data
oneid = spam.create(name='one', size=1)
twoid = spam.create(name='two', size=2)
widgetid = widget.create(title='a widget', spam=[oneid, twoid])

# dumb, simple query
print widget.find(spam=oneid)
print widget.history(widgetid)

... which displays ...

['4']
[('4', <Date 2004-03-25.01:16:1.386365>, '1', 'create', {})]

And of course in the hyperdb, you've got multiple backends to choose from (anydbm, bsddb, bsddb3, metakit, sqlite, mysql and postgresql), full journalling of changes (may be turned off selectively), automatic behaviours (through detectors auditing and reacting), object querying and full-text indexing built-in.

Of course, having come up with this example, I've seen some very simple API changes that could make it even easier to use the hyperdb outside of Roundup. If there's interest, that is...

... and damn it feels good to finally get that sucker out. In short, this release has:

• added postgresql backend (originally from sf patch 761740, many changes since)
• RDBMS backends implement their session and one-time-key stores and full-text indexers; thus they are now performing their own locking internally
• added new "actor" automatic property (indicates user who cause the last "activity")
• all RDBMS backends have sensible data typed columns and indexes on several columns
• we support confirming registration by replying to the email (sf bug 763668)
• all HTML templating methods now automatically check for permissions (either view or edit as appropriate), greatly simplifying templates

See the online CHANGES.txt for the full list (lots of web interface improvements, ZRoundup is back, ....).

There's still a couple of minor changes to make it in, and I need to check over the documentation for the new features, but the hard work for this release is all done.

I'm really hoping that existing users can see their way to spending the half-hour to hour it would take to try this release out while it's in beta.

We've been noticing more wasps than usual out the front of our house lately. Rachel spotted the nest today, so I'm up for some wasp-nuking action tonight. Strangely, the instructions for killing a wasp nest that she found online were at the Museum of Victoria. It's actually part of a set of pages about how European wasps were introduced to Australia back in the mid-20th century.

This is kind of timely, given recent discussions here about removing some of our restrictive quarantine barriers.

(Yes, that last link is to the personal weblog of one of our independent Federal senators. Neat eh?)

I finished off the RDBMS full-text indexing code this morning, meaning that the TODO list for the 0.7 release of Roundup (which has been a long-time coming) now consists of:

• Converting the currently string-only RDBMS tables to proper data types,
• Converting the other backends to use numeric IDs, and
• Performing lots of upgrade tests (for upgrading from 0.6).

Update (6 hours later): postgresql backend is now fully typed, with numeric ids. Sqlite backend uses most of the same code, and happily ignores the datatype declarations :) Now for MySQL. Eugh

This post's mostly for those family members who keep bugging me to put more photos up. Well, here they are :)

MySQL going senile. On the roundup-users list at the moment, we've got a most bizzare problem we're trying to solve. In a nutshell, we have a table called "_msg". We:

select id from _msg

Then for each id we get back, we:

select __retired__ from _msg where id=%s

And for some of the ids returned from the first statement, we don't get a result from the second.

Unfortunately the same statements entered straight into mysql have no problem. The same simple statements entered into a short Python script have no problem. It's just when this code is run in Roundup.

We're still looking into it, but it's mighty frustrating :(

Melbourne House is producing a Transformers video game, and it looks very promising. The Transformers appeared on TV after all I'd already had my fill of Macross and Star Blazers and it just seemed like so much of a cheap US ripoff. But I digress. The game sounds like it's going to be quite fun, and quite a technical achievement as well:

From the five (of the eight) levels I've played so far, it's certainly one of the most exciting console games in prospect for the entire year. Control-wise, it's instantly accessible, with a solid, convincing feel - so important in a game that could have so easily become yet another clunky, dull mechalike.
...
the combat department ... conveys an immensely satisfying sense of fierce, futuristic battle, with enemies always providing a harsh but fair contest - forcing you to go to war with intelligence and tactics rather than merely all guns blazing.
...
But all this would mean very little if it weren't for the impressive power of Melbourne House's game engine which seems capable of rendering an almost impossible degree of scenic detail at vast distances, not to mention immensely impressive character models - some of which truly have to be seen to be believed.

I wonder whether they'll have a Mini Cooper r50 playable? :)

Bloody MySQL. I'm so close to just dropping it from Roundup. Only problem is that I now have users who use it. More than metakit, it would seem. Apparently there's a myth out there that MySQL is good, or something.

As previously noted, I'm having trouble managing the test database. The solution at that time was to "rm -rf /var/lib/mysql/rounduptest". Which is fine for a one-off fix. But currently if I try:

mysql> drop database rounduptest;
Query OK, 0 rows affected (0.02 sec)

NO THE "QUERY" IS NOT "OK" YOU LYING HORSE HOCKEY!

(Aside: M*A*S*H was on TV here recently ;)

According to the docs, the command should actually report the number of files removed from disk (as "rows", but I shan't say anything, noo). So even though it's quite happy that the "Query" is "OK", it's clearly not.

The solution to this mess? I have to manually drop all the tables before I drop the database. This was not required in previous versions of MySQL. It's not mentioned in the documentation.

I've logged a bug report about this... for now though, Roundup's unit tests manually iterate through a "SHOW TABLES" listing and drop all the tables. Eugh.

Several people have posted about Python's new logging module. Most are confused by its complexity and inaccessible documentation (great reference, where's the introduction?) Well, I've used it once now, and here's the simplest I could make it:

import logging
logger = logging.getLogger('myapp')
hdlr = logging.FileHandler('var/myapp.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)

And then to use it:

logger.info('a log message')

... of course, for most people, the ideal usage would be:

import logging
logger = logging.open('var/myapp.log')

Gee, that'd be nice. Maybe I'll submit that as a patch, in one of my spare moments.

Just threw this PIL-using Python script together to automatically resize and add a border with copyright notice to my Photo Friday images:

#!/usr/bin/env python
# -*- coding: iso-8859-1 -*-
import sys, Image, ImageOps

photo = Image.open(sys.argv[1])
photo.thumbnail((500, 500), Image.ANTIALIAS)
copy = Image.open('/home/richard/Documents/�2004 richard@mecanicalcat.net.png')
photo = ImageOps.expand(photo, border=copy.size[1])
photo.paste(copy, (photo.size[0]-copy.size[0], photo.size[1]-copy.size[1]))
photo.save('out.jpg')

Yes, I cheated and used a pre-drawn image for the copyright text. Rendering that is left as an exercise for the reader :)

[Edit: switched to using the thumbnail method, thanks Ian Bicking for the pointer in the comments]

[Edit 2: indicated that I use PIL :)]

Abbey's view of the world.

OK Toby, I'll play too:

Rachel's often commented that my random playlist is a strange mix. I'm glad to see Toby's is equally wierd :)

So far we've lent our Firefly DVDs to four people. They all loved it. My brother was the latest. He watched the whole thing in one sitting. Couldn't stop.

And now it looks like the movie's going ahead:

Based on Whedon's cult hit television series Firefly (which has taken on a second life following its December 9, 2003, DVD release, winning new fans and critical praise worldwide), Serenity will continue and expand upon the adventures of the memorable characters launched in the series, who will be joined by new characters created expressly for the motion picture adaptation. Whedon will act as both screenwriter and director, with Barry Mendel (The Sixth Sense, Unbreakable and the upcoming Wes Anderson film, The Life Aquatic) producing and Chris Buchanan (president of Whedon's Mutant Enemy production company) and Alisa Tager (of Barry Mendel Productions) serving as executive producers. The Serenity cast will include such returning "Firefly" cast members as Nathan Fillion as Captain Malcolm 'Mal' Reynolds, Gina Torres as Zoe Warren, Morena Baccarin as Inara Serra, Jewel Staite as Kaylee Frye, Adam Baldwin as Jayne Cobb, Sean Maher as Dr. Simon Tam and Summer Glau as River Tam.

It's great that they're going to have the original cast, and I'm glad it's got such a good producer behind it (I really liked both The Sixth Sense and Unbreakable.)

I fixed another couple of small problems in PyPI yesterday - one of which was introduced in the last update ;)

• fixed deletion of packages where there were no versions (oops)
• list only new releases in RSS and front page, not any old edit

Admittedly the second one had been bugging me for a while, but I just never remembered to fix it.

MySQL really, really shits me sometimes. My latest:

[root@richardpc root]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3 to server version: 4.0.16-Max

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> drop database rounduptest;
Query OK, 0 rows affected (0.00 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> Bye
[root@richardpc root]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4 to server version: 4.0.16-Max

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> create database rounduptest;
ERROR 1007: Can't create database 'rounduptest'. Database exists
mysql> Bye

So until I can figure how to get rid of that database, my unit tests use "rounduptest2" for testing. Grr.

I've updated PyPI:

• add meaningful titles to pages (uses page heading)
• properly unquote version numbers for release editing page (bug #855883)
• allow removal of more than one release at a time
• make "delete whole package" a form button
• made wording of role admin link more helpful
• hide all current releases when a new release is added

So now the single-visible-release use-case is the one coded in. If you want to make more than one release visible, you need to go to the package's edit page (linked from the sidebar when you're logged in).

Oh, and PyPI has its own PyPI entry now too :)