Richard Jones' Log

I've been playing Half-Life 2 on and off since it was released. I've played it through twice, and have revisited several of the chapters numerous times. I particularly like going through (links go to walkthrough) Point Insertion, Water Hazard (so pretty), Highway 17 and Follow Freeman sections. Oh, and Our Benefactors, Nova Prospekt, Entanglement, Anticitizen One (Dog is so cool :), We Don't Go To Ravenholm, etc. :)

Xander got me Half-Life 2: Raising The Bar, which I had great fun reading.

The experience hasn't been perfect though - I've been occasionally hit with the "stuttering" problem reported early on. It manifests on my system as an occasional hit on the frame rate - a pause of about half a second, or sometimes several in a row. It was pretty obvious that there was some problems with disk access, loading new sounds to be played. A number of solutions were offered, and Valve have been working on making their code better (apparently in some cases removing a bajillion open() failures while opening some sound files -- I've been hit with that type of bug before myself :). Their latest fix helped my system a fair bit, but I still got the occasional "stuttering" in the trainstation opening sequence, when disembarking the train (one of the most common places, as there's a lengthy speech by Dr. Breen).

I'm on holidays at the moment, so I've had some time to really look into the problem. I poked around the various HL2 forums and found some clues as to the possibilities. I installed the latest beta nVidia drivers for my card, which helped a little. I installed the latest mobo drivers (I have on-board sound), which didn't seem to change anything. The main culprit turned out to be my slow disks (slow ATA-66) in combination with a fragmented filesystem. Once I defragmented* the filesystem, things sped up considerably, and I haven't noticed any "stuttering" since.

*: does the default Windows XP disk defrag program blow chunks or what? I looked around for a third-party tool, but they all appear to be "enterprise" level, hence expensive. In the end, I had to practically hand-hold (ie. run and re-run defrag, move some files around, re-run, move files off the disk and back on again) it so it would actually defrag some of the larger HL2 data files. Sigh.

Yeah, it bugged me too... so here's an eminently more readable version of tinyp2p.py. With a couple of bugfixes thrown in too :)

It's still a bit more compact than I guess I'd like, but it's certainly much easier to understand what the code does now.

# simplep2p.py v1.1 (C) 2004, Richard Jones
# with litle touch from Leonardo Santagada
# A slightly more readable version of tinyp2p.py (C) 2004, E.W. Felten
# (also handles binary transmissions now too)
# license: http://creativecommons.org/licenses/by-nc-sa/2.0
# Usage:
#   python simplep2p.py password server hostname portnum [otherurl]
# or
#   python simplep2p.py password client serverurl pattern
import os, SimpleXMLRPCServer, re, hmac, sets, base64
from sys import argv
from xmlrpclib import ServerProxy

def gen_password(url):
   return hmac.new(argv[1], url).hexdigest()
def ls(pat=""):
    ''' List the files in the current working directory that optionall match
    a regular expression "pat". '''
    return [fn for fn in os.listdir(os.getcwd())
        if not pat or re.search(pat, fn)]

if argv[2] == "server":
    my_url = "http://"+argv[3]+":"+argv[4]

    # keep a list of servers we know about
    servers = sets.Set([my_url] + argv[5:])
    def update_servers(new_servers=[]):
        servers.union_update(sets.Set(new_servers))
        return list(servers)
    def discover(other_url):
        if other_url == my_url: return servers
        pw = gen_password(other_url)
        server = ServerProxy(other_url)
        return update_servers(server.list_servers(pw, update_servers()))

    # ask all our known servers about the servers *they* know about
    if servers: [discover(url) for url in list(servers)]

    # serve up the files
    def list_servers(password, arg=[]):
        if password == gen_password(my_url):
            return update_servers(arg)
    def list_files(password, arg):
        if password == gen_password(my_url):
            return ls(arg)
    def get_file(password, arg):
        if password == gen_password(my_url):
            f = file(arg)
            try:
                return base64.encodestring(f.read())
            finally:
                f.close()
    server = SimpleXMLRPCServer.SimpleXMLRPCServer((argv[3], int(argv[4])))
    server.register_function(list_servers)
    server.register_function(list_files)
    server.register_function(get_file)
    server.serve_forever()

# client - contact our server
for url in ServerProxy(argv[3]).list_servers(gen_password(argv[3])):
    # ask for the files we want, that we don't already have
    files = sets.Set(ServerProxy(url).list_files(gen_password(url), argv[4]))
    my_files = sets.Set(ls())
    for fn in files - my_files:
        # and fetch
        c = ServerProxy(url).get_file(gen_password(url), fn)
        f = file(fn, "wb")
        f.write(base64.decodestring(c))
        f.close()

Update: v1.1 incorporates some feedback from Leonardo Santagada.

Note: simplep2p.py will not talk with tinyp2p.py servers.

Spurred on my my lightning talk at the OSDC, I've been hacking on the ufo code again.

Developing OpenGL apps on a powerbook is really neat, but damn these things get hot when you work them.

(Yes, one is enough.)

Getting back into the swing of posting to this log... here's my variant on the usual progress display. Percentage with an ETA if it appears to be a long-running task. Examples at the end of the code. And hey, it's my first iterator (using __iter__ and not the old-school __getitem__, that is ;)

import sys,time

class Progress:
    '''Progress display for console applications.

    See __main__ block at end of file for sample usage.
    '''
    def __init__(self, info, sequence):
        self.info = info
        self.sequence = iter(sequence)
        self.total = len(sequence)
        self.start = self.now = time.time()
        self.num = 0
        self.stepsize = self.total / 100 or 1
        self.steptimes = []
        self.display()

    def __iter__(self): return self

    def next(self):
        self.num += 1

        if self.num > self.total:
            print self.info, 'done', ' '*(75-len(self.info)-6)
            sys.stdout.flush()
            return self.sequence.next()

        if self.num % self.stepsize:
            return self.sequence.next()

        self.display()
        return self.sequence.next()

    def display(self):
        # figure how long we've spent - guess how long to go
        now = time.time()
        steptime = now - self.now
        self.steptimes.insert(0, steptime)
        if len(self.steptimes) > 5:
            self.steptimes.pop()
        steptime = sum(self.steptimes) / len(self.steptimes)
        self.now = now
        eta = steptime * ((self.total - self.num)/self.stepsize)

        # tell it like it is (or might be)
        if now - self.start > 3:
            M = eta / 60
            H = M / 60
            M = M % 60
            S = eta % 60
            s = '%s %2d%% (ETA %02d:%02d:%02d)'%(self.info,
                self.num * 100. / self.total, H, M, S)
        else:
            s = '%s %2d%%'%(self.info, self.num * 100. / self.total)
        sys.stdout.write(s + ' '*(75-len(s)) + '\r')
        sys.stdout.flush()

if __name__ == '__main__':
    for i in Progress('Testing 3...', range(3)):
        time.sleep(5)
    for i in Progress('Testing 1645...', range(1645)):
        time.sleep(.01)

It looks like there's a couple of entries in the cookbook that produce progress display. While this one has additional features, I don't think it's worth submitting.

I've just released Roundup's 8th major feature release. This is a good day, as I've been wanting to release that sucker for a while now :)

My boss has given me the OK to go to PyCon next year. I've signed up to give a presenation on Roundup (which I just gave at the local Open Source Developers Conference to good review). I'm not sure whether I'll be there for any sprints - none are listed yet.

w00t

(separated from the other OSDC post so I don't sully it with langauge flamage ;)

I found it funny that during Damian Conway's closing keynote (which was brilliant, as were all his other talks that I attended - about 10 or so all up) there were little snippets - about half a line - of Perl code that he used as punchlines (along the lines of "and some people try this, which we can all see is totally misguided") ... except nobody in the audience laughs. It just takes too long to decode what's in front of them, figure out what it does - and which obscure Perl side-effect it's triggering - in time to get the joke. Hrm. Looking back, Damian had to spoon-feed the point to pretty much all his perl-code-based punchlines. The rest of the keynote was mostly language-agnostic, with Perl examples, and was quite brilliant, with a superb message about cleaning up your APIs (though a little too much emphasis on having your code use calling context to "guess" what the end-user-coder is trying to do with your API - one of my pet peeves with Perl).

Well, it's all over for this year. The Python track was a great success, with a bunch of interesting talks presented by able speakers. Lots of out-of-band discussions too. My Roundup talk went really well, with about half the audience not even being Python programmers (and I was up against a couple of interesting-looking PHP and Perl talks, so I feel chuffed :). I got lots of interest after the talk - so much that I didn't get to follow up on a few conversations I would've liked to.

I also did a couple of lightning talks - one on PyPI and another on game programming (the LD48 comp, pygame (sdl perl got a mention) and pyopengl). Actually, I really just wanted to play some video games I wrote on the big screen (about 7 meters across) even for just a brief time :) The PyOpenGL one got appropriate ooohs and aaahs from the audience :)

The conference organisers did an amazing job, and I reckon next year can only build on this year (I already have a number of talks planned). They video taped all presentations, which they plan to have available for download or possibly purchase (on DVD) some time in the new year (there's a huge amount of tape to edit).

Next up for me looks like being PyCon - my boss has given a tentative OK, but would like me to present something. I've just got to clear up whether a Roundup talk would be OK, or whether I'd have to do something more work-based. Then I'll submit a proposal.

The conference will be held next year around the same time, probably in the same place in about the same format. What we need is to be able to expand our advertising reach (without having any money to do so ;) so we can reach more developers - not necessarily just Australian or Open Source ones too. Suggestions welcome.