Richard Jones' Log

This is part three of my continuing adventure in packaging my Python application.

Thanks again for the advice given in response to my last post on Linux-specific application packaging issues. I've decided to leave packaging up to the experts and help them as much as possible. To this end I've:

1. split the installation instructions out of README.txt into INSTALL.txt and put in a notice that the INSTALL.txt is not for regular users,
2. looked into how to best support more organised system package maintainers (the INSTALL.txt is my start at that), and
3. generated stand-alone application downloads for Linux, Windows and OS X.

The approach I've taken for the stand-alone packages is as follows:

1. bundle all of bruce, pyglet, cocos, docutils (including roman.py) and pygments into a library zip file,
2. create a simple script for each O/S to run bruce using that zip file, and
3. create the distribution zip files with those contents alongside the README.txt, HOWTO.txt and PKG-INFO.

Since the examples directory is pretty big I've also created a separate "examples" zip file.

This is implemented as a new setup.py command "build_apps" (as BuildApps.) I didn't use zipfile's PyZipFile but rather implemented my own. PyZipFile only includes compiled modules (.pyc or .pyo). The problem I have with this is that the compiled modules are not compatible across Python minor releases (modules compiled for Python 2.5 are not compatible with 2.6 and vice versa.) I would have to include separate library zip files for each system interpreter I wished to support. After some very simple checks I determined that the performance difference is negligible if I just bundle the original .py files instead.

I wrote "build_apps" so it'd fake being an "sdist" command enabling me to upload the resultant file to the Python Package Index (PyPI). This part is really quite inelegant but I'd like to keep the downloads all in one place. Unfortunately the file listing in PyPI lists the files as "Source" which isn't good. PyPI has no concept of an "application" though.

The Windows and OS X application bundles are completely untested.

I intend to investigate py2app and py2exe in the longer term to produce more system-friendly programs (no need to install Python, have icons, etc) but for now I believe this solution is good enough.

Just focusing on the Linux side for a moment.

Thanks to everyone who responded to yesterday's post about Python application packaging. My efforts will now be focused on creating files suitable for Linux package managers. This would initially be Debian software package (.deb) and Red Hat Package Manager (.rpm) - I considered Gentoo Linux's ebuild but really the audience for that is too narrow and they have tools to convert rpm and deb files to their own format.

Tasks for my application:

1. I'll have to organise my install and generate meta-data files according to the two formats: Debian package basics and maintainer's guide (which may differ from Ubuntu's) and RPM package basics,
2. Generate the package files: .deb via stdeb (there's also easy-deb) and .rpm using the built-in distutils bdist_rpm), and
3. Host the files somewhere.

If you don't want to use stdeb or easy-deb there's a nice ShowMeDo screencast of creating a .deb for a Python program by Horst Jens.

One of the commenters mentioned above, Daniel, has already produced .deb files using stdeb. I say files because he produced one each for pyglet, cocos2d, and bruce. The docutils and pygments libraries are already available in Ubuntu's package repository.

Already I have a problem: Ubuntu also packages a version of pyglet. The pyglet version packaged is 1.0 which is quite out of date compared to the version I'd like to package, 1.1 (or even 1.2 pre-release). If I install the Ubuntu version and then try to upgrade to Daniel's version I get the rather unfriendly error:

richard@shiny:/tmp$ sudo dpkg -i python-pyglet_1.1.2-1_all.deb
(Reading database ... 165323 files and directories currently installed.)
Preparing to replace python-pyglet 1.0.dfsg-1 (using .../python-pyglet_1.1.2-1_all.deb) ...
Unpacking replacement python-pyglet ...
Setting up python-pyglet (1.1.2-1) ...
pycentral: pycentral pkginstall: not overwriting local files
pycentral pkginstall: not overwriting local files
dpkg: error processing python-pyglet (--install):
subprocess post-installation script returned error exit status 1
Processing triggers for python-support ...
Errors were encountered while processing:
python-pyglet

Removing the existing python-pyglet package resolves this issue of course. I don't know what the state is for other Linux distributions.

So, there's still some issues:

1. There's a potential issue when a Linux distribution starts packaging bruce, pyglet and cocos2d itself. As mentioned above there's already one conflict on Ubuntu.
2. I've still got an issue with compatibility. While I control the package files that's not a big deal but once the distributions start doing so I no longer have any control, and
3. Is it better to require users to download .deb files or would it be better to be able to just add "deb http://pypi.python.org/debian/ pypi" to Debian's apt configuration, and whatever the similar configuration is for rpm-based systems (yum or urpmi)?

I also noticed getdeb and CNR (click n run) while poking around. They could be good application promotion tools. There's probably others.

(Updated to include link to Ubuntu packaging guide)

I've got an application that I'd like to make more user-friendly to install.

1. On OS X and Windows applications have everything (program, libraries, data) bundled. This is good as it removes issues around library compatibility and installation. In a previous job I've handled installing on these platforms but I also had ready access to them to develop & test on and now as a FOSS developer I don't.
2. On OS X I know about py2app. Since there's no "installation" this is effectively done. That'll help.
3. On Windows I know about py2exe but I still need to create an installer for the application (to add it to "Program Files", the start menu and possibly associate file extensions). There's a link to Inno Setup from the py2exe tutorial and I think it'll help.
4. On Linux applications may be a stand-alone bundle but are encouraged to use a standard install pattern (programs in /usr/bin, libraries in /usr/lib/, data files in /usr/share). If I don't follow that pattern I won't get picked up by Linux distributions.
5. In the Python world we have two technologies: distutils and setuptools. Parts of distutils are used in py2app and py2exe. At first glance distutils and setuptools appear to be suitable for the Linux case, except...
6. Installing with distutils can cause problems if the internal library contents change, resulting in asking users to manually remove any old install.
7. Python eggs, and the associated Python library search path / site.py shenanigans, cause developer and end-user pain when an application has its own local copy of an installed egg library. Many scary user warnings out on the console, and also possibly incorrect behavior. virtualenv can help solve this but would require bundling a build of Python with the application.
8. Shared libraries can cause compatibility issues that I'd like to avoid.

So, does anyone have any advice?

1. Is there a py2exe/py2app for Linux that I've not found? Do I have to use auto(conf|tools)?
2. I don't believe I should package Python 2.5 but if I'm not using a "local" version then I've a good chance of running into site-packages egg issues and I'd rather not have to:

   import sys
   sys.path = [e for e in sys.path
       if not e.endswith('site-packages') and not e.endswith('.egg')]

   at the start of my application.
3. Also, are there handy systems I could access to run my py2app/py2exe?

Aside: having just finished writing this post, proof-reading it and looking for one more link I stumbled upon this and this and the huge discussion that lead up to those. The discussion appears to be focusing on packages (libraries) rather than applications.

Update: I've had to withdraw the presentation due to a clash of scheduling that means I won't be able to attend PyCon 2009.

Yes, just the one :)

The summary as submitted was:

I will talk about the way Python is used in the operations of ekit.com Inc, a telecommunications provider for travellers. I will briefly touch on the overall architecture before delving into some specific examples of how Python is being used to rapidly develop robust and flexible systems. This talk is meant to be entertaining, informative and possibly instructive.

My thanks go to the thoughtful comments and reviews from the program committee which helped focus the proposal and should help guide final delivery.

Python 3.0 that is.

As someone who has to deal with unicode and charset issues I'm really looking forward to the day I can ditch the current two string types and all the confusion and bugs that arise from them. Unicode / encoding will be so much easier to explain now!

I also learnt something new in Anthony's "what's new in Python" talk at OSDC: comprehensions are everywhere now!

>>> [chr(c) for c in (83, 80, 65, 77)]
['S', 'P', 'A', 'M']
>>> {chr(c) for c in (83, 80, 65, 77)}
{'A', 'P', 'S', 'M'}
>>> {c:chr(c) for c in (83, 80, 65, 77)}
{80: 'P', 65: 'A', 83: 'S', 77: 'M'}
>>> ''.join(chr(c) for c in (83, 80, 65, 77))
'SPAM'

That's awesome (for those new to things, that's a list, set, dictionary and generator comprehension).

Also, kudos to Ubuntu. "apt-get install python3" worked, and more importantly didn't replace the "python" command (rather requiring "python3" to invoke it). Unfortunately the same isn't true under Windows where the installer replaces the existing ".py" file association to point to Python 3 instead of 2.6, thus breaking all installed Python scripts. Boo.

The old Shuttle HTPC which had a troubled life finally died a week or so ago (no longer recognised the boot disk.)

I've finally settled on its replacement, detailed in this post in the forums of an enthusiastic Australian HTPC community. Don't be fooled by the Windows branding all over it - Linux gets quite a fair dealing in there and I've been given a lot of good advice regarding mythtv (using mythbuntu) and XBMC.

As a bonus, the new hardware is almost half the price of the old. Way under $1000. It'll be more future-proof and repairable (being standard components). And it'll look nicer too.

ps. never, ever buy a Shuttle unless it's covered by a really good warranty with full replacement.

Yeah, this blog still exists... and here's a meme to prove it...

(all the cool kids are doing it, apparently...)

Taken with the crappy webcam in my Dell laptop using the KDE screenshot tool to capture the window contents of the configuration panel of Kopete and using the Gimp to crop out the picture. I did have to play with the color balance of the webcam so I didn't come out too purple. I hope that's not against "the rules."

I just realised that I released the latest version of Roundup (1.4.5, a maintenance release) 7 years almost to the day after my first public release (0.1.1, released on the 18th of August, 2001.)

Just setting up some convenience aliases so I can deal better with timezones, I discovered this ...

rjones@l-rjones:$ TZ=BST date
Wed Aug 20 05:13:00 UTC 2008
rjones@l-rjones:~$ TZ=GB date
Wed Aug 20 06:13:40 BST 2008
rjones@l-rjones:~$

So it doesn't recognise the timezone TZ=BST but it displays the Great Britain time as "BST" :)

The convenience aliases, BTW, are:

alias edate="TZ=US/Eastern date"
alias wdate="TZ=US/Western date"
alias ukdate="TZ=GB date"

'cos I'm sick of trying to do this in my head (and getting it wrong) and because the few GUI programs out there suck.

I've been tasked to do some performance analysis on our new Travel Journal product (now "officially" with Fire Eagle support). The last time I worked for ekit I did a bunch of website performance testing and improvements.

I even wrote a tool to help me at the time! Last touched 5 years ago, and still at "rc1" :)

I'm now updating it (to Python 2.3 heh) and will release that 1.0 final that should've been released back then...

Excellent timing too, with the next challenge in a month's time!

If you're not already subscribed, head over to the Python Magazine site to read the article.

Bruce, the Presentation Tool has grown syntax-highlighting support thanks to Pygments. It's optional, and used if the presentation has either a doctest block or a ".. code:: <language>" block.

Bruce the Presentation Tool

Bruce the Presentation Tool is for people who are tired of fighting with presentation tools. Presentations are composed (edited) in plain text files. In its basic form it allows text, code, image, interative Python sessions and video. It uses pyglet to render in OpenGL.

Get it from PyPI

.. load-style:: big-centered

Text displayed centered on the default white background.

----

A new page, separated from the previous using the four
dashes.

Ut enim ad minim veniam.

A Page Title
------------

Pages may optionally have titles which are displayed
centered at the top by default.

Oh for fucks sake:

Victorian Premier John Brumby says he probably leans towards saying that Bill Henson's controversial photographs of naked youths "crossed the line" [from art to pornography].

Speaking on radio 3AW today, Mr Brumby said he had not seen the photographs.

Anyone who has seen the one (NSFW) image from the Roslyn Oxley9 Gallery exhibition available to the public (or any of his prior exhibitions, apparently) can attest that there's absolutely nothing you could describe as "sexualisation".

Bruce, the Presentation Tool is getting incrementally better. It's a nice little hobby project because there's always just a little more that could be done, and never a huge task at any given step. Thus I can do a little each time when I'm on the train (30 minutes of hacking on my EEE PC).

ReStructuredText is now the only input format - the old "Socrates"-based implementation is gone. Everything that it could do in terms of formatting may now be done in ReST - and more.

Some of the stuff I mentioned in my last post has been resolved - though it's not final I'm definitely leaning towards staying with directives as they give me more, and Bruce will be able to be more intelligent at producing HTML than the generic html4css2 writer (when dealing with Bruce presentations). Title-less pages are there, so are video pages. Titles are incorporated into the page decoration, allowing some nice tricks. Enumerated list generation handles more styles.

Next up I'll be looking at reintroducing the python interpreter and python code page types - but in the New Bruce they're actually just page elements that may form part of any page.

ps. for the people who asked to see it in action - grab the SVN and run something like "python -m bruce examples/example.rst" or any of the .rst in the examples directory. Bruce is being developed against the pyglet SVN HEAD though (it might work against 1.1, I'm not sure)

As previously hinted Bruce, the Presentation Tool may now display presentations authored in ReStructuredText. I've had a chance to do some more work on it lately, and need to take a step back and think about what I'm trying to do :)

The ReST capabilities currently available are:

1. Sections denote pages (just like all the other ReST presentation tools),
2. Lists are handled (some features missing). A lot of inline markup is handled. Images are handled, both inline and stand-alone.
3. The stylesheet and other configuration may be changed on the fly with ".. config::" directives.
4. The background decoration* may be specified with a ".. decoration::" directive.

*: the decoration stuff is new as of very recently too. Currently it controls the background colour, but also allows rendering of quads (with colour gradient if you like) and images in the background. There's still much to do like scaling the decoration layer to the screen size, and adding more toys to decorate with like lines and possibly splines. Not sure how far to take it.

Missing from the ReST side though is:

1. Pages without titles (this would require some sort of "page" directive to indicate a new page has begun).
2. Other page types like the Python interpreter, Python code and Video.
3. Handling notes and running-sheet HTML generation sensibly.
4. Allowing custom page types, perhaps through ".. custom:: <module name>"

Those things aren't insurmountable. I'm becoming increasibly convinced that ReST is a better way to go than the custom markup format, but I'm having trouble with the final decision to give up on the old format.

Of course maintaining two parsers is ... silly.

I'm pretty sure I've made my decision, but thought I'd throw this post out anyway in case anyone had any thoughts or encouragement...

PyWeek 6 is all done. The entries are in and rated. It's been another great PyWeek with continued growth in participation and games produced at the end. So many games that I'm considering extending the two weeks for judging!

Greg Ewing has proposed that there be an additional Pyggy Awards challenge which encourages developers to take games produced during the latest PyWeek and improve on them over three months. At the end we run a new round of awards.

python -m bruce presentation.rst

The 6th Python Game Programming Challenge starts this weekend. Theme voting is underway and there's still plenty of time to sign up. Come along and join in the fun!

I'm unwinding from PyCon in my cousin's apartment in Pittsburgh. Mostly this involves catching up with Nat, trying to get some sleep (unsuccessfully - I seem to have some sort of delayed-jetlag insomnia), catch up on email (partially successfully) and dump a bunch of photos from the camera. Photos have been updated and I now have a new title image on my blog page (thanks Toby for the old one :). I've compiled all the panoramas that were worth it and I think there's some nice ones in there. I got some nice photos of Evelyn and the PyAr guys from our downtown touristing (somehow Gui managed to dodge the camera... will have to wait to see Lucio's photos).

It's snowing outside.

Can't wait to be home.

The conference itself met and exceeded expectations. I saw some enlightening talks had some great conversations with some bright people. As already mentioned pyglet was a hit. Bruce was too, which was fun. I think I may have a contributor in Brian Dorsey, who was really nice and had some ideas and made the "python code" (in a presentation page write/edit a python program and hit F4 to execute it) Bruce page work.