Richard Jones' Log

Just focusing on the Linux side for a moment.

Thanks to everyone who responded to yesterday's post about Python application packaging. My efforts will now be focused on creating files suitable for Linux package managers. This would initially be Debian software package (.deb) and Red Hat Package Manager (.rpm) - I considered Gentoo Linux's ebuild but really the audience for that is too narrow and they have tools to convert rpm and deb files to their own format.

Tasks for my application:

1. I'll have to organise my install and generate meta-data files according to the two formats: Debian package basics and maintainer's guide (which may differ from Ubuntu's) and RPM package basics,
2. Generate the package files: .deb via stdeb (there's also easy-deb) and .rpm using the built-in distutils bdist_rpm), and
3. Host the files somewhere.

If you don't want to use stdeb or easy-deb there's a nice ShowMeDo screencast of creating a .deb for a Python program by Horst Jens.

One of the commenters mentioned above, Daniel, has already produced .deb files using stdeb. I say files because he produced one each for pyglet, cocos2d, and bruce. The docutils and pygments libraries are already available in Ubuntu's package repository.

Already I have a problem: Ubuntu also packages a version of pyglet. The pyglet version packaged is 1.0 which is quite out of date compared to the version I'd like to package, 1.1 (or even 1.2 pre-release). If I install the Ubuntu version and then try to upgrade to Daniel's version I get the rather unfriendly error:

richard@shiny:/tmp$ sudo dpkg -i python-pyglet_1.1.2-1_all.deb
(Reading database ... 165323 files and directories currently installed.)
Preparing to replace python-pyglet 1.0.dfsg-1 (using .../python-pyglet_1.1.2-1_all.deb) ...
Unpacking replacement python-pyglet ...
Setting up python-pyglet (1.1.2-1) ...
pycentral: pycentral pkginstall: not overwriting local files
pycentral pkginstall: not overwriting local files
dpkg: error processing python-pyglet (--install):
subprocess post-installation script returned error exit status 1
Processing triggers for python-support ...
Errors were encountered while processing:
python-pyglet

Removing the existing python-pyglet package resolves this issue of course. I don't know what the state is for other Linux distributions.

So, there's still some issues:

1. There's a potential issue when a Linux distribution starts packaging bruce, pyglet and cocos2d itself. As mentioned above there's already one conflict on Ubuntu.
2. I've still got an issue with compatibility. While I control the package files that's not a big deal but once the distributions start doing so I no longer have any control, and
3. Is it better to require users to download .deb files or would it be better to be able to just add "deb http://pypi.python.org/debian/ pypi" to Debian's apt configuration, and whatever the similar configuration is for rpm-based systems (yum or urpmi)?

I also noticed getdeb and CNR (click n run) while poking around. They could be good application promotion tools. There's probably others.

(Updated to include link to Ubuntu packaging guide)

I've got an application that I'd like to make more user-friendly to install.

1. On OS X and Windows applications have everything (program, libraries, data) bundled. This is good as it removes issues around library compatibility and installation. In a previous job I've handled installing on these platforms but I also had ready access to them to develop & test on and now as a FOSS developer I don't.
2. On OS X I know about py2app. Since there's no "installation" this is effectively done. That'll help.
3. On Windows I know about py2exe but I still need to create an installer for the application (to add it to "Program Files", the start menu and possibly associate file extensions). There's a link to Inno Setup from the py2exe tutorial and I think it'll help.
4. On Linux applications may be a stand-alone bundle but are encouraged to use a standard install pattern (programs in /usr/bin, libraries in /usr/lib/, data files in /usr/share). If I don't follow that pattern I won't get picked up by Linux distributions.
5. In the Python world we have two technologies: distutils and setuptools. Parts of distutils are used in py2app and py2exe. At first glance distutils and setuptools appear to be suitable for the Linux case, except...
6. Installing with distutils can cause problems if the internal library contents change, resulting in asking users to manually remove any old install.
7. Python eggs, and the associated Python library search path / site.py shenanigans, cause developer and end-user pain when an application has its own local copy of an installed egg library. Many scary user warnings out on the console, and also possibly incorrect behavior. virtualenv can help solve this but would require bundling a build of Python with the application.
8. Shared libraries can cause compatibility issues that I'd like to avoid.

So, does anyone have any advice?

1. Is there a py2exe/py2app for Linux that I've not found? Do I have to use auto(conf|tools)?
2. I don't believe I should package Python 2.5 but if I'm not using a "local" version then I've a good chance of running into site-packages egg issues and I'd rather not have to:

   import sys
   sys.path = [e for e in sys.path
       if not e.endswith('site-packages') and not e.endswith('.egg')]

   at the start of my application.
3. Also, are there handy systems I could access to run my py2app/py2exe?

Aside: having just finished writing this post, proof-reading it and looking for one more link I stumbled upon this and this and the huge discussion that lead up to those. The discussion appears to be focusing on packages (libraries) rather than applications.

Update: I've had to withdraw the presentation due to a clash of scheduling that means I won't be able to attend PyCon 2009.

Yes, just the one :)

The summary as submitted was:

I will talk about the way Python is used in the operations of ekit.com Inc, a telecommunications provider for travellers. I will briefly touch on the overall architecture before delving into some specific examples of how Python is being used to rapidly develop robust and flexible systems. This talk is meant to be entertaining, informative and possibly instructive.

My thanks go to the thoughtful comments and reviews from the program committee which helped focus the proposal and should help guide final delivery.

Python 3.0 that is.

As someone who has to deal with unicode and charset issues I'm really looking forward to the day I can ditch the current two string types and all the confusion and bugs that arise from them. Unicode / encoding will be so much easier to explain now!

I also learnt something new in Anthony's "what's new in Python" talk at OSDC: comprehensions are everywhere now!

>>> [chr(c) for c in (83, 80, 65, 77)]
['S', 'P', 'A', 'M']
>>> {chr(c) for c in (83, 80, 65, 77)}
{'A', 'P', 'S', 'M'}
>>> {c:chr(c) for c in (83, 80, 65, 77)}
{80: 'P', 65: 'A', 83: 'S', 77: 'M'}
>>> ''.join(chr(c) for c in (83, 80, 65, 77))
'SPAM'

That's awesome (for those new to things, that's a list, set, dictionary and generator comprehension).

Also, kudos to Ubuntu. "apt-get install python3" worked, and more importantly didn't replace the "python" command (rather requiring "python3" to invoke it). Unfortunately the same isn't true under Windows where the installer replaces the existing ".py" file association to point to Python 3 instead of 2.6, thus breaking all installed Python scripts. Boo.