Richard Jones' Log: Sane Python application packaging?
I've got an application that I'd like to make more user-friendly to install.
- On OS X and Windows applications have everything (program, libraries, data) bundled. This is good as it removes issues around library compatibility and installation. In a previous job I've handled installing on these platforms but I also had ready access to them to develop & test on and now as a FOSS developer I don't.
- On OS X I know about py2app. Since there's no "installation" this is effectively done. That'll help.
- On Windows I know about py2exe but I still need to create an installer for the application (to add it to "Program Files", the start menu and possibly associate file extensions). There's a link to Inno Setup from the py2exe tutorial and I think it'll help.
- On Linux applications may be a stand-alone bundle but are encouraged to use a standard install pattern (programs in /usr/bin, libraries in /usr/lib/, data files in /usr/share). If I don't follow that pattern I won't get picked up by Linux distributions.
- In the Python world we have two technologies: distutils and setuptools. Parts of distutils are used in py2app and py2exe. At first glance distutils and setuptools appear to be suitable for the Linux case, except...
- Installing with distutils can cause problems if the internal library contents change, resulting in asking users to manually remove any old install.
- Python eggs, and the associated Python library search path / site.py shenanigans, cause developer and end-user pain when an application has its own local copy of an installed egg library. Many scary user warnings out on the console, and also possibly incorrect behavior. virtualenv can help solve this but would require bundling a build of Python with the application.
- Shared libraries can cause compatibility issues that I'd like to avoid.
So, does anyone have any advice?
- Is there a py2exe/py2app for Linux that I've not found? Do I have to use auto(conf|tools)?
- I don't believe I should package Python 2.5 but if I'm not using a "local" version then I've a good chance of running into site-packages egg issues and I'd rather not have to:
import sys sys.path = [e for e in sys.path if not e.endswith('site-packages') and not e.endswith('.egg')]at the start of my application. - Also, are there handy systems I could access to run my py2app/py2exe?
Aside: having just finished writing this post, proof-reading it and looking for one more link I stumbled upon this and this and the huge discussion that lead up to those. The discussion appears to be focusing on packages (libraries) rather than applications.
A quick tip: Do NOT use setuptools. It's a pain and it doesn't unpack your package properly. Save yourself from a lot of headaches.
The only reliable way of packaging an application so that it works on all flavours of Linux is to give a source tarball with INSTALL.txt detailing the instructions (install library X, Y, Z, install Python 2.U, install Python libraries V, W). There's no portable binary application format (LSB is trying to define one, based on RPM, but I haven't heard about anyone using it to distribute anything).
Obviously, installing from source is inconvenient for the users. The users want packages (rpm, deb) that work on their Linux distro of choice (Fedora, OpenSuSE, Ubuntu). Since the source is available, you don't have to do all the packaging yourself, but it would help (at least initially, to get people to try it out so that a demand appears for the distros to have it).
This whole scheme doesn't work well for closed-source packages.
Regarding setuptools, I've come to believe that it's a step in the right direction (and at least five more steps in some wrong directions). Having your dependencies explicitly listed in setup.py is Good (but fails when you need to depen on non-Python things like C libraries). Using the binary '.egg' files is Bad. The 'easy_install' command is Evil (willfully violates the FHS by installing stuff into /usr/lib, which is the exclusive domain of the distribution package management system, i.e. dpkg or rpm). People have figured out ways (virtualenv, zc.buildout) of using setuptools that work reasonably well for certain kinds of applications (web application servers, usually) in the hands of experienced developers (who know how to get the native system packaging tools to install the requisite C libraries headers for the extension modules that setuptools will try to compile after fetching the sources from PyPI), but not so much for end-users who just want to click and play.
There is no easy answer to this, because the question is wrong: You should ask, if there is a solution for Windows, Mac, Ubuntu, Debian, SUSE, RedHat, Gentoo, Slackware, ... Because from the packaging point of view there is no Linux solution. So the question is: for which distribution do you want to create a package? If it's for Debian or Ubuntu (or one of the other 115 Debian based Distros listed on Distrowatch.org) then you should create a .deb package. For these distributions, a deb package will be much better then any egg or source package. (I guess the commenter who told you to create a source package with information meant that others will then eventually create a deb for you). The debian package management will handle all the dependencies and you don't have to worry about changing or missing libraries.
If you want a package for the 16 RedHat based distros, then an rpm is best, although I can not tell you how good this will handle dependencies, since I don't use it.
If this is too much work, you might want to try the OpenSUSE build service:
http://en.opensuse.org/Build_Service
which is meant to easily build packages for SUSE, Debain, RedHat, Mandriva, Ubuntu.
You might also like to read this:
http://np237.livejournal.com/21033.html
Good luck :-)
You could try using paver[1] and deploy to a virtual env on linux? I know it's not installing in to /usr/bin etc but using paver you will deploy into a virtual env on ANY linux distro, BSD & Solaris. Anyway, good luck, deploying any app on unix is always a series of compromises and everyone likes to bike shed.
[1] http://www.blueskyonmars.com/projects/paver/
I'll try to create .debs for Bruce and its dependencies, then convert with alien and upload for tests.
Even if I'm not skilled enough to pull it off, the basic idea is:
Linux distros already have the concept of 'installing an application' fleshed out. Dependencies, versions, data/docs/libs dirs, etc. So I'll try to convert to one format that handles all this (.deb, using stdeb). Then I'd check whether an automatic tool (alien) can convert to usable files for other distros (RPMs).
Building .debs (using stdeb) was pretty easy, doing it right might be more involved but these files install (and uninstall!) fine in Ubuntu 8.04 (Hardy):
RPM generation from these .debs was easy too.
Richard, the only suggestion I make to ease packaging is adding more info in standard ways: LICENSE and install_requires (instead of in README.txt)
Here at work we're using bbfreeze to succesfully "freeze" distribute a python application which makes heavy use of pkg_resources (eggs) and C extensions. It's currently only distributed to windows clients but we're planning on creating installers for linux and mac osx too. It's the only option I managed to find that correctly handles the pkg_resources API.
http://pypi.python.org/pypi/bbfreeze/0.95.4
I've been saying for years that Python applications are too difficult to distribute and install, but alas it's way to easy to ignore me.
Much of the emphasis in packaging of Python solutions is essentially about adding functionality to an existing Python installation. This doesn't help someone with an application to distribute (or, for that matter, someone wishing to install a Python application).
If you find a suitable solution you will be doing Python a huge favor.