Richard Jones' Log: Please, Linux vendors, fix your Python!

I don't think any of the BSDs make this mistake, and OS X doesn't either (though it makes some others).

Yup, FreeBSD does this correctly - I'd no idea it was different on Linux, good to know.

I think this definitely stems from an over-adherence to packaging policies, most linux vendors like to package stuff into a foo, foo-dev and a foo-doc (with things like foo-bin, foo-i-have-too-much-time-on-my-hands appearing too). To a linux packager python's distutils and .h stuff looks like something you'd never use unless you are developing stuff, which is a narrow view of things. They just think of python like a shared library, unless you are compiling against that lib you'd never want the dev package.

Since most linux vendors tend to copy each other's packaging, it probably took only one to start this python trend.

SuSE's python always bugged me in particular, they've split it into an absurd amount of packages, I always seem to be missing one or other of the core python components. Not to mention some slightly flakey patches which rendered zope inoperable with SuSE's python for ages. Even better, re-building SuSE's python package requires a huge amount of deps, as they build every optional component they can think of (placing each into it's own python-component package of course).

Debian appears to do this "right", mostly. If you're expecting to see header files, though, you'll definitely need to install the pythonX.Y-dev package(s). In that sense, building C extensions with distutils requires another package. This is a model that most developers on Linux distributions are used to. "Let's see. I need to compile X. That means I'll need the headers and libraries that X depends upon. Time to install the -dev package. Since the -dev package depends upon the shared libraries, I won't have to worry about that; apt-get will handle the dependencies..."

"Debian appears to do this "right", mostly."

Oh, I know of one person who very recently had a WTF moment with respect to this erroneous packaging micromanagement, so "right" is very much a subjective term. Half the point of Python is that you can use it interactively and develop programs yourself. Snapping bits off to shave a few K off the base package (when some bloated "essential" GUI package probably dwarfs it a thousand times over) makes virtually no sense.

"If you're expecting to see header files, though, you'll definitely need to install the pythonX.Y-dev package(s). In that sense, building C extensions with distutils requires another package. This is a model that most developers on Linux distributions are used to. "Let's see. I need to compile X. That means I'll need the headers and libraries that X depends upon. Time to install the -dev package."

Well, my policy is to install everything these days to avoid such horsing around, but I'd rephrase the developers thought process to something like this: "I need to compile X. Broken? Time to install n different -dev packages for no sensible reason whilst cursing the idiot packagers. Thank you for not getting it, paper pushers!"

FWIW, Conectiva Linux and Mandrake Linux (and now Mandriva Linux) do include distutils in the main package.

Hi, I've been developing for that purpose a kind of virtual python machine (pyvm). Although it runs on suse 9.2 and fedora 3 I think it could be a solution to broken distro packaged python. It doesn't require any administrative account. I hope that will help someone with the same problems I had.

You do *not* need distutils on every host in order to install python packages. You need distutils on *one* host: your build/development host. You then distribute the binary package you built to your other hosts. This is why distutils has the "bdist" target. For example, to build an RPM of a python package like so: "python setup.py bdist_rpm"

Folks, not every computer is used for development. Hosts in a DMZ, for example, do not get development packages to make it that much harder for potential intruders to build their rootkits. Critical servers may be pared down to the absolute minimum package list to minimize re-install time. Embedded or small footprint devices (PDAs, ramdisk based hosts) do not need unnecessary development related files consuming their precious space. Web kiosks and other turnkey systems do not need development packages, as no one will ever be able to get to a shell prompt.

The maintainers of the distros you are complaining about understand that their distro will be used for many different purposes and have accommodated that. And systems and network administrators, embedded device developers, etc. greatly appreciate it.

If you know you're going to be doing development on a host, install the *-devel packages, which are usually all bundled in some "Development" group, so that at install time, you can select that group and get all the *-devel packages. In fact, hard drive prices and sizes being what they are, just install every package in the distro, and you know you will be OK.

I'm siding with Chris on this. Distutils is not required for the Python runtime, it is used for building Python modules and therefore is extra baggage for the base Python package. On an RPM-based system you should never be doing "python setup.py install", you should be installing the resulting RPM from "python setup.py bdist_rpm", which can be installed on a system without Distutils installed.

Look at it this way. If you include Distutils into the base Python package, users who do not wish to build packages on their systems (security-oriented deployments, memory/disc constrained systems) have no choice. If you split out Distutils into a separate package, users who need it can easily install it.

Perhaps what you should be complaining about is that the package selection you made did not include python-devel? I don't know exactly where it is or is not included, but perhaps python-devel should be included in more of the package collections available in the installer, that way it would show up on more default installs and could get pruned down by people who really need minimal sets. Of course, there's only so much stuff that can be put on the first CD, and it's nice to be able to have the minimal install on Red Hat/Fedora only require the first CD.

Sean

I recently encountered a Python packaged without a working distutils for the first time (on Fedora Core 2), and was fairly disappointed by the way the packages had been split up like this. Now, maybe it's distutils' fault that it needs information about the Python build system, but it shouldn't really need python-devel for pure Python modules when it is just building them.

And don't get me started on these distros that still fail to supply a shared library for Python...

"""Distutils is not required for the Python runtime, it is used for building Python modules and therefore is extra baggage for the base Python package."""

Bzzzt. Sorry, wrong answer. Distutils is part of the Python *standard library*, and programs written in Python are well within their rights to import things from the distutils and use them. It contains a logging module, a textwrapping module, an archive utilities module, and half a dozen other things that don't have anything to with "developing" or "building" Python modules.

(The security argument is also bogus, by the way, since the distutils don't allow you to do anything you couldn't do by writing the same code by hand.)

Header files? Okay, *maybe* you could make an argument that they're only needed for building extensions, and that they're not part of the standard library. But distutils? Sorry, it's standard library and is required by definition. If you don't include it, what you've got isn't a valid Python installation.

A long time ago, it was deliberately decided to keep distutils separate in SUSE. However it looks like this will change in the future.