Python distutils installs into 'site-packages' instead of 'dist-packages' when a prefix is set

yes, this works as designed. --prefix always overrides the default that Debian's system python does set. If you want to install into /usr/local/lib/python2.6/dist-packages, just omit the --prefix.

> $ python setup.py install --prefix=/usr
>
> installs modules at '/usr/lib/python2.6/site-packages'.

you should not install into /usr/lib/python2.6 yourself. this has the potential to override system compnents.

see https://lists.ubuntu.com/archives/ubuntu-devel/2009-February/027439.html

> you should not install into /usr/lib/python2.6 yourself. this has the potential to override system compnents.

Indeed, to avoid this I was used to set the prefix to /usr/local and I recommended this also when giving instructions for installing a module - but this recommendation fails now on Jaunty, as /usr/local/lib/python2.6/site-packages is not in the sys.path anymore.

> see https://lists.ubuntu.com/archives/ubuntu-devel/2009-February/027439.html

I see the renaming to dist-packages is pretty useful. To be sure here: /usr/local/lib/python2.6/site-packages is reserved for custom Python installations and will not be part of the default sys.path anymore?

> To be sure here: /usr/local/lib/python2.6/site-packages is reserved for custom
> Python installations and will not be part of the default sys.path anymore?

yes.

Matthias: I think the problem here is that the logic for selecting between dist-packages and site-packages is not consistent. Ignoring --install-layout for a moment, distutils.sysconfig says, in effect:

  is_default_prefix = not prefix or os.path.normpath(prefix) in ('/usr', '/usr/local')
  if is_default_prefix and 'PYTHONUSERBASE' not in os.environ and 'real_prefix' not in sys.__dict__:
      return os.path.join(libpython, "dist-packages")

... while distutils.command.install says, in effect:

  if self.prefix_option or 'real_prefix' in sys.__dict__:
      self.select_scheme("unix_prefix")
  else:
      self.select_scheme("unix_local")

I think that we should be consistent here, and apply the same "is the prefix /usr or /usr/local?" check in distutils.command.install (and perhaps check PYTHONUSERBASE too, though that's a side issue) - if it is, we should use the unix_local scheme and thus install into dist-packages rather than site-packages.

This bug seems more important than wishlist - it breaks third-party build systems that use --prefix=/usr/local in the expectation that it will install things somewhere sensible where the version of python you're using will be able to get at them. (Such build systems can simply remove --prefix=/usr/local as a workaround, but with older packaged versions of python in Debian/Ubuntu that will install into /usr which is obviously undesirable.)

subscribing barry ...

yes, there is some kind of conflict.

we have to distinguish between a (manual) python installation going to /usr/local by default and using /usr/local/lib/python2.x/site-packages and the system python using /usr/local/lib/python2.x/dist-packages. IMO it should still be possible to install into the former using the system python. how would you do this? otoh we could just ignore this case and require the locally installed python for installations into /usr/local/lib/python2.x/site-packages. in any case, site-packages should be used for all occasions where prefix is not /usr or /usr/local or when stuff like virtualenv or a userbase installation is wanted.

Installing into site-packages using the system python seems fraught with difficulties anyway; what if extensions are involved and the system and manual python installations are extension-ABI-incompatible for some reason? Requiring use of the locally-installed python for this case seems reasonably justifiable to me, unless I'm missing something.

This bug was fixed in the package python2.6 - 2.6.2-0ubuntu2

---------------
python2.6 (2.6.2-0ubuntu2) karmic; urgency=low

  * Update to 20090619, taken from the 2.6 release branch.
  * distutils: Always install into `/usr/local/lib/python2.6/dist-packages'
    if an option `--prefix=/usr/local' is present (except for virtualenv
    and PYTHONUSERBASE installations). LP: #362570.

 -- Matthias Klose <email address hidden> Fri, 19 Jun 2009 14:40:09 +0200

For me, installing with --prefix=/usr/local still installs into a directory (/usr/local/lib/python2.6/site-packages/). That choice is OK with me, but the problem is that this directory is not in the default search path and so the original bug has not been resolved and this breaks installations of third-party packages that used to work in 8.10.

Here's an example showing that installing a package (rdiff-backup in this case) with the --prefix=/usr/local puts it in a location that is not in the python search path:

/usr/local/src/rdiff-backup-1.2.5 (450) $ dpkg -l python
...
ii python 2.6.2-0ubuntu1 An interactive high-level object-oriented language (default version)

/usr/local/src/rdiff-backup-1.2.5 (453) $ sudo python setup.py install --prefix=/usr/local --record files-installed
running install
...
writing list of installed files to 'files-installed'

/usr/local/src/rdiff-backup-1.2.5 (454) $ grep Main.pyc files-installed
/usr/local/lib/python2.6/site-packages/rdiff_backup/Main.pyc

/usr/local/src/rdiff-backup-1.2.5 (457) $ rdiff-backup --version
Traceback (most recent call last):
  File "/usr/local/bin/rdiff-backup", line 20, in <module>
    import rdiff_backup.Main
ImportError: No module named rdiff_backup.Main

/usr/local/src/rdiff-backup-1.2.5 (462) $ sudo bash -c 'echo "/usr/local/lib/python2.6/site-packages/" >/usr/lib/python2.6/dist-packages/local.pth'

/usr/local/src/rdiff-backup-1.2.5 (464) $ rdiff-backup --version
rdiff-backup 1.2.5

So I think /usr/local/lib/python2.6/site-packages/ needs to be added to the default search path.

I didn't check it myself, but the fix is released for python2.6 (2.6.2-0ubuntu2) karmic; urgency=low. You might want to try that out. I see that you have 2.6.2-0ubuntu1...

Thanks for the explanation for why the fix doesn't work for me :). I can't seem to (easily) install the karmic package so I'll have to wait until I upgrade to test the fix.

Is the fix coming to jaunty? What do I tell to users that want to install my modules?

I have the problem described in #10 now with Maverick.

That means I can't offer my software as tarball with just a setup.py, unless I tell people to add export PYTHONPATH=/usr/lib/python2.6/site-packages to their .bashrc.

Eek, forget what I said about PYTHONPATH, that doesn't work even with the right path. Sorry.

I am experiencing exactly what is described in comment #10 on Precise. `setup.py install` is putting things into site-packages directory, and then when I try to run the program it raises ImportError because sys.path does not contain any variation of 'site-packages' whatsoever.

The workaround I've created is ugly beyond belief: https://github.com/robru/gottengeography/commit/54f52fa0a3f97e22fb61e70bc0b5ec77c1a19015

The code all worked fine on Fedora without the above patch, so something is pretty broken with Ubuntu's distutils that it's installing packages to places where python can't even find them.

This also needs to be fixed for python3...

On May 02, 2016, at 08:28 PM, tripzero wrote:

>This also needs to be fixed for python3...

What exactly needs fixing? `sudo python3 setup.py install` will install into
/usr/local/.../dist-packages which is exactly where it should install, and is
definitely on the system Python's sys.path.

Status changed to 'Confirmed' because the bug affects multiple users.