Ubuntu
python3.5 package

sysconfig.get_path('purelib') points to site-packages directory

Bug #1408092 reported by Gökçen Eraslan on 2015-01-06

60

This bug affects 12 people

Affects		Status	Importance	Assigned to	Milestone
	python3-defaults (Ubuntu)	Confirmed	Undecided	Unassigned
	python3.5 (Ubuntu)	Confirmed	Undecided	Unassigned

Bug Description

purelib and platlib paths in Python 3 sysconfig module point to site-packages directory:

$ python3 -m sysconfig

Platform: "linux-x86_64"
Python version: "3.4"
Current installation scheme: "posix_prefix"

Paths:
data = "/usr"
include = "/usr/include/python3.4m"
platinclude = "/usr/include/python3.4m"
platlib = "/usr/lib/python3.4/site-packages"
platstdlib = "/usr/lib/python3.4"
purelib = "/usr/lib/python3.4/site-packages"
scripts = "/usr/bin"
stdlib = "/usr/lib/python3.4"
...

whereas Python 2 points to dist-packages:

Platform: "linux-x86_64"
Python version: "2.7"
Current installation scheme: "posix_local"

Paths:
data = "/usr/local"
include = "/usr/local/include/python2.7"
platinclude = "/usr/local/include/python2.7"
platlib = "/usr/local/lib/python2.7/dist-packages"
platstdlib = "/usr/lib/python2.7"
purelib = "/usr/local/lib/python2.7/dist-packages"
scripts = "/usr/local/bin"
stdlib = "/usr/lib/python2.7"
...

One severe consequence of this is that automake is installing Python modules to a location which is not in sys.path, namely site-packages directory.

When one uses automake to create a Makefile that installs a Python module to the system, it is installed into /usr/local/lib/python3.4/site-packages since automake/aclocal uses sysconfig to determine where the package will be installed.

From /usr/share/aclocal-1.14/python.m4 file:

  AC_CACHE_CHECK([for $am_display_PYTHON script directory],
    [am_cv_python_pythondir],
    [if test "x$prefix" = xNONE
     then
       am_py_prefix=$ac_default_prefix
     else
       am_py_prefix=$prefix
     fi
     am_cv_python_pythondir=`$PYTHON -c "
$am_python_setup_sysconfig
if can_use_sysconfig:
    sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'}) <-----------
else:
    from distutils import sysconfig
...

Here, it can be seen that automake uses purelib path defined by the sysconfig module of Python. However, since this path is not included in sys.path, and thus the installed package cannot be used:

$ python3 -c "import sys; print(sys.path)"
['', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']

Ubuntu version:14.10

$ apt-cache policy python3-minimal
python3-minimal:
  Installed: 3.4.2-1
  Candidate: 3.4.2-1
  Version table:
*** 3.4.2-1 0
        500 http://mirror.23media.de/ubuntu/ utopic/main amd64 Packages
        100 /var/lib/dpkg/status

Gökçen Eraslan (gkcn) on 2015-01-06

summary:

- sysconfig paths point to site-packages directory
+ sysconfig.get_path('purelib') points to site-packages directory

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-08-05:

#1

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python3-defaults (Ubuntu):
status:	New → Confirmed

Revision history for this message

astro (bernard-godard) wrote on 2016-03-08:

#2

This bug also affects python 3.5 in Ubuntu 16.04.

Note that automake scripts use also platlib in addition to purelib. platlib is also wrongly pointing to site-packages.

Revision history for this message

astro (bernard-godard) wrote on 2016-03-08:

#3

Additionally if for some reason the sysconfig module cannot be used, automake scripts will fallback to using distutils.sysconfig like this:

distutils.sysconfig.get_python_lib(1,0,prefix=$prefix))

The result of this is (Ubuntu 16.04):
/usr/local/lib/python3/dist-packages

This is correctly set to dist-packages. However, sys.path contains /usr/local/lib/python3.5/dist-packages not /usr/local/lib/python3/dist-packages
Hence the modules are not installed in the correct path unless a symlink python3->python3.5 is manually made.

Revision history for this message

Yury V. Zaytsev (zyv) wrote on 2016-03-08:

#4

Hi Barry,

Just trying to bring this to your attention...

Thanks!

Revision history for this message

Marian (djzielony) wrote on 2016-07-14:

#5

Please fix this, I can't normally package and install packages I develop because of this.

Revision history for this message

Dmitry Eremin-Solenikov (lumag) wrote on 2016-09-19:

#6

Seconding this bug.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-09-19:

#7

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python3.5 (Ubuntu):
status:	New → Confirmed

Revision history for this message

Jack Hale (mttl) wrote on 2016-11-18:

#8

Perhaps it gets lost in this thread I wanted to confirm that the problem highlighted by @bernard-godard comment https://bugs.launchpad.net/ubuntu/+source/python3.5/+bug/1408092/comments/3 is also causing serious issues when using distutils to install into the prefix /usr/local.

Revision history for this message

Stephen Sinclair (radarsat1) wrote on 2017-06-21:

#9

Hello, yes this last issue (distutils.sysconfig) is causing a problem on a project I work on, so I'd like to confirm as well. It appears to have been previously brought up here without reply: https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-October/017005.html

I am seeing the issue on Ubuntu 17.04 so it seems this bug has not been fixed.

Revision history for this message

Stephen Sinclair (radarsat1) wrote on 2017-06-21:

#10

In fact looking at /usr/lib/python3.4/distutils/sysconfig.py, it appears the fix is quite easy. In get_python_inc(), the variable `python_dir = python_dir = 'python' + get_python_version() + build_flags` is used to build the path. However, in get_python_lib(), "python3" is a hard-coded string: `return os.path.join(prefix, "lib", "python3", "dist-packages")`

This is a special case for the value of prefix if it's something like /usr/local.. Currently to get around this I am forced to provide a fake argument to `prefix` and then doing a string replace for /usr/local, which is just silly.

Revision history for this message

Jeff Larson (jmlarson) wrote on 2017-08-28:

#11

Is there any motion towards resolving this issue?

Revision history for this message

Jeff Larson (jmlarson) wrote on 2017-08-28:

#12

I just checked and this issue persists in python3.6.2.

Revision history for this message

linas (linasvepstas) wrote on 2018-05-23:

#13

my downstream package suffers from this: https://github.com/opencog/link-grammar/issues/530

Revision history for this message

Kevron Rees (tripzero) wrote on 2018-06-25:

#14

The issue is that /usr/lib/python3/sysconfig.py is hardcoded to 'site-packages' (see _INSTALL_SCHEMES variable in the code). If your code uses sysconfig.get_path() to get the python3 site dir, it will fail on ubuntu.

So either the fix needs to change /usr/lib/python3/sysconfig.py or packages need to be fixed to use distutils

Revision history for this message

Gabriel Rauter (raetiacorvus) wrote on 2018-10-23:

#15

Mesons python module uses the purelib/platlib path when using the function install_sources() to install a python module.

https://mesonbuild.com/Python-module.html

It would be helpful to know if ubuntu is going to fix this or not so corresponding bug reports can be opened in projects depending on this.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

auto-github-opencog-link-grammar #530
[closed python] Edit

Bug watches keep track of this bug in other bug trackers.