virtualenv not working under opensuse

Description of problem:
Inside a virtual environment created with the 'virtualenv' command, the /usr/lib64/python2.5/site-packages path is missing.

Version-Release number of selected component (if applicable):
python-virtualenv-1.3-1.fc9.noarch

Steps to Reproduce:
1. Create a virtual environment like with 'virtualenv /tmp/env'
2. Compare paths outside and inside using
   "python -c 'import sys, pprint; pprint.pprint(sys.path);'" and
   "/tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'"
   respectively.

Actual results:
% python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/usr/lib64/python25.zip',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/plat-linux2',
 '/usr/lib64/python2.5/lib-tk',
 '/usr/lib64/python2.5/lib-dynload',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib/python2.5/site-packages']

Expected results:
See how /usr/lib64/python2.5/site-packages (and thus all paths below, like Numeric, PIL, etc.) are missing in the second output.

Additional info:
A similar problem (on OpenSuSE) was described here https://bugs.launchpad.net/virtualenv/+bug/248558/comments/2 (note that that is different from the original bug report) and propagated to OpenSuSE's bugtracker here: https://bugzilla.novell.com/show_bug.cgi?id=430761. That might be related if Fedora is also patching site.py.

Thanks for the report. I'll look into pushing up an upstream patch.

I'll give the patch listed at https://bugs.launchpad.net/fedora/+source/python-virtualenv/+bug/2485 a run and see if that fixes the issue on a 64bit host. It was applied on 2008-09-29.

python-virtualenv-1.3.1-1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/python-virtualenv-1.3.1-1.fc10

(In reply to comment #3)
> python-virtualenv-1.3.1-1.fc10 has been submitted as an update for Fedora 10.

This doesn't fix the issue -- and wasn't supposed to, because upstream's 1.3.1 fixes the original problem reported in launchpad's bug 248558, not the problem described in comment 2 thereof.

Here's the current output on fc10, with
python-2.5.2-1.fc10.x86_64
python-setuptools-0.6c8-1.fc10.noarch
python-virtualenv-1.3.1-1.fc10.noarch

% python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/usr/lib64/python25.zip',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/plat-linux2',
 '/usr/lib64/python2.5/lib-tk',
 '/usr/lib64/python2.5/lib-dynload',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

vs.

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib/python2.5/site-packages']

Still, in the second output, paths /usr/lib64/python2.5/site-packages and below are missing.

The problem is that fedora patches site.py, which virtualenv doesn't (can't?) detect.

Ok, I see what you mean. virtualenv is writing out it's site.py via a zipped version it holds in virtualenv.py as base64 encoded zipped data. I'll take a look at the patches being applied in the python fedora package that might make sense to pull into virtualenv's copy.

These all end up mapping to pth files and are owned by different packages ... here is an example from my F9 laptop ...

[steve@tachikoman site-packages]$ for x in `find . -name "*.pth"`; do
    echo -n "$x: ";
    echo `rpm -q --whatprovides /usr/lib/python2.5/site-packages/$x`;
done
./PasteDeploy-1.3.1-py2.5-nspkg.pth: python-paste-deploy-1.3.1-2.fc9.noarch
./Numeric.pth: python-numeric-24.2-11.fc9.i386
./Paste-1.6-py2.5-nspkg.pth: python-paste-1.6-1.fc9.noarch
./PIL.pth: python-imaging-1.1.6-9.fc9.i386
./pygtk.pth: pygobject2-2.14.2-1.fc9.i386
./easy-install.pth: file /usr/lib/python2.5/site-packages/easy-install.pth is not owned by any package
./pygst.pth: gstreamer-python-0.10.11-2.fc9.i386
./DecoratorTools-1.7-py2.5-nspkg.pth: python-decoratortools-1.7-1.fc9.noarch
./PasteScript-1.6.2-py2.5-nspkg.pth: python-paste-script-1.6.2-2.fc9.noarch
./wx.pth: wxPython-2.8.9.1-1.fc9.i386

or on my F10 desktop

[steve@powerhouse site-packages]$ for x in `find . -name "*.pth"`; do
> echo -n "$x: ";
> echo `rpm -q --whatprovides /usr/lib64/python2.5/site-packages/$x`;
> done
./pygtk.pth: pygobject2-2.15.4-3.fc10.x86_64
./Numeric.pth: python-numeric-24.2-11.fc9.x86_64
./PIL.pth: python-imaging-1.1.6-12.fc10.x86_64
./pygst.pth: gstreamer-python-0.10.12-1.fc10.x86_64
./scim.pth: scim-python-0.1.13rc1-1.fc10.x86_64

I looked quickly at the patching that is done on site.py in the python package and it's pretty minimal and it didn't look to be related.

I guess I could make a patch that generates full path pth's off of whats in the site-packages dir(s). I don't know if that really makes sense though since it's per package.

I could still be missing something though ... let me know if I am Thomas.

(In reply to comment #6)

> I looked quickly at the patching that is done on site.py in the python package
> and it's pretty minimal and it didn't look to be related.

Well, it is surely related, because it (at least) changes one of the sitedirs and adds another one on a 64 bit system. Here's one of the most relevant chunks from python/F-10/python-2.5-lib64.patch (applied as patch No. 102):

--- Python-2.5b1/Lib/site.py.lib64 2006-06-12 04:23:02.000000000 -0400
+++ Python-2.5b1/Lib/site.py 2006-06-22 12:20:35.000000000 -0400
@@ -182,9 +182,14 @@
                 sitedirs = [os.path.join(prefix, "Lib", "site-packages")]
             elif os.sep == '/':
                 sitedirs = [os.path.join(prefix,
- "lib",
+ "lib64",
                                          "python" + sys.version[:3],
                                          "site-packages"),
+ os.path.join(prefix,
+ "lib",
+ "python" + sys.version[:3],
+ "site-packages"),
+ os.path.join(prefix, "lib64", "site-python"),
                             os.path.join(prefix, "lib", "site-python")]
             else:
                 sitedirs = [prefix, os.path.join(prefix, "lib", "site-packages")]

Imho that's exactly the sort of patch that should either be removed from the python package, or added to the virtualenv package. And there might be some other relevant chunks in that patch.

Maybe we should get one of the main python package's maintainers on board here?

> I guess I could make a patch that generates full path pth's off of whats
> in the site-packages dir(s). I don't know if that really makes sense though
> since it's per package.

That doesn't sound like the right (and neither a feasible) solution to me.

When building, grab site.py from the python package and patch it as necessary.

The problem with grabbing site.py from the python package is that it is vastly different than the site.py zipped and base encoded in virtualenv. I can modify what virtualenv holds though using the python package's site.py file.

In the actual unzipped virtualenv source they talk about extra paths using pth's and give an example of debain's lib locations which is what made me look that the pth's being placed by things like Numeric, PIL, gst and gtk.

If you guys really think using the python packages patched version will fix the issue I'm willing to give it a shot ... but at first run virtualenv failed for me when trying to use it like that.

(In reply to comment #9)
> The problem with grabbing site.py from the python package is that it is vastly
> different than the site.py zipped and base encoded in virtualenv.

Most of the differences come from the fact though that virtualenv's site.py is from python 2.3. The differences can be seen using (zsh syntax):

diff -u /usr/lib64/python2.5/site.py <(python -c 'import virtualenv; print virtualenv.SITE_PY')

I have a confession.

This is not the first time I've seen this bug.

I thought about it LONG AND HARD for the better part of a month when I initially hit it, and this was the only sane method I could find for solving it. Unfortunately I never got around to actually implementing it.

Here's a naive patch:

Index: python-virtualenv.spec
===================================================================
RCS file: /cvs/pkgs/rpms/python-virtualenv/devel/python-virtualenv.spec,v
retrieving revision 1.5
diff -r1.5 python-virtualenv.spec
12a13
> Patch0: virtualenv-lib64.patch
28c29,33
< %setup -q -n virtualenv-%{version}
---
> %setup0 -q -n virtualenv-%{version}
> %if "%{_lib}" == "lib64"
> %patch0 -p1 -b .lib64
> %endif
> %{__python} rebuild-script.py

diff -up virtualenv-1.3.1/support-files/site.py.orig virtualenv-1.3.1/support-files/site.py
--- virtualenv-1.3.1/support-files/site.py.orig 2008-11-25 03:27:26.000000000 +0100
+++ virtualenv-1.3.1/support-files/site.py 2008-12-03 00:38:40.000000000 +0100
@@ -213,6 +213,10 @@ def addsitepackages(known_paths, sys_pre

             elif os.sep == '/':
                 sitedirs = [os.path.join(prefix,
+ "lib64",
+ "python" + sys.version[:3],
+ "site-packages"),
+ os.path.join(prefix,
                                          "lib",
                                          "python" + sys.version[:3],
                                          "site-packages"),

This seems to work, but now there are too many dirs in the path:

% /tmp/env/bin/python -c 'import sys, pprint; pprint.pprint(sys.path);'
['',
 '/tmp/env/lib64/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
 '/tmp/env/lib64/python25.zip',
 '/tmp/env/lib64/python2.5',
 '/tmp/env/lib64/python2.5/plat-linux2',
 '/tmp/env/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/lib-dynload',
 '/usr/lib/python2.5',
 '/usr/lib64/python2.5',
 '/usr/lib64/python2.5/lib-tk',
 '/tmp/env/lib64/python2.5/site-packages',
 '/tmp/env/lib/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages',
 '/usr/lib64/python2.5/site-packages/Numeric',
 '/usr/lib64/python2.5/site-packages/PIL',
 '/usr/lib64/python2.5/site-packages/gst-0.10',
 '/usr/lib64/python2.5/site-packages/gtk-2.0',
 '/usr/lib/python2.5/site-packages']

Note that simply copying the system site.py did not work for me neither. According to a message found on their mailinglist, the version in the virtualenv package is slightly modified. So maybe the right way is to use system's site.py, but apply virtualenv's modfications.

Well, if that works and the extra paths are not harming anything, I'm fine with using that patch ... at least for now. I'm going to run some tests with a modified version of the patch (- original virtualenv site.py, + my changes to the original virtualenv setup.py)....

--- /tmp/site.py.virtualenv 2008-12-11 19:40:58.000000000 -0500
+++ lib/python2.5/site.py 2008-12-11 20:50:55.000000000 -0500
@@ -201,11 +201,27 @@
                                              "python" + sys.version[:3], "site-packages")]

             elif os.sep == '/':
- sitedirs = [os.path.join(prefix,
- "lib",
- "python" + sys.version[:3],
- "site-packages"),
- os.path.join(prefix, "lib", "site-python")]
+ # PATCH:BZ471617
+ # https://bugzilla.redhat.com/show_bug.cgi?id=471617
+ import platform
+ # Maps the processor type with the library path name
+ lib_map = {'lib': 'lib',
+ 'x86_64': 'lib64'}
+
+ def __easy_path(cpu):
+ """
+ Simply returns a python path based on cpu and lib_map.
+ """
+ if cpu not in lib_map.keys():
+ return ''
+ return os.path.join(prefix,
+ lib_map[cpu],
+ "python" + sys.version[:3],
+ "site-packages")
+
+ sitedirs = [__easy_path("lib")]
+ sitedirs.append(__easy_path(platform.processor()))
+ # END PATCH:BZ471617
                 try:
                     # sys.getobjects only available in --with-pydebug build
                     sys.getobjects

(In reply to comment #13)
> Well, if that works and the extra paths are not harming anything, I'm fine with
> using that patch ... at least for now.

Did you talk to upstream about this problem already? Maybe they have some idea how to cleanly solve it.

> modified version of the patch (- original virtualenv site.py, + my changes to
> the original virtualenv setup.py)....

That code looks far more complicated to me than it needs to be, but if it works *shrug*.

Maybe Ignacio can comment?

From 1.3.2 release

    * Added platform-specific paths, like /usr/lib/pythonX.Y/plat-linux2

I now get the paths added in this version:

<snip>
'/usr/lib/python2.5/site-packages',
'/usr/lib64/python2.5/site-packages',
'/usr/lib64/python2.5/site-packages/Numeric',
'/usr/lib64/python2.5/site-packages/PIL',
'/usr/lib64/python2.5/site-packages/gst-0.10',
'/usr/lib64/python2.5/site-packages/gtk-2.0',
'/usr/lib64/python2.5/site-packages/scim-0.1']

Just an FYI as I'll be pushing the new version up with a spec fix with this new version.

http://koji.fedoraproject.org/koji/buildinfo?buildID=76438

The package has been pushed to stable.

Seeing this again on F17:

# virtualenv /tmp/env
New python executable in /tmp/env/bin/python
Installing setuptools............done.
Installing pip...............done.

# python -c 'import sys; print "\n".join(sys.path)' | grep -v '\.local'

/usr/lib64/python27.zip
/usr/lib64/python2.7
/usr/lib64/python2.7/plat-linux2
/usr/lib64/python2.7/lib-tk
/usr/lib64/python2.7/lib-old
/usr/lib64/python2.7/lib-dynload
/usr/lib64/python2.7/site-packages
/usr/lib64/python2.7/site-packages/PIL
/usr/lib64/python2.7/site-packages/gst-0.10
/usr/lib64/python2.7/site-packages/gtk-2.0
/usr/lib64/python2.7/site-packages/wx-2.8-gtk2-unicode
/usr/lib/python2.7/site-packages
/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info

# /tmp/env/bin/python -c 'import sys; print "\n".join(sys.path)'

/tmp/env/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/tmp/env/lib/python2.7/site-packages/pip-1.1-py2.7.egg
/tmp/env/lib64/python27.zip
/tmp/env/lib64/python2.7
/tmp/env/lib64/python2.7/plat-linux2
/tmp/env/lib64/python2.7/lib-tk
/tmp/env/lib64/python2.7/lib-old
/tmp/env/lib64/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib64/python2.7
/usr/lib64/python2.7/lib-tk
/tmp/env/lib/python2.7/site-packages

This yields to import failures, e.g.

# python -c 'import yaml'
# /tmp/env/bin/python -c 'import yaml'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named yaml

Works on F21 with virtualenv --system-site-packages.