system path should be ignorable/overridable

On Oct 30, 2007, at 1:05 PM, Martijn Faassen wrote:

> Public bug reported:
>
> The following problem keep biting the Grok project. We regularly get
> users who already have Zope 3 installed, or Twisted, or
> zope.interface,
> and so on, and as a result can't install and/or run Grok, due to the
> wrong packages appearing on the import either during recipe execution
> time, or during startup time.

It would be helpful if you could double check this. See below.

> We know people have to compile and install their own Python, but we
> can't expect beginning users to compile a custom Python (or even a
> virtual environment) just to try something out (on Windows and to a
> lesser extend MacOS the tools are likely not even installed to
> compile a
> custom Python). In one training session I ran into this problem with
> many of the participants, on 3 different platforms, Linux, Window and
> Mac OS X.

+1

> Having the ability to isolate a buildout entirely from site-packages
> would be very helpful, or alternatively let site-packages come last in
> the order.

Both seem reasonable.

> The latter option would not break people's expectations:
> their system-installed packages that the buildout doesn't override
> will
> continue to be importable.

Generated scripts always put eggs found at the start of the path.
buildout, like setuptools, searches site packages for eggs to install
and probably selects these some times, so I think this largely boils
down to the rules for finding eggs in the first place.

It would be helpful if you could verify this by verifying that, in
fact, the generated scripts have "system" eggs inserted by the script.

Jim

--
Jim Fulton
Zope Corporation

hi jim, martijn,

On Oct 30, 2007 Jim Fulton wrote:
> Generated scripts always put eggs found at the start of the path.
> buildout, like setuptools, searches site packages for eggs to install
> and probably selects these some times, so I think this largely boils
> down to the rules for finding eggs in the first place.

we've just run into a problem that very same or anyway closely related problem, and i was about to file a bug after some analysis i just did. sparing you a long read with all the details i think the essential problem here is that `bootstrap.py` uses a globally, non-eggish setuptools if it finds one. that results in `.../site-packages` to be inserted into `sys.path` in `bin/buildout` as well as the other scripts in `bin/`, i.e. `bin/instance`.

now if you install, let's say a zope2 instance via buildout and additionally have some globally installed eggs like `zope.interface` (as commonly installed system-wide by installing twisted, for example), starting up the zope instance will pick up the global egg instead of the module coming from your zope installation, because `.../site-packages` comes before `parts/zope2/lib/python` in the modified `sys.path`. obviously this would cause problems if the versions differ.

however, if setuptools isn't installed as a "regular" package in the system, but either not at all or as an egg via `easy_install`, `bootstrap.py` will download or copy it into its `eggs/` directory and `.../site-packages` will not be included into `sys.path`, effectively fixing these problems. on the other hand people need setuptools and we cannot tell everyone to not use packages but install it via `easy_install` instead. but i think this could be fixed by changing `bootstrap.py` to insist in always using an egg for setuptools, that is even downloading it when a setuptools package already exists in the system. or would that have other implications?

cheers,

andi

ps: the above is just what i gathered from looking at a zope2 buildout, but i'd suspect things to work in a similar way with zope3 or grok...

On Nov 19, 2007, at 9:09 AM, Andi Zeidler wrote:

> hi jim, martijn,
>
> On Oct 30, 2007 Jim Fulton wrote:
>> Generated scripts always put eggs found at the start of the path.
>> buildout, like setuptools, searches site packages for eggs to install
>> and probably selects these some times, so I think this largely boils
>> down to the rules for finding eggs in the first place.
>
> we've just run into a problem that very same or anyway closely related
> problem, and i was about to file a bug after some analysis i just did.
> sparing you a long read with all the details i think the essential
> problem here is that `bootstrap.py` uses a globally, non-eggish
> setuptools if it finds one. that results in `.../site-packages` to be
> inserted into `sys.path` in `bin/buildout` as well as the other
> scripts
> in `bin/`, i.e. `bin/instance`.

I don't believe this has anything to do with it. Buildout, as a
matter of policy, includes whatever Python path is used to run the
buildout, regardless of where it finds the setuptools eggs. It would
be fairly straightforward task to have an option to search only
buildout-defined directories for eggs.

> now if you install, let's say a zope2 instance via buildout and
> additionally have some globally installed eggs like `zope.interface`
> (as
> commonly installed system-wide by installing twisted, for example),
> starting up the zope instance will pick up the global egg instead of
> the
> module coming from your zope installation, because `.../site-packages`
> comes before `parts/zope2/lib/python` in the modified `sys.path`.
> obviously this would cause problems if the versions differ.
>
> however, if setuptools isn't installed as a "regular" package in the
> system, but either not at all or as an egg via `easy_install`,
> `bootstrap.py` will download or copy it into its `eggs/` directory and
> `.../site-packages` will not be included into `sys.path`, effectively
> fixing these problems.

Have you seen this "fix" the problem? I'm 99% that when constructing
working sets (paths) that the entire Python path is searched
regardless of where setuptools is installed.

> on the other hand people need setuptools and we
> cannot tell everyone to not use packages but install it via
> `easy_install` instead. but i think this could be fixed by changing
> `bootstrap.py` to insist in always using an egg for setuptools, that
> is
> even downloading it when a setuptools package already exists in the
> system. or would that have other implications?

I don't believe this is a bootstrapping issue.

Jim

--
Jim Fulton
Zope Corporation

On Nov 19, 2007, at 7:51 PM, Jim Fulton wrote:
> I don't believe this has anything to do with it.

well, i've tried all this before posting the comment. actually things
were working fine for me (on osx with a setuptools egg), but nouri's
reported a problem on ubuntu, so i took my gf's computer and spent
about two hours debugging the various combinations. so, what can i
say? i'm pretty sure it does have to do with this... :)

> Buildout, as a
> matter of policy, includes whatever Python path is used to run the
> buildout, regardless of where it finds the setuptools eggs.

imho, `bootstrap.py` already creates the problem. please try
installing setuptools as a package in site-packages, run
`bootstrap.py` and have a look at the resulting `bin/buildout`, then
do the same with setuptools installed as a global egg and without
having it installed at all.

>> however, if setuptools isn't installed as a "regular" package in the
>> system, but either not at all or as an egg via `easy_install`,
>> `bootstrap.py` will download or copy it into its `eggs/` directory
>> and
>> `.../site-packages` will not be included into `sys.path`, effectively
>> fixing these problems.
>
> Have you seen this "fix" the problem?

yes, this fixed the problem for me and nouri's also verified it.

> I'm 99% that when constructing
> working sets (paths) that the entire Python path is searched
> regardless of where setuptools is installed.

that may be the case, but the problem is that `site-packages` gets
injected into `sys.path` before `lib/python` of the zope installation,
so egg installed there will shadow modules in zope, which is not what
one would expect, imho. again, this is not the case when setuptools
is installed as an egg. however, as setuptools are always required,
`working_set` will return `.../site-packages` as its location — in
case it's a global package.

> I don't believe this is a bootstrapping issue.

jim, please excuse me, but this really doesn't help too much. could
you please try it at least? it should only take a few minutes... i
will create a workaround in the recipe if you cannot reproduce it, but
the problem really is in buildout, imho.

best regards,

andi

--
zeidler it consulting - http://zitc.de/ - <email address hidden>
friedelstraße 31 - 12047 berlin - telefon +49 30 25563779
pgp key at http://zitc.de/pgp - http://wwwkeys.de.pgp.net/
plone 3.0.3 released! -- http://plone.org/products/plone

I had the same problem. I could not verify, that setuptools gets installed if not already in site-packages, but otherwise I have the same results as andi (see below). Whenever site-packages, was not listed in sys.path, a local setuptools egg was listed instead.

For me, it would feel most natural, if the system-wide site-packages would come after all the paths that point to components installed by the buildout, which probably means it should not be listed at all in the assignment to sys.path[0:0], as it is anyway listed further down the list. The behaviour for setuptools installed as egg, is exactly what I would expect.

The way bootstrap and buildout detect setuptools seems to differ, as can be seen, when setuptools is installed both as egg and as package. As I see it, it would be nicest to fix both, bootstrap, and buildout, to ...

... either move the position of site-packages in sys.path to the end of the sys.path[0:0] assignment (I did manually, and it solves the problem)
... or to never use setuptools installed system-wide, be it an egg or a package

I think the first would be cleanest and further, no local setuptools egg should be installed, if there is a system-wide one, that could be used instead (given its not out-dated or something). I hope there is no voodoo going on in setuptools, which prevents that solution.

best regards
florian

<testing>
setuptools installed as package:

$ grep -C2 site-packages bin/buildout
import sys
sys.path[0:0] = [
  '/usr/lib64/python2.4/site-packages',
  '/usr/lib64/python2.4/site-packages',
  '/usr/lib64/python2.4/site-packages',
  '/usr/lib64/python2.4/site-packages',
  '/home/flo/eestec_ploneldap/eggs/zc.buildout-1.0.0b31-py2.4.egg',
  ]
$
$ grep -C2 site-packages bin/instance
  '/home/flo/eestec_ploneldap/eggs/plone.recipe.zope2instance-0.9-py2.4.egg',
  '/home/flo/eestec_ploneldap/eggs/zc.recipe.egg-1.0.0-py2.4.egg',
  '/usr/lib64/python2.4/site-packages',
  '/home/flo/eestec_ploneldap/eggs/zc.buildout-1.0.0b31-py2.4.egg',
  '/home/flo/eestec_ploneldap/eggs/python_openid-2.0.1-py2.4.egg',
--
  '/home/flo/eestec_ploneldap/eggs/five.customerize-0.2-py2.4.egg',
  '/home/flo/eestec_ploneldap/eggs/five.localsitemanager-0.2-py2.4.egg',
  '/usr/lib64/python2.4/site-packages',
  '/home/flo/eestec_ploneldap/eggs/python_openid-2.0.1-py2.4.egg',
  ]
$
$ grep -C2 site-packages bin/zopepy
  '/home/flo/eestec_ploneldap/eggs/five.customerize-0.2-py2.4.egg',
  '/home/flo/eestec_ploneldap/eggs/five.localsitemanager-0.2-py2.4.egg',
  '/usr/lib64/python2.4/site-packages',
  '/home/flo/eestec_ploneldap/eggs/python_openid-2.0.1-py2.4.egg',
  '/usr/lib64/zope-2.10.4/lib/python',
$

site-packages occurs in sys.path of buildout, instance and zopepy.

No setuptools installed:

$ python bootstrap.py
Traceback (most recent call last):
  File "bootstrap.py", line 46, in ?
    PYTHONPATH=
AttributeError: 'NoneType' object has no attribute 'location'
$

setuptools installed as package + setuptools as egg.

$ python bootstrap.py
Generated script '/home/flo/eestec_ploneldap/bin/buildout'.
$ grep -C2 site-packages bin/buildout
$

Looks good so far. However:

$ ./bin/buildout -No
Uninstalling zopepy.
Uninstalling instance.
Unins...

Forgot to mention my platform:

Gentoo Linux, amd64, python-2.4.4.

Please let me know, if you need further information.

On Nov 19, 2007, at 10:56 PM, Andreas Zeidler wrote:
> imho, `bootstrap.py` already creates the problem. please try
> installing setuptools as a package in site-packages, run
> `bootstrap.py` and have a look at the resulting `bin/buildout`, then
> do the same with setuptools installed as a global egg and without
> having it installed at all.

the following is a transcript from a freshly set up, minimal debian
system that demonstrates this...

---------- snip ----------------------------------------------
debian:~# aptitude install gcc subversion python2.4-dev python-
zopeinterface
[...snip...]
debian:~# cat > buildout.cfg

[buildout]
parts = zope3 zpy

[zope3]
recipe = zc.recipe.zope3checkout
url = svn://svn.zope.org/repos/main/Zope3/branches/3.3

[zpy]
recipe = zc.recipe.egg
eggs = zc.recipe.egg # otherwise bin/zpy will have a syntax error
interpreter = zpy
extra-paths = ${zope3:location}/src
scripts = zpy

debian:~# svn export svn://svn.zope.org/repos/main/zc.buildout/trunk/
bootstrap/bootstrap.py
A bootstrap.py
Export complete.
debian:~# python bootstrap.py
Downloading http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c7-py2.4.egg
Creating directory '/root/bin'.
Creating directory '/root/parts'.
Creating directory '/root/eggs'.
Creating directory '/root/develop-eggs'.
Generated script '/root/bin/buildout'.
debian:~# ./bin/buildout
Getting distribution for 'zc.recipe.zope3checkout'.
Got zc.recipe.zope3checkout 1.2.
Getting distribution for 'zc.recipe.egg'.
Got zc.recipe.egg 1.0.0.
Installing zope3.
[...snip...]
Installing zpy.
Generated interpreter '/root/bin/zpy'.
debian:~# ./bin/zpy

 >>> import zope.interface
 >>> zope.interface.__file__
'/root/parts/zope3/src/zope/interface/__init__.py'
 >>>
debian:~# rm -rf bin/ develop-eggs/ eggs/ parts/ .installed.cfg
debian:~# aptitude install python-setuptools
Reading package lists... Done
Building dependency tree... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done
Reading task descriptions... Done
Building tag database... Done
The following NEW packages will be installed:
   python-setuptools
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/204kB of archives. After unpacking 758kB will be used.
Writing extended state information... Done
Selecting previously deselected package python-setuptools.
(Reading database ... 27858 files and directories currently installed.)
Unpacking python-setuptools (from .../python-
setuptools_0.6c3-3_all.deb) ...
Setting up python-setuptools (0.6c3-3) ...

debian:~# python bootstrap.py
Creating directory '/root/bin'.
Creating directory '/root/parts'.
Creating directory '/root/eggs'.
Creating directory '/root/develop-eggs'.
Generated script '/root/bin/buildout'.
debian:~# ./bin/buildout
Getting distribution for 'zc.recipe.zope3checkout'.
Got zc.recipe.zope3checkout 1.2.
Getting distribution for 'zc.recipe.egg'.
Got zc.recipe.egg 1.0.0.
Installing zope3.
[...snip...]
Installing zpy.
Generated interpreter '/root/bin/zpy'.
debian:~# ./bin/zpy

 >>> import zope.interface
 >>> zope.interface.__file__
'/usr/lib/python2.4/site-packages/zope/...

On Nov 20, 2007, at 6:10 PM, Florian Friesdorf wrote:
> As I see it, it would be nicest to fix both, bootstrap, and
> buildout, to ...
>
> ... either move the position of site-packages in sys.path to the end
> of the sys.path[0:0] assignment (I did manually, and it solves the
> problem)

just as a side note, imho "site-packages" shouldn't be injected into
`sys.path` at all as it's put in there by default when you start a
python interpreter. that is, unless you run it with "-S", of course,
but then you explicitly request it to be excluded.

> ... or to never use setuptools installed system-wide, be it an egg
> or a package
>
> I think the first would be cleanest and further, no local setuptools
> egg
> should be installed, if there is a system-wide one, that could be used
> instead (given its not out-dated or something).

actually i think your second suggestion would be make more sense here
as the buildout should be as self-contained as possible. it would of
course require another download, but that should hurt too much, since
usually buildout's gonna download a lot of things anyway. and this
way you'd never end up with a version of setuptools that is too old
for buildout, either.

best regards,

andi

--
zeidler it consulting - http://zitc.de/ - <email address hidden>
friedelstraße 31 - 12047 berlin - telefon +49 30 25563779
pgp key at http://zitc.de/pgp - http://wwwkeys.de.pgp.net/
plone 3.0.3 released! -- http://plone.org/products/plone

I notice the same thing, please read my post http://article.gmane.org/gmane.comp.web.zope.plone.user/78086 about this. This may relate to the issue.

Imo, having the right path order would resolve the problem without any more explicit support for isolation.

On Nov 29, 2007, at 10:46 AM, Balazs Ree wrote:
> Imo, having the right path order would resolve the problem without any
> more explicit support for isolation.

actually `site-packages` should never get injected into `sys.path` at
all, since python will handle this anyway.

andi

--
zeidler it consulting - http://zitc.de/ - <email address hidden>
friedelstraße 31 - 12047 berlin - telefon +49 30 25563779
pgp key at http://zitc.de/pgp - http://wwwkeys.de.pgp.net/
plone 3.0.3 released! -- http://plone.org/products/plone

Followup: I used to have a Debian global apt-get installation of setuptools in site-packages.
In addition, another global installation of setuptools as an egg, nto /usr/local/lib/site-packages, with a fresh version.

The latter one normally would override the previous one. This was the situation when the error happened at first.

After removing the Debian installation and repeating the buildout (bootstrap included) the problem goes away,
even though I _do_ have a globally installed Zope. That's very good.

So the problem is as Andi says concerned about setuptools (or maybe a lack of bootstrap on my side, but I don't think so. I can confirm later if needed.)

Are there any news on this?

Just ran into the problem again as I somehow ended up with a setuptools.egg-lnk in develop-eggs. python bootstrap.py did not pick that one up and therefore no site-packages was listed in bin/buildout. However, the buildout.cfg referenced develop-eggs, found setuptools and listed site-packages in ./bin/instance and many more.

The solution seems to be very easy: don't search setuptools anywhere, just get it, put it in eggs/ and use it.
Or does anybody see problems with that procedure?o

Interestingly, the problem also persists when using ./bin/python from virtualenv --no-site-packages . run inside the buildout directory.

On Mar 27, 2008, at 11:03 PM, Florian Friesdorf wrote:

> Are there any news on this?

No.

> Just ran into the problem again as I somehow ended up with a
> setuptools
> .egg-lnk in develop-eggs. python bootstrap.py did not pick that one up
> and therefore no site-packages was listed in bin/buildout. However,
> the
> buildout.cfg referenced develop-eggs, found setuptools and listed
> site-
> packages in ./bin/instance and many more.
>
> The solution seems to be very easy: don't search setuptools
> anywhere, just get it, put it in eggs/ and use it.
> Or does anybody see problems with that procedure?o
>
> Interestingly, the problem also persists when using ./bin/python from
> virtualenv --no-site-packages . run inside the buildout directory.

As I've mentioned before, the location os setuptools has nothing to do
with whether site-packages is used. site-packages is always used. I
plan to add an option to exclude site-packages. I haven't done this
yet. I have higher priorities. Reviewed checkins are accepted.

Jim

--
Jim Fulton
Zope Corporation

On Mar 28, 2008, at 2:49 PM, Jim Fulton wrote:
> As I've mentioned before, the location os setuptools has nothing to do
> with whether site-packages is used. site-packages is always used.

that's right, it is always used. however, normally it is only
implicitly added to the search path. using a globally installed
setuptools package makes buildout inject the path to `site-packages`
again explicitly into `sys.path`. the problem here is that this way
site-packages might get preferred over other versions of the same
package — please see the example transcript[*] i posted, which
demonstrates this behaviour. i've tried to keep it minimal, so it
should really only take a minute or two to see the problem...

imho, this is clearly a bug and could be easily resolved the way
florian suggested.

> I plan to add an option to exclude site-packages.

that won't help with this particular problem.

cheers,

andi

[*] https://bugs.launchpad.net/zc.buildout/+bug/158756/comments/7

--
zeidler it consulting - http://zitc.de/ - <email address hidden>
friedelstraße 31 - 12047 berlin - telefon +49 30 25563779
pgp key at http://zitc.de/pgp - http://wwwkeys.de.pgp.net/
plone 3.1rc1 released! -- http://plone.org/products/plone/

On Apr 2, 2008, at 3:51 PM, Andi Zeidler wrote:
> On Mar 28, 2008, at 2:49 PM, Jim Fulton wrote:
>> As I've mentioned before, the location os setuptools has nothing to
>> do
>> with whether site-packages is used. site-packages is always used.
>
> that's right, it is always used. however, normally it is only
> implicitly added to the search path. using a globally installed
> setuptools package makes buildout inject the path to `site-packages`
> again explicitly into `sys.path`. the problem here is that this way
> site-packages might get preferred over other versions of the same
> package — please see the example transcript[*] i posted, which
> demonstrates this behaviour. i've tried to keep it minimal, so it
> should really only take a minute or two to see the problem...

It took longer than that. The real problem, in this example, is that
system-installed eggs on ubuntu are installed in such a way that the
egg "location", the thing that gets added to the path is site-
packages. If you got any package this way, and it happened to be
earlier in the path, it would cause lots of things to be hidden
downstream. I hate this way of installing eggs. :(

> imho, this is clearly a bug and could be easily resolved the way
> florian suggested.

I don't agree. The same problem can occur any time you pick up a
"single-version-externally-managed" package. It is the case that
doing this during bootstrapping aggravates the problem. I could live
with a change to bootstrap.py that downloaded setuptools if the
already-installed setuptools was installed this way.

>> I plan to add an option to exclude site-packages.
>
> that won't help with this particular problem.

I guess it wouldn't help because then you'd lose setuptools. I can
see how moving "system packages" to the end of the path would help,
assuming you could figure out what they were. It would be easier to
simply move directories named "site-packages". That makes me
uncomfortable because it could cause an ordering that conflicts with
PYTHONPATH, but site-packages is at the end of sys.path under normal
circumstances.

So, I'm +1 on moving site-packages to the end of generated paths.
This would prevent system packages, including setuptools, from being
as viral as they are now.

Thanks for being persistent.

Jim

--
Jim Fulton
Zope Corporation

On Apr 2, 2008, at 11:50 PM, Jim Fulton wrote:
> The real problem, in this example, is that
> system-installed eggs on ubuntu are installed in such a way that the
> egg "location", the thing that gets added to the path is site-
> packages.

that doesn't only seem to be the case for ubuntu, but anyway, could
you give me a hint on how they're installed the wrong way? or rather,
what would be the correct way of installing them? is this about how
the .pth looks?

>> imho, this is clearly a bug and could be easily resolved the way
>> florian suggested.
>
> I don't agree. The same problem can occur any time you pick up a
> "single-version-externally-managed" package.

makes sense. so the bug's not necessarily in zc.buildout then... :)

> So, I'm +1 on moving site-packages to the end of generated paths.

> This would prevent system packages, including setuptools, from being
> as viral as they are now.

right, i'll see if i can find some time to submit a patch.

> Thanks for being persistent.

thanks for looking into it again.

andi

--
zeidler it consulting - http://zitc.de/ - <email address hidden>
friedelstraße 31 - 12047 berlin - telefon +49 30 25563779
pgp key at http://zitc.de/pgp - http://wwwkeys.de.pgp.net/
plone 3.1rc1 released! -- http://plone.org/products/plone/

On Apr 3, 2008, at 5:56 AM, Andi Zeidler wrote:
> On Apr 2, 2008, at 11:50 PM, Jim Fulton wrote:
>> The real problem, in this example, is that
>> system-installed eggs on ubuntu are installed in such a way that the
>> egg "location", the thing that gets added to the path is site-
>> packages.
>
> that doesn't only seem to be the case for ubuntu,

No, it's not just ububtu.

> but anyway, could
> you give me a hint on how they're installed the wrong way? or rather,
> what would be the correct way of installing them? is this about how
> the .pth looks?

They are following the recommended way. See "Creating system
packages", under

   http://peak.telecommunity.com/DevCenter/setuptools#what-your-users-should-know

In particular "--single-version-externally-managed".

The end result is that packages installed this way share a common
location, site-packages. I think this is extremely stupid.

If all you care about is installing one version of a distribution in a
system Python, this works fine, but it is a royal PITA for buildout.

Jim

--
Jim Fulton
Zope Corporation