pypi-mirror does not pickup all necessary python packages

Bug #1294381 reported by Steven DuChene
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Fix Committed
Gregory Haynes

Bug Description

Following all of the directions at:

Once I went to use the resulting mirror referenced from another system with a fresh checkout of the tripleo-incubator sources, the resulting run would error out complaining there was no suitable pip installable source.

Downloading/unpacking pystache (from os-apply-config)
  Running egg_info for package pystache
    pystache: using: version '3.3' of <module 'setuptools' from '/root/.cache/tripleo/tripleo-incubator/openstack-tools/local/lib/python2.7/site-packages/setuptools/__init__.pyc'>

Could not find any downloads that satisfy the requirement pip>=1.0 in ./tripleo-incubator/openstack-tools/lib/python2.7/site-packages (from pbr>=0.5.21,<1.0->python-cinderclient)
Downloading/unpacking pip>=1.0 (from pbr>=0.5.21,<1.0->python-cinderclient)
Cleaning up...
No distributions at all found for pip>=1.0 in ./tripleo-incubator/openstack-tools/lib/python2.7/site-packages (from pbr>=0.5.21,<1.0->python-cinderclient)

I looked around and found the only files under /mirror/pip/ was an index.html file and a pip-1.5.4-py2.py3-none-any.whl file.

To solve the immediate problem I went out to and downloaded pip-1.5.4.tar.gz, put it in the directory, did an md5 sum of it, and then added the appropriate line to the index.html file in the same directory.

In short if run-mirror was doing the right thing then I would not have had to do this manual step of adding pip-1.5.4.tar.gz to the mirror.

I am including the $HOME/.pip/pip.log file where the error occurance is shown.

Tags: pypi-mirror
Revision history for this message
Steven DuChene (steven-a-duchene) wrote :
Revision history for this message
Clark Boylan (cboylan) wrote :

I think the problem here is pip and friends are special cased in pypi-mirror. In that special case they do not use the --no-use-wheel flag to pip so you only get wheels for these packages. and should probably have --no-use-wheel passed to pip as well. Though that second run on line 310 doesn't have a corresponding pip wheel so you may need to add that to the second run too.

summary: - run-mirror does not pickup all necessary python packages
+ pypi-mirror does not pickup all necessary python packages
Changed in openstack-ci:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Manuel Desbonnet (manuel-desbonnet) wrote :

I don't think adding --no-use-wheel to the 'upgrade_format' command (lines 260/310 above) will fix this.
If the particular package (specifically pip) is already at the latest available version in the virtualenv, pip does nothing.

I have been using the following patch locally to get around this issue:

Revision history for this message
Clark Boylan (cboylan) wrote :

I think 84081 fixes this as a side effect or pip being a transitive dependency for projects (pbr depends on it). The explicit installs of pip, virtualenv, wheel and friends does not use the pip command, they use the upgrade command. We should make sure that the explicit installs of pip and friends do the correct thing instead of relying on the convenient but potentially unreliable behavior 84041 exploits.

Not only that but we don't actually want to reinstall every package. Mirror builds are much quicker the second time around because it takes advantage of precached packages. If we must reinstall pip and friends we should only reinstall pip and friends and not the others.

Revision history for this message
Manuel Desbonnet (manuel-desbonnet) wrote :

Good point about the caching.

(I'm not sure I quite understand your point about pip being a transitive dependency - it seems to me that if its listed as a requirement it should be added to the mirror regardless of why it's there - but I guess that's moot in light of what --force-reinstall does with caching).

Is the solution to add the --no-use-wheel and --force-reinstall flags to the 'upgrade_format' template ?

James E. Blair (corvus)
tags: added: pypi-mirror
Revision history for this message
Clark Boylan (cboylan) wrote :

The openstack infra team is no longer using this tool. I am marking the bug wontfix as we have no intention of continuing to support and feed this tool. Potential future maintainers are more than welcome to setup a bug tracker for this project and suck this bug in though.

Changed in openstack-ci:
status: Triaged → Won't Fix
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to pypi-mirror (master)

Fix proposed to branch: master

Changed in openstack-ci:
assignee: nobody → Gregory Haynes (greghaynes)
status: Won't Fix → In Progress
James E. Blair (corvus)
Changed in openstack-ci:
status: In Progress → Won't Fix
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to pypi-mirror (master)

Submitter: Jenkins
Branch: master

commit fb16c5dc6845374b06b2ffa60f0d49038499c2dc
Author: Gregory Haynes <email address hidden>
Date: Tue Jul 8 21:12:35 2014 -0700

    Force --no-use-mirror installaions

    Some packages require this in order to get an sdist installation.

    Change-Id: I8e6fca71f13835b5019366c34535c64c3c58c6ea
    Closes-Bug: #1294381

Changed in openstack-ci:
status: Won't Fix → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.