markupsafe package not found

A quick check shows that http://pypi.openstack.org/simple/MarkupSafe/ does index the latest version of MarkupSafe and has done so since June 30th (I think that is when the bandersnatch mirror was initially built).

It is odd that pip couldn't find a package that worked given that the sdist is there. Could this be related to using markupsafe as the package name vs MarkupSafe? Maybe dstufft can enlighten us.

This is sort of a perfect storm kind of thing.

PyPI stores packages using the name that the author entered, instead of normalizing it. This means that the URL is expected to be exactly what the name of the package is, like "/simple/MarkupSafe/". PyPI itself has some smarts that will redirect things like "/simple/markupsafe/" to "/simple/MarkupSafe/" which enables people to be lax about what they use in their requirements.

Bandersnatch stores things like PyPI does, using the name that is on PyPI, so you'll again have things like "/simple/MarkupSafe/index.html", however unlike PyPI there is nothing that makes /simple/markupsafe/ redirect to /simple/MarkupSafe/. This will cause pip and setuptools to fallback to just /simple/ and look for links to something named MarkupSafe with the case insensitive/normalized matched, normally this would find it, then fetch /simple/MarkupSafe/ and everything would be happy.

However, with the move to remove external links, pip won't follow a link unless it's explicitly marked as internal and trusted, essentially this means that the fallback to /simple/ and detecting the /simple/MarkupSafe/ URL from that no longer works without the --allow-external and --allow-unverified flags.

The tl;dr of this is that it needs to be fixed both on PyPI and in pip so that PyPI stores things in normalized form so that there is a predeteremined representation of a particular name and it isn't just "whatever the author put in". Once PyPI does that, bandersnatch can sync it and then pip can be modified so it always looks for that normalized name.

Hit the same thing today for passlib:

http://logs.openstack.org/71/143771/5/check/check-tempest-dsvm-postgres-full/33b8d3f/logs/devstacklog.txt.gz#_2015-01-07_14_11_57_771

And others it looks like:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTm8gZGlzdHJpYnV0aW9ucyBhdCBhbGwgZm91bmQgZm9yXCIgQU5EIE5PVCBcIihmcm9tIHBicilcIiBBTkQgdGFnczpcImNvbnNvbGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyMDczMDAxMDQ4Nn0=

Related fix proposed to branch: master
Review: https://review.openstack.org/145808

Reviewed: https://review.openstack.org/145808
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=9fe4c2fd493417df681e74c88a7ff2e35e508ae7
Submitter: Jenkins
Branch: master

commit 9fe4c2fd493417df681e74c88a7ff2e35e508ae7
Author: Matt Riedemann <email address hidden>
Date: Thu Jan 8 07:15:04 2015 -0800

    Add query for infra/pypi mirror bug 1355050

    This has been around for awhile, sounds like
    it's something not easy to fix, so let's just
    track it.

    31 hits in 7 days, check and gate, all failures
    when filtering out the "from pbr" case.

    Change-Id: Ia31e0a06e7774a401dbed4f2d01d56b6fec060c1
    Related-Bug: #1355050

We should mark this invalid now, it doesn't appear to be an issue anymore.