Swift installation crashes

This is an important issue. I have not see this installation error before. Can you please provide the output of "pip freeze", or additional information about the environment/system you're installing into so I can attempt to duplicate the issue?

It's possible that a newer version of setuptools/distribute would fix the issue.

Oh, of course.
First I have tried to install Swift on CentOS 6.4 and then on Ubuntu 12.04 as in docs, and I have same error.
Machine with CentOS is dedicated server and machine with Ubuntu is a virtual machine in virtual box.

Output of "pip freeze" on machine with Ubuntu:

pip freeze
FormEncode==1.2.4
GnuPGInterface==0.3.2
Jinja2==2.6
Landscape-Client==12.05
MarkupSafe==0.15
PAM==0.4.2
PIL==1.1.7
Paste==1.7.5.1
PasteDeploy==1.5.0
PasteScript==1.7.5
Pygments==1.4
Sphinx==1.1.3
Twisted-Core==11.1.0
apt-xapian-index==0.44
argparse==1.2.1
chardet==2.0.1
command-not-found==0.2.44
coverage==3.4
d2to1==0.2.10
distribute==0.7.3
dnspython==1.9.4
docutils==0.8.1
eventlet==0.9.16
greenlet==0.3.1
httplib2==0.7.2
keyring==0.9.2
language-selector==0.1
launchpadlib==1.9.12
lazr.restfulclient==0.12.0
lazr.uri==1.0.3
lxml==2.3.2
mock==0.7.2
netifaces==0.6
nose==1.1.2
oauth==1.0.1
pbr==0.5.21
pyOpenSSL==0.12
pycrypto==2.4.1
pyserial==2.5
python-apt==0.8.3ubuntu7.1
python-debian==0.1.21ubuntu1
python-openid==2.2.5
-e git+https://github.com/openstack/python-swiftclient.git@ec14619ef05a2f2571805466956a1f0f567a1eea#egg=python_swiftclient-dev
scgi==1.13
simplejson==2.3.2
stevedore==0.10
-e git+https://github.com/openstack/swift.git@bc75061740465513887bf374ecf0056c383c12d9#egg=swift-dev
ufw==0.31.1-1
unattended-upgrades==0.1
virtualenv==1.10.1
virtualenv-clone==0.2.4
wadllib==1.3.0
wsgiref==0.1.2
xattr==0.6.2
zope.interface==3.6.1

Clay - FYI my tox broke here with my latest rebase and when I tried a new freshly cloned repo I get the same issue as reported above. Let me know if there's add't info I can provide as well. My setup is all screwed up now

pip freeze
Warning: cannot find svn location for distribute==0.6.24dev-r0
Brlapi==0.5.6
FormEncode==1.2.4
GnuPGInterface==0.3.2
Jinja2==2.7
Mako==0.5.0
MarkupSafe==0.15
PAM==0.4.2
PIL==1.1.7
Paste==1.7.5.1
PasteDeploy==1.5.0
PasteScript==1.7.5
Pygments==1.6
Sphinx==1.2b1
Twisted-Core==11.1.0
Twisted-Names==11.1.0
Twisted-Web==11.1.0
adium-theme-ubuntu==0.3.2
apt-xapian-index==0.44
apturl==0.5.1ubuntu3
argparse==1.2.1
cffi==0.7.2
chardet==2.0.1
colorama==0.2.5
command-not-found==0.2.44
configglue==1.0
coverage==3.4
d2to1==0.2.10
debtagshw==0.1
defer==1.0.2
dirspec==3.0.0
## FIXME: could not find svn URL in dependency_links for this package:
distribute==0.6.24dev-r0
dnspython==1.9.4
docutils==0.10
duplicity==0.6.18
eventlet==0.9.16
flake8==2.0
git-review==1.22
greenlet==0.3.1
httplib2==0.7.2
jockey==0.9.7
keyring==0.9.2
language-selector==0.1
launchpadlib==1.9.12
lazr.restfulclient==0.12.0
lazr.uri==1.0.3
louis==2.3.0
mccabe==0.2.1
mock==1.0.1
netifaces==0.6
nose==1.1.2
nosehtmloutput==0.0.4
nosexcover==1.0.8
nvidia-common==0.0.0
oauth==1.0.1
onboard==0.97.1
oneconf==0.2.8.1
openstack.nose-plugin==0.11
pbr==0.5.21
pep8==1.4.5
pexpect==2.3
piston-mini-client==0.7.2
protobuf==2.4.1
py==1.4.15
pyOpenSSL==0.12
pycparser==2.10
pycrypto==2.4.1
pycups==1.9.61
pycurl==7.19.0
pyflakes==0.7.2
pyinotify==0.9.2
pyserial==2.5
pysmbc==1.0.13
python-apt==0.8.3ubuntu7.1
python-dateutil==1.5
python-debian==0.1.21ubuntu1
python-openid==2.2.5
-e git+https://github.com/openstack/python-swiftclient.git@cc2fbf0c7435729c89d4b4f27cb7411a7a1cfe61#egg=python_swiftclient-master
python-virtkey==0.60.0
pyxdg==0.19
reportlab==2.5
rhythmbox-ubuntuone==3.0.0
scgi==1.13
sessioninstaller==0.0.0
setuptools-git==1.0b1
simplejson==2.3.2
software-center-aptd-plugins==0.0.0
-e git+https://github.com/openstack/swift.git@61a8a9af18f501e2b7940666b4131dfaf713d6a1#egg=swift-master
system-service==0.1.6
termcolor==1.1.0
tox==1.6.0
ubuntuone-couch==0.3.0
ubuntuone-installer==3.0.2
ubuntuone-storage-protocol==3.0.2
ufw==0.31.1-1
unattended-upgrades==0.1
unity-lens-video==0.3.5
unity-scope-video-remote==0.3.5
usb-creator==0.2.23
virtualenv==1.10.1
wadllib==1.3.0
wsgiref==0.1.2
xattr==0.6.2
xdiagnose==2.5.3
xkit==0.0.0
zope.interface==3.6.1

As a work around, for now, try to pre-install the dependencies with "sudo pip install -r requirements.txt"

So, new version of setuptools was released yesterday - https://pypi.python.org/pypi/setuptools/1.1

Looks a lot like: https://bugs.launchpad.net/openstack-ci/+bug/1203340

sudo pip install -r requirements.txt

Requirement already satisfied (use --upgrade to upgrade): pbr>=0.5.21,<1.0 in /usr/local/lib/python2.7/dist-packages/pbr-0.5.21-py2.7.egg (from -r requirements.txt (line 1))
Downloading/unpacking dnspython>=1.10.0 (from -r requirements.txt (line 2))
  Downloading dnspython-1.11.0.zip (220Kb): 220Kb downloaded
  Running setup.py egg_info for package dnspython

Requirement already satisfied (use --upgrade to upgrade): eventlet>=0.9.15 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): greenlet>=0.3.1 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): netifaces>=0.5 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): pastedeploy>=1.3.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): simplejson>=2.0.9 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): xattr>=0.4 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): python-swiftclient in /home/swift/python-swiftclient (from -r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): pip>=1.0 in /usr/lib/python2.7/dist-packages (from pbr>=0.5.21,<1.0->-r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): d2to1>=0.2.10,<0.3 in /usr/local/lib/python2.7/dist-packages/d2to1-0.2.10-py2.7.egg (from python-swiftclient->-r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg (from d2to1>=0.2.10,<0.3->python-swiftclient->-r requirements.txt (line 9))
Installing collected packages: dnspython
  Found existing installation: dnspython 1.9.4
    Uninstalling dnspython:
      Successfully uninstalled dnspython
  Running setup.py install for dnspython

Successfully installed dnspython
Cleaning up...

And after that I could install swift. Thank very much!

baah, mine is still messed up. I tried reinstalling a few other things last night and get a different error now. Will look it at closer after another cup of coffee

sudo pip install -r requirements.txt
Downloading/unpacking pbr>=0.5.21,<1.0 (from -r requirements.txt (line 1))
  Downloading pbr-0.5.21.tar.gz (123kB): 123kB downloaded
  Running setup.py egg_info for package pbr
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
      warnings.warn(msg)
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
      warnings.warn(msg)
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'include_package_data'
      warnings.warn(msg)
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'zip_safe'
      warnings.warn(msg)
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tests_require'
      warnings.warn(msg)
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found

    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/tmp/pip_build_root/pbr/setup.py", line 22, in <module>
        **util.cfg_to_args())
      File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
        dist.run_commands()
      File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "<string>", line 14, in replacement_run
      File "pbr/packaging.py", line 582, in find_sources
        mm.run()
      File "/usr/local/lib/python2.7/dist-packages/setuptools/command/egg_info.py", line 327, in run
        self.add_defaults()
      File "pbr/packaging.py", line 548, in add_defaults
        sdist.sdist.add_defaults(self)
      File "/usr/local/lib/python2.7/dist-packages/setuptools/command/sdist.py", line 225, in add_defaults
        if not self.distribution.include_package_data:
    AttributeError: Distribution instance has no attribute 'include_package_data'
    Complete output from command python setup.py egg_info:
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'

  warnings.warn(msg)

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'

  warnings.warn(msg)

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'include_package_data'

  warnings.warn(msg)

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'zip_safe'

  warnings.warn(msg)

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tests_require'

  warnings.warn(msg)

running egg_info

creating pip-egg-info/pbr.egg-info

[pbr] Processing SOURCES.txt

writing manifest file 'pip-egg-info/pbr.egg-info/SOURCES.txt'

warning: LocalManifestMaker: standard file '-c' not found

Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/tmp/pip_build_root/...

BTW, don't keep the bug open for me, my issue I'm sure would be resolved as well if I hadn't unisntalled/reinstalled so many other things trying to get thigns working

@paul - no worries, we need to get you working, and we need to figure out what's the fuss with the setuptools update too.

Personally, I didn't have good luck getting setuptools<1.1 installed. When I try to "pip install setuptools" it actually installs distribute, and when I try to "pip install distribute" it just pulls in "setuptools > 0.7" which grabs the latest setuptools. Very strange.

fyi the issue still exists - I just attempted a clean install using http://docs.openstack.org/developer/swift/development_saio.html in a fresh ubuntu 12.04 vm and bombed on the main build with the same error.

the workaround [sudo pip install -r requirements.txt] worked for me too...thanks!

"sudo pip install -e ." seems to work in place of "python setup.py develop" too.

I'm pretty sure I managed to get "sudo python setup.py develop" to work out of the box on a fresh vm once by some magical sequence of installation of pip, python-setuptools (distribute), and some older version of setuptools:

https://pypi.python.org/packages/source/s/setuptools/

But I can't seem to duplicate that.

Everything in the commit log from 1.0 to 1.1 seems pretty innocuous:

https://bitbucket.org/pypa/setuptools/commits/all

It has something to do with distutils/core.py's Distribution class not being patched to setuptools.dist.Distribution in time during the dependency installation, because the "TypeError: dist must be a Distribution instance" that's being raised is from the distutils code is lazy importing distutils.dist.Distribution and gets setuptools.dist.Distribution, but the "dist" instance is checking is_subclass on is really a distutils.dist.Distribution - because it was instantiated as such back in the dependencies call to setuptools.setup (which I guess is *really* distutils.core.setup).

What's curious is that when I drop a pdb.set_trace() in distutils.core around line # 102 to catch the case where Distribution.__module__.startswith('distutils') - I find it's *only* distutils.core.Distribution that's un-patched. sys.modules['distutils'].dist.Distribution verifies that there it is in fact setuptools.dist.Distribution, but that must have happened after this module imported it. If I reload(sys.modules['setuptools.dist']) the side-effect monkey patching fixes everything and the install completes just fine.

K, this works on a fresh clean precise vm:

    wget https://pypi.python.org/packages/source/s/setuptools/setuptools-0.9.8.tar.gz
    tar zxvf setuptools-0.9.8.tar.gz
    cd setuptools-0.9.8/
    sudo python setup.py install
    cd
    sudo easy_install pip
    sudo aptitude install libffi-dev
    git clone <email address hidden>:openstack/swift.git
    cd swift/
    sudo python setup.py install

I have an idea for a setuptools patch...

If I s/setuptools-0.9.8/setuptools-1.1/g those commands, the swift install fails with the Distribution TypeError

If I patch the setuptools source before I install it like so:

curl http://paste.openstack.org/raw/45403/ | patch -p1

It seems to work.

Anyone know how to work this thing:

https://bitbucket.org/pypa/setuptools

Filed bug upstream:

https://bitbucket.org/pypa/setuptools/issue/73/

We also stumbled onto that bug in swiftsync, tried setuptools 0.9.8, 1.1, 1.1.7 without any luck: some requirements still trigger that TypeError.
Thus I can't find a clean way around this bug other than doing a manual "pip install -e" before the "setup.py install".

Anyway, the bug filed upstream is now marked 'invalid'... So any ideas what should we do now ?

It the bug is instigated by monkey patching "pbr", which probably could use some fixing.

A setuptools maintainer, Jason R. Coombs, was so kind as to do some digging:

https://bitbucket.org/pypa/setuptools/issue/73/typeerror-dist-must-be-a-distribution#comment-7517002

I was getting this same issue on previous versions of ubuntu. But looks like the problem is fixed in Ubuntu Trusty!

I now have SAIO running inside an Ubuntu Trusty LXC container.

If it is a problem is with setuptools, then the version of python-setuptools in trusty has been updated. it is now version 3.3:

<snip>
$ pip list |grep setuptools
setuptools (3.3)
</snip>

Is there a definitive way to install swift on ubuntu 12.0.4 then? We could not install it on 12.0.4 but could do so on 14.04 as mentioned in this thread. However we are required to be on 12.0.4 due to some other dependencies. Is there a way out to do so?

thanks,
sameer

try to install system packaged python-pip (sudo apt-get install python-pip), i believe on precise it's new enough that pbr can use it.

If you already have pip, then try `pip install --upgrade setuptools`

Alternatively you might be able to `pip install -e .` instead of `python setup.py develop`

If none of these suit you try 'install' instead of 'develop'

I'm going to try and file a bug on pbr:

https://bugs.launchpad.net/pbr/+bug/1324784

Hey Sameer,

You can install swift on Ubuntu 12.04 by adding 1 additional command.
In the SAIO instructions before running command 2 in "Getting the code" (after running step 1), run:
      sudo pip install -r $HOME/python-swiftclient/requirements.txt
Which can be shortened to:
      sudo pip install $HOME/python-swiftclient/

Or if it's easier, replace the command in step 2 in "Getting the code" with:
      cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd -

Regards,
Matt

Reviewed: https://review.openstack.org/103446
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=7904358a43b196e8b3afb1213ec119cbe7a6c2db
Submitter: Jenkins
Branch: master

commit 7904358a43b196e8b3afb1213ec119cbe7a6c2db
Author: Matthew Oliver <email address hidden>
Date: Mon Jun 30 12:55:37 2014 +1000

    Add SAIO fix for Ubuntu 12.04

    The version of setup tools in Ubuntu 12.04 is too old and is causing
    the SAIO instructions to fail when installing python-swiftclient.

    The work around is to install python-swiftclient's dependencies
    before running running:

      python setup.py develop

    This change adds a note to users of Ubuntu 12.04 to replace step 2
    of "Getting the code" with:

      cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; \
                                   python setup.py develop; cd -

    Change-Id: I63f57bbf1f1158f8740f6137ad55ff49f12a316c
    Closes-Bug: #1217288

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/105536

Reviewed: https://review.openstack.org/105536
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fd19daee7f042f8f98ab71a082784121e50f0c51
Submitter: Jenkins
Branch: feature/ec

commit 2f45600c7db48417626ff4152ec46f8d4054a8a6
Author: YummyBian <email address hidden>
Date: Sun Jun 29 22:03:33 2014 +0800

    Document the unnecessary method invoking in FakeRing

    Add comment to explain why we invoke the get_part method even if the
    _part_shift is equal to 32.

    Closes-Bug: #1335581

    Change-Id: I160e9383b5e65f75ed5e89511cc7e63c51958a25

commit f6ff06b6782ba0682e16f2015d77ea6fb53c2d5b
Author: Christian Berendt <email address hidden>
Date: Fri May 30 00:06:44 2014 +0200

    Use except x as y instead of except x, y

    According to https://docs.python.org/3/howto/pyporting.html the
    syntax changed in Python 3.x. The new syntax is usable with
    Python >= 2.6 and should be preferred to be compatible with Python3.

    Enabled hacking check H231.

    Change-Id: I2c41dc3ec83e79181e8fd50e76771a74c393269c

commit 08006685577c0e3c85f0709945175f0c8689ae49
Author: Paul Luse <email address hidden>
Date: Wed Jul 2 14:39:42 2014 -0700

    Fix potential missing key error in container_info

    If upgrading from a non-storage policy enabled version of
    swift to a storage policy enabled version its possible that
    memcached will have an info structure that does not contain
    the 'storage_policy" key resulting in an unhandled exception
    during the lookup. The fix is to simply make sure we never
    return the dict without a storage_policy key defined; if it
    doesn't exist its safe to make it '0' as this means you're
    in the update scenario and there's xno other possibility.

    Change-Id: If8e8f66d32819c5bfb2d1308e14643f3600ea6e9

commit b823e1602e4c5cb6bcf5360b3e1f6e8410e46401
Author: Samuel Merritt <email address hidden>
Date: Wed Jul 2 11:37:26 2014 -0700

    Fix exception raising in FakeConn

    Timeout isn't an Exception, so Timeouts in tests weren't getting
    raised. Instead, you'd sometimes have an HTTPResponse's .status be a
    Timeout object, not an integer, which greatly confuses code that
    expects an integer.

    Also reorder the test that exposed the failure in the gate so it blows
    up most times instead of sometimes do demonstrate the failure with out
    this fix to FakeConn.

    Change-Id: I76367a0575f84cad6b2f03e814f3f16bf96bc7d1

commit 620ff9b6738797f20b992a027301c32cf0dee17c
Author: Clay Gerrard <email address hidden>
Date: Wed Jul 2 12:20:05 2014 -0700

    Fix order dependent test in proxy.test_server

    TestObjectController.test_POST_backend_headers was being too picky about the
    order of backend requests which when pushed through eventlet will not have a
    stable order. This change preserves the expectations and assertions while
    removing the dependency on the order of the requests.

    Change-Id: I7176ccb9223cd3dfc3c922b8b3d81eb514891d05

commit 8a3b65107d788a8590349fc4aa02e5c8c2ad9202
Author: Clay Gerrard <email address hidden>
Date: Mon Jun 30 21:49:49 2014 -0700

    Fix pop_queue for ...