Bug #1270710 “sporadic pip timeouts during download” : Bugs : OpenStack-Gate

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-20: Related fix proposed to elastic-recheck (master)

#1

Related fix proposed to branch: master
Review: https://review.openstack.org/67764

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-22: Related fix merged to elastic-recheck (master)

#2

Reviewed: https://review.openstack.org/67764
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=410158631dc8fdbfe323c2f2665d9a6927f31493
Submitter: Jenkins
Branch: master

commit 410158631dc8fdbfe323c2f2665d9a6927f31493
Author: Joe Gordon <email address hidden>
Date: Sun Jan 19 22:00:51 2014 -0800

Add query for bug 1270710

bug 1270710 is an infrequent CI bug (11 hits at the time this was
written).

Change-Id: I3ffc57578b21646ea75e45a0c451abf8b73dfdd2
Related-Bug: #1270710

Revision history for this message

Peter Portante (peter-a-portante) wrote on 2014-02-04:

#3

This might share the same root cause with bug 1275994.

James E. Blair (corvus) on 2014-02-04

Changed in openstack-ci:
status:	New → Incomplete
importance:	Undecided → Medium
tags:	added: gate-failure

Revision history for this message

Launchpad Janitor (janitor) wrote on 2014-04-06:

#4

[Expired for OpenStack Core Infrastructure because there has been no activity for 60 days.]

Changed in openstack-ci:
status:	Incomplete → Expired

Revision history for this message

melanie witt (melwitt) wrote on 2014-07-07:

#5

I just hit this bug but the logstash query doesn't pick it up because the line number is different for the _download_url in pip:

Traceback (most recent call last):
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
    do_download,
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
    self.session,
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/download.py", line 572, in unpack_http_url
    download_hash = _download_url(resp, link, temp_location)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/download.py", line 433, in _download_url
    for chunk in resp_read(4096):
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/download.py", line 421, in resp_read
    chunk_size, decode_content=False):
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
    data = self._fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
timeout: timed out

http://logs.openstack.org/72/101972/2/check/check-tempest-dsvm-neutron/b2e0b17/console.html

I just hit this bug but the logstash query doesn't pick it up because the line number is different for the _download_url in pip:

Traceback (most recent call last):
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
    do_download,
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
    self.session,
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/download.py", line 572, in unpack_http_url
    download_hash = _download_url(resp, link, temp_location)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/download.py", line 433, in _download_url
    for chunk in resp_read(4096):
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/download.py", line 421, in resp_read
    chunk_size, decode_content=False):
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/opt/stack/new/tempest/.tox/smoke-serial/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
    data = self._fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
timeout: timed out

http://logs.openstack.org/72/101972/2/check/check-tempest-dsvm-neutron/b2e0b17/console.html

Revision history for this message

Julie Pichon (jpichon) wrote on 2014-07-09:

#6

Here, too:

2014-07-09 09:36:33.940 | Cleaning up...
2014-07-09 09:36:33.940 | Exception:
2014-07-09 09:36:33.940 | Traceback (most recent call last):
2014-07-09 09:36:33.940 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
2014-07-09 09:36:33.940 | status = self.run(options, args)
2014-07-09 09:36:33.940 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
2014-07-09 09:36:33.940 | requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-07-09 09:36:33.941 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
2014-07-09 09:36:33.941 | do_download,
2014-07-09 09:36:33.941 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
2014-07-09 09:36:33.941 | self.session,
2014-07-09 09:36:33.942 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/download.py", line 572, in unpack_http_url
2014-07-09 09:36:33.942 | download_hash = _download_url(resp, link, temp_location)
2014-07-09 09:36:33.942 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/download.py", line 433, in _download_url
2014-07-09 09:36:33.942 | for chunk in resp_read(4096):
2014-07-09 09:36:33.942 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/download.py", line 421, in resp_read
2014-07-09 09:36:33.942 | chunk_size, decode_content=False):
2014-07-09 09:36:33.943 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
2014-07-09 09:36:33.943 | data = self.read(amt=amt, decode_content=decode_content)
2014-07-09 09:36:33.943 | File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
2014-07-09 09:36:33.943 | data = self._fp.read(amt)
2014-07-09 09:36:33.943 | File "/usr/lib/python2.7/httplib.py", line 561, in read
2014-07-09 09:36:33.943 | s = self.fp.read(amt)
2014-07-09 09:36:33.944 | File "/usr/lib/python2.7/socket.py", line 380, in read
2014-07-09 09:36:33.944 | data = self._sock.recv(left)
2014-07-09 09:36:33.944 | timeout: timed out

http://logs.openstack.org/76/105476/1/gate/gate-horizon-python27-django14/15756a1/console.html

Here, too:

2014-07-09 09:36:33.940 | Cleaning up...
2014-07-09 09:36:33.940 | Exception:
2014-07-09 09:36:33.940 | Traceback (most recent call last):
2014-07-09 09:36:33.940 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
2014-07-09 09:36:33.940 |     status = self.run(options, args)
2014-07-09 09:36:33.940 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
2014-07-09 09:36:33.940 |     requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-07-09 09:36:33.941 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
2014-07-09 09:36:33.941 |     do_download,
2014-07-09 09:36:33.941 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
2014-07-09 09:36:33.941 |     self.session,
2014-07-09 09:36:33.942 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/download.py", line 572, in unpack_http_url
2014-07-09 09:36:33.942 |     download_hash = _download_url(resp, link, temp_location)
2014-07-09 09:36:33.942 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/download.py", line 433, in _download_url
2014-07-09 09:36:33.942 |     for chunk in resp_read(4096):
2014-07-09 09:36:33.942 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/download.py", line 421, in resp_read
2014-07-09 09:36:33.942 |     chunk_size, decode_content=False):
2014-07-09 09:36:33.943 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
2014-07-09 09:36:33.943 |     data = self.read(amt=amt, decode_content=decode_content)
2014-07-09 09:36:33.943 |   File "/home/jenkins/workspace/gate-horizon-python27-django14/.tox/py27dj14/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
2014-07-09 09:36:33.943 |     data = self._fp.read(amt)
2014-07-09 09:36:33.943 |   File "/usr/lib/python2.7/httplib.py", line 561, in read
2014-07-09 09:36:33.943 |     s = self.fp.read(amt)
2014-07-09 09:36:33.944 |   File "/usr/lib/python2.7/socket.py", line 380, in read
2014-07-09 09:36:33.944 |     data = self._sock.recv(left)
2014-07-09 09:36:33.944 | timeout: timed out

http://logs.openstack.org/76/105476/1/gate/gate-horizon-python27-django14/15756a1/console.html

Revision history for this message

melanie witt (melwitt) wrote on 2014-07-09:

#8

Here is the logstash query to find more hits last 15m:

tags:"console" AND message:"download.py\", line 433"

http://logstash.openstack.org/#eyJzZWFyY2giOiJ0YWdzOlwiY29uc29sZVwiIEFORCBtZXNzYWdlOlwiZG93bmxvYWQucHlcXFwiLCBsaW5lIDQzM1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI5MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA0OTM4MDI2OTQ3fQ==

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-09: Related fix proposed to elastic-recheck (master)

#9

Related fix proposed to branch: master
Review: https://review.openstack.org/105884

Revision history for this message

melanie witt (melwitt) wrote on 2014-07-09:

#10

Setting this back to Incomplete from Expired, since it's still happening.

Changed in openstack-ci:
status:	Expired → Incomplete

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-10: Related fix merged to elastic-recheck (master)

#11

Reviewed: https://review.openstack.org/105884
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=8cc901a458d6f74bca8415b83deb4cebb3a7741e
Submitter: Jenkins
Branch: master

commit 8cc901a458d6f74bca8415b83deb4cebb3a7741e
Author: melanie witt <email address hidden>
Date: Wed Jul 9 21:25:05 2014 +0000

Extend query for pip download timeout bug 1270710

For python2.6, the download error message occurs on line 495.
For python2.7, the download error message occurs on line 433.

Change-Id: I08224e2b87e23f1c5af05928cc0e32b602b7a09d
Related-Bug: #1270710

Revision history for this message

Matt Riedemann (mriedem) wrote on 2014-07-25:

#12

Download full text (4.6 KiB)

Seeing something a bit different here:

http://logs.openstack.org/06/102006/9/check/gate-nova-python26/49e6d5f/console.html

2014-07-23 16:40:43.226 | Downloading/unpacking pbr>=0.6,!=0.7,<1.0 (from nova==2014.2.dev3.gecfe94d)
2014-07-23 16:40:43.226 | Cleaning up...
2014-07-23 16:40:43.227 | Exception:
2014-07-23 16:40:43.227 | Traceback (most recent call last):
2014-07-23 16:40:43.227 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/basecommand.py", line 122, in main
2014-07-23 16:40:43.227 | status = self.run(options, args)
2014-07-23 16:40:43.227 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/commands/install.py", line 278, in run
2014-07-23 16:40:43.227 | requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-07-23 16:40:43.227 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/req.py", line 1177, in prepare_files
2014-07-23 16:40:43.227 | url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
2014-07-23 16:40:43.227 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 196, in find_requirement
2014-07-23 16:40:43.227 | url_name = self._find_url_name(Link(self.index_urls[0], trusted=True), url_name, req) or req.url_name
2014-07-23 16:40:43.227 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 361, in _find_url_name
2014-07-23 16:40:43.228 | page = self._get_page(index_url, req)
2014-07-23 16:40:43.228 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 568, in _get_page
2014-07-23 16:40:43.228 | session=self.session,
2014-07-23 16:40:43.228 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 670, in get_page
2014-07-23 16:40:43.228 | resp = session.get(url, headers={"Accept": "text/html"})
2014-07-23 16:40:43.228 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/sessions.py", line 468, in get
2014-07-23 16:40:43.228 | return self.request('GET', url, **kwargs)
2014-07-23 16:40:43.228 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/download.py", line 237, in request
2014-07-23 16:40:43.228 | return super(PipSession, self).request(method, url, *args, **kwargs)
2014-07-23 16:40:43.228 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/sessions.py", line 456, in request
2014-07-23 16:40:43.228 | resp = self.send(prep, **send_kwargs)
2014-07-23 16:40:43.229 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
2014-07-23 16:40:43.229 | r.content
2014-07-23 16:40:43.229 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/models.py", line 694, in content
2014-07-23 16:...

Seeing something a bit different here:

http://logs.openstack.org/06/102006/9/check/gate-nova-python26/49e6d5f/console.html

2014-07-23 16:40:43.226 | Downloading/unpacking pbr>=0.6,!=0.7,<1.0 (from nova==2014.2.dev3.gecfe94d)
2014-07-23 16:40:43.226 | Cleaning up...
2014-07-23 16:40:43.227 | Exception:
2014-07-23 16:40:43.227 | Traceback (most recent call last):
2014-07-23 16:40:43.227 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/basecommand.py", line 122, in main
2014-07-23 16:40:43.227 |     status = self.run(options, args)
2014-07-23 16:40:43.227 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/commands/install.py", line 278, in run
2014-07-23 16:40:43.227 |     requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-07-23 16:40:43.227 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/req.py", line 1177, in prepare_files
2014-07-23 16:40:43.227 |     url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
2014-07-23 16:40:43.227 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 196, in find_requirement
2014-07-23 16:40:43.227 |     url_name = self._find_url_name(Link(self.index_urls[0], trusted=True), url_name, req) or req.url_name
2014-07-23 16:40:43.227 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 361, in _find_url_name
2014-07-23 16:40:43.228 |     page = self._get_page(index_url, req)
2014-07-23 16:40:43.228 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 568, in _get_page
2014-07-23 16:40:43.228 |     session=self.session,
2014-07-23 16:40:43.228 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/index.py", line 670, in get_page
2014-07-23 16:40:43.228 |     resp = session.get(url, headers={"Accept": "text/html"})
2014-07-23 16:40:43.228 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/sessions.py", line 468, in get
2014-07-23 16:40:43.228 |     return self.request('GET', url, **kwargs)
2014-07-23 16:40:43.228 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/download.py", line 237, in request
2014-07-23 16:40:43.228 |     return super(PipSession, self).request(method, url, *args, **kwargs)
2014-07-23 16:40:43.228 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/sessions.py", line 456, in request
2014-07-23 16:40:43.228 |     resp = self.send(prep, **send_kwargs)
2014-07-23 16:40:43.229 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
2014-07-23 16:40:43.229 |     r.content
2014-07-23 16:40:43.229 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/models.py", line 694, in content
2014-07-23 16:40:43.229 |     self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
2014-07-23 16:40:43.229 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/models.py", line 627, in generate
2014-07-23 16:40:43.229 |     for chunk in self.raw.stream(chunk_size, decode_content=True):
2014-07-23 16:40:43.229 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
2014-07-23 16:40:43.229 |     data = self.read(amt=amt, decode_content=decode_content)
2014-07-23 16:40:43.229 |   File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
2014-07-23 16:40:43.229 |     data = self._fp.read(amt)
2014-07-23 16:40:43.229 |   File "/usr/lib64/python2.6/httplib.py", line 522, in read
2014-07-23 16:40:43.230 |     return self._read_chunked(amt)
2014-07-23 16:40:43.230 |   File "/usr/lib64/python2.6/httplib.py", line 580, in _read_chunked
2014-07-23 16:40:43.230 |     value.append(self._safe_read(chunk_left))
2014-07-23 16:40:43.230 |   File "/usr/lib64/python2.6/httplib.py", line 619, in _safe_read
2014-07-23 16:40:43.230 |     chunk = self.fp.read(min(amt, MAXAMOUNT))
2014-07-23 16:40:43.230 |   File "/usr/lib64/python2.6/socket.py", line 383, in read
2014-07-23 16:40:43.230 |     data = self._sock.recv(left)
2014-07-23 16:40:43.230 | timeout: timed out

Revision history for this message

Dolph Mathews (dolph) wrote on 2014-07-25:

#13

Another similar traceback:

Downloading/unpacking SQLAlchemy>=0.7.8,!=0.9.5,<=0.9.99 (from keystone==2014.1.2.dev12.g077dd36)
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 196, in find_requirement
    url_name = self._find_url_name(Link(self.index_urls[0], trusted=True), url_name, req) or req.url_name
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 361, in _find_url_name
    page = self._get_page(index_url, req)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 468, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 596, in send
    r.content
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/models.py", line 694, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/models.py", line 627, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
    data = self._fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 601, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
timeout: timed out

Another similar traceback:

Downloading/unpacking SQLAlchemy>=0.7.8,!=0.9.5,<=0.9.99 (from keystone==2014.1.2.dev12.g077dd36)
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 196, in find_requirement
    url_name = self._find_url_name(Link(self.index_urls[0], trusted=True), url_name, req) or req.url_name
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 361, in _find_url_name
    page = self._get_page(index_url, req)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 468, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py", line 596, in send
    r.content
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/models.py", line 694, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/models.py", line 627, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
    data = self._fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 601, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
timeout: timed out

Revision history for this message

jiang, yunhong (yunhong-jiang) wrote on 2014-07-30:

#14

Download full text (3.3 KiB)

Another similar trace at http://logs.openstack.org/53/76053/13/check/check-tempest-dsvm-full/e425b78/console.html :

2014-07-30 22:24:08.194 | Requirement already up-to-date: jsonschema>=2.0.0,<3.0.0 in /usr/local/lib/python2.7/dist-packages (from tempest==2.dev714.g4d90d67)
2014-07-30 22:24:08.194 | http://pypi.openstack.org/simple/testtools/ uses an insecure transport scheme (http). Consider using https if pypi.openstack.org has it available
2014-07-30 22:24:08.194 | Requirement already up-to-date: testtools>=0.9.34 in /usr/local/lib/python2.7/dist-packages (from tempest==2.dev714.g4d90d67)
2014-07-30 22:24:08.195 | http://pypi.openstack.org/simple/lxml/ uses an insecure transport scheme (http). Consider using https if pypi.openstack.org has it available
2014-07-30 22:24:08.195 | Downloading/unpacking lxml>=2.3 from http://pypi.openstack.org/packages/source/l/lxml/lxml-3.3.5.tar.gz#md5=88c75f4c73fc8f59c9ebb17495044f2f (from tempest==2.dev714.g4d90d67)
2014-07-30 22:24:08.195 | Cleaning up...
2014-07-30 22:24:08.195 | Exception:
2014-07-30 22:24:08.195 | Traceback (most recent call last):
2014-07-30 22:24:08.195 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
2014-07-30 22:24:08.195 | status = self.run(options, args)
2014-07-30 22:24:08.195 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
2014-07-30 22:24:08.195 | requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-07-30 22:24:08.195 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
2014-07-30 22:24:08.195 | do_download,
2014-07-30 22:24:08.195 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
2014-07-30 22:24:08.196 | self.session,
2014-07-30 22:24:08.196 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/download.py", line 546, in unpack_http_url
2014-07-30 22:24:08.196 | resp = session.get(target_url, stream=True)
2014-07-30 22:24:08.196 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 468, in get
2014-07-30 22:24:08.196 | return self.request('GET', url, **kwargs)
2014-07-30 22:24:08.196 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/download.py", line 237, in request
2014-07-30 22:24:08.196 | return super(PipSession, self).request(method, url, *args, **kwargs)
2014-07-30 22:24:08.196 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 456, in request
2014-07-30 22:24:08.196 | resp = self.send(prep, **send_kwargs)
2014-07-30 22:24:08.196 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 559, in send
2014-07-30 22:24:08.196 | r = adapter.send(request, **kwargs)
2014-07-30 22:24:08.197 | File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/adap...

Another similar trace at http://logs.openstack.org/53/76053/13/check/check-tempest-dsvm-full/e425b78/console.html :

2014-07-30 22:24:08.194 | Requirement already up-to-date: jsonschema>=2.0.0,<3.0.0 in /usr/local/lib/python2.7/dist-packages (from tempest==2.dev714.g4d90d67)
2014-07-30 22:24:08.194 | http://pypi.openstack.org/simple/testtools/ uses an insecure transport scheme (http). Consider using https if pypi.openstack.org has it available
2014-07-30 22:24:08.194 | Requirement already up-to-date: testtools>=0.9.34 in /usr/local/lib/python2.7/dist-packages (from tempest==2.dev714.g4d90d67)
2014-07-30 22:24:08.195 | http://pypi.openstack.org/simple/lxml/ uses an insecure transport scheme (http). Consider using https if pypi.openstack.org has it available
2014-07-30 22:24:08.195 | Downloading/unpacking lxml>=2.3 from http://pypi.openstack.org/packages/source/l/lxml/lxml-3.3.5.tar.gz#md5=88c75f4c73fc8f59c9ebb17495044f2f (from tempest==2.dev714.g4d90d67)
2014-07-30 22:24:08.195 | Cleaning up...
2014-07-30 22:24:08.195 | Exception:
2014-07-30 22:24:08.195 | Traceback (most recent call last):
2014-07-30 22:24:08.195 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
2014-07-30 22:24:08.195 |     status = self.run(options, args)
2014-07-30 22:24:08.195 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
2014-07-30 22:24:08.195 |     requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-07-30 22:24:08.195 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
2014-07-30 22:24:08.195 |     do_download,
2014-07-30 22:24:08.195 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
2014-07-30 22:24:08.196 |     self.session,
2014-07-30 22:24:08.196 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/download.py", line 546, in unpack_http_url
2014-07-30 22:24:08.196 |     resp = session.get(target_url, stream=True)
2014-07-30 22:24:08.196 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 468, in get
2014-07-30 22:24:08.196 |     return self.request('GET', url, **kwargs)
2014-07-30 22:24:08.196 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/download.py", line 237, in request
2014-07-30 22:24:08.196 |     return super(PipSession, self).request(method, url, *args, **kwargs)
2014-07-30 22:24:08.196 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 456, in request
2014-07-30 22:24:08.196 |     resp = self.send(prep, **send_kwargs)
2014-07-30 22:24:08.196 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 559, in send
2014-07-30 22:24:08.196 |     r = adapter.send(request, **kwargs)
2014-07-30 22:24:08.197 |   File "/opt/stack/new/tempest/.tox/full/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 384, in send
2014-07-30 22:24:08.197 |     raise Timeout(e, request=request)
2014-07-30 22:24:08.197 | Timeout: HTTPConnectionPool(host='pypi.openstack.org', port=80): Read timed out. (read timeout=15)

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2014-08-11:

#15

Seen this 2-3 times today:

2014-08-11 09:56:10.594 | Downloading/unpacking msgpack-python>=0.4.0 (from ceilometer==2014.2.dev68.g9936b34)
2014-08-11 09:56:10.648 | http://pypi.openstack.org/simple/msgpack-python/ uses an insecure transport scheme (http). Consider using https if pypi.openstack.org has it available
2014-08-11 09:56:25.925 | Cleaning up...
2014-08-11 09:56:25.929 | Exception:
2014-08-11 09:56:25.929 | Traceback (most recent call last):
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
2014-08-11 09:56:25.929 | status = self.run(options, args)
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
2014-08-11 09:56:25.929 | requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1197, in prepare_files
2014-08-11 09:56:25.929 | do_download,
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1375, in unpack_url
2014-08-11 09:56:25.929 | self.session,
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 572, in unpack_http_url
2014-08-11 09:56:25.929 | download_hash = _download_url(resp, link, temp_location)
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 433, in _download_url
2014-08-11 09:56:25.929 | for chunk in resp_read(4096):
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 421, in resp_read
2014-08-11 09:56:25.929 | chunk_size, decode_content=False):
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
2014-08-11 09:56:25.929 | data = self.read(amt=amt, decode_content=decode_content)
2014-08-11 09:56:25.929 | File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
2014-08-11 09:56:25.929 | data = self._fp.read(amt)
2014-08-11 09:56:25.929 | File "/usr/lib/python2.7/httplib.py", line 567, in read
2014-08-11 09:56:25.929 | s = self.fp.read(amt)
2014-08-11 09:56:25.929 | File "/usr/lib/python2.7/socket.py", line 380, in read
2014-08-11 09:56:25.929 | data = self._sock.recv(left)
2014-08-11 09:56:25.929 | timeout: timed out

Seen this 2-3 times today:

2014-08-11 09:56:10.594 | Downloading/unpacking msgpack-python>=0.4.0 (from ceilometer==2014.2.dev68.g9936b34)
2014-08-11 09:56:10.648 |   http://pypi.openstack.org/simple/msgpack-python/ uses an insecure transport scheme (http). Consider using https if pypi.openstack.org has it available
2014-08-11 09:56:25.925 | Cleaning up...
2014-08-11 09:56:25.929 | Exception:
2014-08-11 09:56:25.929 | Traceback (most recent call last):
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
2014-08-11 09:56:25.929 |     status = self.run(options, args)
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
2014-08-11 09:56:25.929 |     requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1197, in prepare_files
2014-08-11 09:56:25.929 |     do_download,
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1375, in unpack_url
2014-08-11 09:56:25.929 |     self.session,
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 572, in unpack_http_url
2014-08-11 09:56:25.929 |     download_hash = _download_url(resp, link, temp_location)
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 433, in _download_url
2014-08-11 09:56:25.929 |     for chunk in resp_read(4096):
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 421, in resp_read
2014-08-11 09:56:25.929 |     chunk_size, decode_content=False):
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 240, in stream
2014-08-11 09:56:25.929 |     data = self.read(amt=amt, decode_content=decode_content)
2014-08-11 09:56:25.929 |   File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 187, in read
2014-08-11 09:56:25.929 |     data = self._fp.read(amt)
2014-08-11 09:56:25.929 |   File "/usr/lib/python2.7/httplib.py", line 567, in read
2014-08-11 09:56:25.929 |     s = self.fp.read(amt)
2014-08-11 09:56:25.929 |   File "/usr/lib/python2.7/socket.py", line 380, in read
2014-08-11 09:56:25.929 |     data = self._sock.recv(left)
2014-08-11 09:56:25.929 | timeout: timed out

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-13: Fix proposed to elastic-recheck (master)

#16

Fix proposed to branch: master
Review: https://review.openstack.org/113774

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-27: Change abandoned on elastic-recheck (master)

#17

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/113774
Reason: Abandoning after 2 weeks with no response.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-04: Related fix proposed to elastic-recheck (master)

#18

Related fix proposed to branch: master
Review: https://review.openstack.org/119121

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-04: Related fix merged to elastic-recheck (master)

#19

Reviewed: https://review.openstack.org/119121
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=9bce596e76b87b303a2fc515cd59116ea9069dd4
Submitter: Jenkins
Branch: master

commit 9bce596e76b87b303a2fc515cd59116ea9069dd4
Author: Matt Riedemann <email address hidden>
Date: Wed Sep 3 17:32:25 2014 -0700

Update query for pip timeout bug 1270710

    I noticed the uncategorized fail rates for the
    gate-nova-python27 job were low and one of the
    reasons is due to the pip timeout bug and the
    existing query not hitting on new traces.

    I really hate that we're updating this and
    fingerprinting on line numbers in the packaging
    code but I'm not aware of anything better at
    the moment, so I guess this will just keep
    being a moving target.

In 24 hours this does bring the hits up from
250 to 298.

Change-Id: I90ce79f4b268c036627b30fab03f9680eaa32304
Related-Bug: #1270710

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-22: Related fix proposed to elastic-recheck (master)

#20

Related fix proposed to branch: master
Review: https://review.openstack.org/123147

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-22: Change abandoned on elastic-recheck (master)

#21

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/123147

Revision history for this message

Launchpad Janitor (janitor) wrote on 2014-11-22:

#22

[Expired for OpenStack Core Infrastructure because there has been no activity for 60 days.]

Changed in openstack-ci:
status:	Incomplete → Expired

Jeremy Stanley (fungi) on 2014-12-04

Changed in openstack-gate:
status:	New → Incomplete
importance:	Undecided → Medium

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Core Infrastructure	Expired	Medium	Unassigned
	OpenStack-Gate	Incomplete	Medium	Unassigned

OpenStack-Gate

sporadic pip timeouts during download

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches