TooManyConcurrentRequests error when ssh connection fails (bzr crashes when pulling)

I'm getting this as well when using 5-a-day on Ubuntu Intrepid. It's reproducible.

bzr/intrepid uptodate 1.5-1

Trying to register for GlobalBugJam same thing.

Applet fails too.

jimf@jimf-ibook:/usr/bin$ sudo 5-a-day --update
[sudo] password for jimf:
Permission denied (publickey).
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x10468870>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 633, in run
    branch_from = Branch.open_from_transport(location_transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 115, in open_from_transport
    control = bzrdir.BzrDir.open_from_transport(transport, _unsupported)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 688, in open_from_transport
    redirected)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 1664, in do_catching_redirections
    return action(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 665, in find_format
    transport, _server_formats=_server_formats)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1422, in find_format
    return format.probe_transport(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 2406, in probe_transport
    request = client.get_request()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 417, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 592, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x10468870>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.3.1 on python 2.5.2.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'pull', 'bzr+ssh://<email address hidden>/~jedijf/5-a-day-data/main']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
bzr failed with error code 1024

Same here, for a 'merge' using a DNS name that could not be resolved, using the smart server:
$ bzr merge bzr://bzr/vp-3.1-hot-installer
The dns name 'bzr' was not known on my machine and it caused the same problem. Perhaps any connection error causes the above message?

bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartTCPClientMedium object at 0x8473d0c>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 857, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2901, in run
    location, revision, remember, possible_transports, pb)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2979, in _get_merger_from_branch
    possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 142, in open_containing
    possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 826, in open_containing
    return BzrDir.open_containing_from_transport(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 847, in open_containing_from_transport
    result = BzrDir.open_from_transport(a_transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 806, in open_from_transport
    return format.open(transport, _found=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1733, in open
    return self._open(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 2613, in _open
    return remote.RemoteBzrDir(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 76, in __init__
    response = self._client.call('BzrDir.open', path)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 111, in call
    result, protocol = self.call_expecting_body(method, *args)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 124, in call_expecting_body
    method, args, expect_response_body=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 69, in _call_and_read_response
    readv_body=readv_body)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 39, in _send_request
    protocol_version)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 94, in _construct_protocol
    request = self._medium.get_request()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 589, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 764, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartTCPClientMedium object at 0x8473d0c>' has reached its concurrent request...

Hi,

These errors are usually secondary, meaning that there was first
the real problem, and as bzr tried to clean up it triggered this problem.

Another user reported this problem with 5-a-day, and said that it was
due to either not having an SSH key registered for their launchpad account,
or having the wrong one, could you check whether that is the case for you?

Also, could you check that your username in ~/.5-a-day matches your
launchpad username?

Thanks,

James

I can reproduce this.

The problem is that the first thing bzr tries to do with a pull or merge is try to fetch the URL as file in case it is a bundle. If this fails due to being unable to connect (e.g. because SSH auth fails), the ConnectionReset error is intercepted by bzrlib.bundle.read_mergeable_from_transport's "except TransportError" and turned into a NotABundle error. Bzr then continues to try open the URL as a BzrDir, rather than immediately giving up and reporting the connection problem to the user.

There's two problems here, I think. First is that ConnectionReset isn't like the other errors that code wants to catch (e.g. the URL is a directory thus the 'get' fails); the connection is fundamentally broken and unusable, so continuing with the BzrDir opening is the wrong things to do here.

Secondly, if a stream-based smart medium has failed due to ConnectionReset, it should probably keep raising ConnectionReset if some code tries to keep using it, rather than getting into the state that produces the TooManyConcurrentRequests error. Ideally once a medium raises ConnectionReset it should not be used again, but if something does then reraisng ConnectionReset is more useful than TooManyConcurrentRequests.

Fixing the first part of this should be easy, so I'll do that for 1.8.

I've sent a patch to fix this to the mailing list for review.

Not sure of the current status of this one, but it also applies to commits to a bound branch when the parent bzr+ssh branch is inaccessible. I'm using Bzr 1.10 on Cygwin, so a bit old; apologies if my info is out of date. I just got this when committing to a bound branch when the parent's server happened to be temporarily offline (names/project references genericized):

Doug 2% bzr commit -m "Skeleton file set."
ssh: connect to host bzr-central.office.com port 22: Connection timed out
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x7fa1ceac>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "commands.py", line 893, in run_bzr_catch_errors
  File "commands.py", line 839, in run_bzr
  File "commands.py", line 539, in run_argv_aliases
  File "builtins.py", line 2435, in run
  File "decorators.py", line 192, in write_locked
  File "workingtree_4.py", line 235, in commit
  File "decorators.py", line 192, in write_locked
  File "mutabletree.py", line 202, in commit
  File "commit.py", line 303, in commit
  File "commit.py", line 464, in _check_bound_branch
  File "decorators.py", line 138, in read_locked
  File "branch.py", line 1984, in get_master_branch
  File "branch.py", line 123, in open
  File "bzrdir.py", line 766, in open
  File "bzrdir.py", line 808, in open_from_transport
  File "bzrdir.py", line 1770, in open
  File "bzrdir.py", line 2646, in _open
  File "remote.py", line 93, in __init__
  File "remote.py", line 51, in _call
  File "client.py", line 122, in call
  File "client.py", line 135, in call_expecting_body
  File "client.py", line 80, in _call_and_read_response
  File "client.py", line 42, in _send_request
  File "client.py", line 105, in _construct_protocol
  File "medium.py", line 672, in get_request
  File "medium.py", line 870, in __init__
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x7fa1ceac>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.10 on python 2.5.2 (cygwin)
arguments: ['/usr/bin/bzr', 'commit', '-m', 'Skeleton file set.']
encoding: 'US-ASCII', fsenc: 'US-ASCII', lang: None
plugins:
  bookmarks /home/Doug/.bazaar/plugins/bookmarks [unknown]
  bzrtools /home/Doug/.bazaar/plugins/bzrtools [1.10]
  cvsps_import /home/Doug/.bazaar/plugins/cvsps_import [unknown]
  email /home/Doug/.bazaar/plugins/email [unknown]
  fastimport /home/Doug/.bazaar/plugins/fastimport [unknown]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  service /home/Doug/.bazaar/plugins/service [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
Doug 2%