"ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem" when pushing a branch

Happened to me again in: https://code.edge.launchpad.net/~beuno/launchpad/ui_bugs_n_branches_revenge

bzr: ERROR: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

Traceback from .bzr.log:

9.270 fetch up to rev {<email address hidden>}
46.975 Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 849, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 795, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 495, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 913, in run
    use_existing_dir=use_existing_dir)
  File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162, in _show_push_branch
    stop_revision=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1749, in push
    _override_hook_source_branch=_override_hook_source_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2476, in _run_with_write_locked_target
    target.unlock()
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1520, in unlock
    self._unlock(branch_token, repo_token)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1488, in _unlock
    self._translate_error(err, token=str((branch_token, repo_token)))
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1361, in _translate_error
    self.repository._translate_error(err, branch=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 359, in _translate_error
    self.bzrdir._translate_error(err, repository=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 90, in _translate_error
    _translate_error(err, bzrdir=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1796, in _translate_error
    raise errors.UnknownErrorFromSmartServer(err)
UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

46.976 return code 3

On Sat, 08 Nov 2008 00:25:01 Martin Albisetti wrote:
> Happened to me again in:
> https://code.edge.launchpad.net/~beuno/launchpad/ui_bugs_n_branches_r
>evenge
>
> bzr: ERROR: Server sent an unexpected error: ('error',
> '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')
>
> Traceback from .bzr.log:
>
> 9.270 fetch up to rev
> {<email address hidden>}
> 46.975 Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 849, in run_bzr_catch_errors return run_bzr(argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 795, in run_bzr ret = run(*run_argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 495, in run_argv_aliases return self.run(**all_cmd_args)
> File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line
> 913, in run use_existing_dir=use_existing_dir)
> File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162,
> in _show_push_branch stop_revision=revision_id)
> File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line
> 138, in read_locked result = unbound(self, *args, **kwargs)
> File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line
> 1749, in push
> _override_hook_source_branch=_override_hook_source_branch) File
> "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2476, in
> _run_with_write_locked_target target.unlock()
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1520, in unlock self._unlock(branch_token, repo_token)
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1488, in _unlock self._translate_error(err, token=str((branch_token,
> repo_token))) File
> "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1361, in
> _translate_error self.repository._translate_error(err, branch=self,
> **context) File "/usr/lib/python2.5/site-packages/bzrlib/remote.py",
> line 359, in _translate_error self.bzrdir._translate_error(err,
> repository=self, **context) File
> "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 90, in
> _translate_error _translate_error(err, bzrdir=self, **context)
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1796, in _translate_error raise
> errors.UnknownErrorFromSmartServer(err)
> UnknownErrorFromSmartServer: Server sent an unexpected error:
> ('error', '<ProtocolError for
> xmlrpc.lp.internal:8097/branchfilesystem: -1 >')
>
> 46.976 return code 3
>
>
> ** Changed in: launchpad-bazaar
> Importance: Undecided => High
> Status: New => Triaged

Can you pass this information on to Aaron? Preferably also giving him
access to the branch. Please don't delete the branch yet.

Jono, can you take a look at this?

I expect it's the timeout due to the scanner processing the same branch thing.

bzr 2.4 no longer uses xmlrpc before accessing the branch over ssh, so this bug should become less important.

This probably should be critical as it would OOPS today; it also has nothing to do with bzr's xmlrpc usage - note the *internal* name: this is the codehosting use of the internal xmlrpc service (part of our existing SOA infrastructure).

oh of course

Escalating to critical on the grounds it's a user-affecting external failure.

This is also impacting the udd package importer in for example http://package-import.ubuntu.com/status/language-pack-gnome-ca.html#2011-07-18 07:27:28.730318

Though it's a slightly different error:

bzrlib.errors.UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/codehosting: 503 Service Unavailable>')

 * i don't know if this is recording an oops
 * if there is an oops, the bzr server is failing to pass it through
 * the ssh server could possibly retry a couple of times (I don't know if it would help)

You can grep on carob to find an oops; if there isn't one that should also be a bug (critical too) as thats crucial signal for us.

I think the ssh server probably wants to fail fast here and let the client try again (or not). However, the 503 may be due to asking a server that is being deployed to at the time, in which case, until we have the graceful-deploy tickets completed, this will be a rare, but expected situation.

One possible oops for this is OOPS-2022I35, where it's timing out after 9s updating the branch last-mirror time. It doesn't seem like it ought to be slow so perhaps the database was heavily loaded or locked at the time it happened...

I don't at the moment see any failures from the ssh front end but I may just have not looked.

I got possibly this error using "Bazaar (bzr) 2.5.1" pushing existing source to a new branch. It then left the target directory in a broken state. Trying the same push again to a different title then suceedeed (so the error appears to be ephemeral).

edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: Server sent an unexpected error: ('error', 'xmlrpclib.ProtocolError', '<ProtocolError for xmlrpc.lp.internal:8097/codehosting: 502 Bad Gateway>')
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: At lp:~eda-qa/leaflang/primordial_type you have a valid .bzr control directory, but not a branch or repository. This is an unsupported configuration. Please move the target directory out of the way and try again.
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_types
Using default stacking branch /+branch-id/725500 at chroot-87084816:///~eda-qa/leaflang/
Created new stacked branch referring to /+branch-id/725500.