bzr suggests wrong URL for break-lock for smart-server branches

I'm sure this was originally filed against Launchpad-Bazaar, not Bazaar, not sure how it got changed. It does seem that Bazaar gets break-lock urls wrong for the hpss all on its own though...

I have same issue in version
gnomefreak@Development:~$ policy bzr
bzr:
  Installed: 1.6.1-1
  Candidate: 1.6.1-1
  Version table:
 *** 1.6.1-1 0
        500 http://gb.archive.ubuntu.com intrepid/main Packages
        100 /var/lib/dpkg/status

Me too! This is very annoying and still in bzr version 1.9.

$ bzr push lp:~ted-gould/inkscape/trunk
^Cbzr: interrupted
$ bzr push lp:~ted-gould/inkscape/trunk
Enter passphrase for key '/home/ted/.ssh/id_launchpad':
Unable to obtain lock lp-44825360:///~ted-gould/inkscape/trunk/.bzr/branch/lock
held by <email address hidden> on host crowberry [process #21264]
locked 8 minutes, 8 seconds ago
Will continue to try until 18:42:57, unless you press Ctrl-C
If you're sure that it's not being modified, use bzr break-lock lp-44825360:///~ted-gould/inkscape/trunk/.bzr/branch/lock
bzr: ERROR: Could not acquire lock "(remote lock)"
$ bzr break-lock lp-44825360:///~ted-gould/inkscape/trunk/.bzr/branch/lock
bzr: ERROR: Unsupported protocol for url "lp-44825360:///~ted-gould/inkscape/trunk/.bzr/branch/lock"
$

bumping importance due to more people hitting it, and not providing a solution :)

I had the same problem and managed to find a workaround :
Instead of typing : bzr break-lock lp-45673808:///~bertrand-goareguer/ufw/ufw-qos/.bzr/branch/lock
I typed : bzr break-lock bzr+ssh://<email address hidden>/~bertrand-goareguer/ufw/ufw-qos/.bzr/branch/lock
Then I simply answered 'yes' to the question asked by bzr.

Same problem here, using bzr 1.12:

maqroll[IPython]> bzr push
Using saved push location: lp:~fdo.perez/ipython/trunk-dev
Unable to obtain lock lp-45955728:///~fdo.perez/ipython/trunk-dev/.bzr/branch/lock
held by <email address hidden> on host crowberry [process #25930]
locked 23 seconds ago
Will continue to try until 03:41:08, unless you press Ctrl-C
If you're sure that it's not being modified, use bzr break-lock lp-45955728:///~fdo.perez/ipython/trunk-dev/.bzr/branch/lock
bzr: ERROR: Could not acquire lock "(remote lock)"
maqroll[IPython]> bzr break-lock lp-45955728:///~fdo.perez/ipython/trunk-dev/.bzr/branch/lock
bzr: ERROR: Unsupported protocol for url "lp-45955728:///~fdo.perez/ipython/trunk-dev/.bzr/branch/lock"

I really hope a fix comes at some point, because the situation feels a bit crazy:

bzr: There is a problem, type 'foo' to fix it
you: foo
bzr: foo? Why are you typing foo? I don't know what foo is. There is a problem, type 'foo' to fix it.

:)

I tried to dig a bit where this happens but got stuck.

It doesn't happen (clientside) in bzrlib.lockdir.LockDir.wait_lock() (though I'd guess that is where it gets raised on the lp side of things).

Somewhere deep down some file-like object gets written to with the following bytes:
     "d16:Software version7:1.15deves6l17:Branch.lock_write25:~larstiq/+junk/bug250451/0:0:ee"

And then the message about unable to obtain the lock appears on stdout.

Backtrace, some linenumbers changed due to `import pdb` etc:
  /home/larstiq/src/bzr/bzr.dev/bzrlib/commands.py(559)run_argv_aliases()
-> return self.run(**all_cmd_args)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/builtins.py(1066)run()
-> use_existing_dir=use_existing_dir)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/push.py(128)_show_push_branch()
-> remember)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/bzrdir.py(1275)push_branch()
-> overwrite, stop_revision=revision_id)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/decorators.py(138)read_locked()
-> result = unbound(self, *args, **kwargs)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/branch.py(2240)push()
-> _override_hook_source_branch=_override_hook_source_branch)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/branch.py(2968)_run_with_write_locked_tar
-> target.lock_write()
  /home/larstiq/src/bzr/bzr.dev/bzrlib/remote.py(2125)lock_write()
-> remote_tokens = self._remote_lock_write(token)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/remote.py(2116)_remote_lock_write()
-> repo_token or '', **err_context)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/remote.py(55)_call()
-> return self._client.call(method, *args)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/client.py(129)call()
-> result, protocol = self.call_expecting_body(method, *args)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/client.py(142)call_expecting_body()
-> method, args, expect_response_body=True)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/client.py(76)_call_and_read_respons
-> readv_body=readv_body, body_stream=body_stream)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/client.py(60)_send_request()
-> encoder.call(method, *args)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/protocol.py(1239)call()
-> self._write_end()
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/protocol.py(1106)_write_end()
-> self.flush()
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/protocol.py(1073)flush()
-> self._real_write_func(''.join(self._buf))
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/medium.py(393)accept_bytes()
-> self._accept_bytes(bytes)
  /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/medium.py(921)_accept_bytes()
-> self._medium._accept_bytes(bytes)
> /home/larstiq/src/bzr/bzr.dev/bzrlib/smart/medium.py(765)_accept_bytes()
-> self._write_to.write(bytes)

> Somewhere deep down some file-like object gets written to with the following bytes:
> "d16:Software version7:1.15deves6l17:Branch.lock_write25:~larstiq/+junk/bug250451/0:0:ee"

Ignoring headers and whatnot, this is just a simple HPSS call to Branch.lock_write:

>>> bdecode("l17:Branch.lock_write25:~larstiq/+junk/bug250451/0:0:e")
['Branch.lock_write', '~larstiq/+junk/bug250451/', '', '']

The -Dhpss logging would be an easier way to observe the HPSS traffic. As you notice, the useless URL is not coming from the client. The two most likely fixes are:

 a) make the client ignore whatever URL the server returns when generating the break-lock message, or
 b) make the server send usable URL (or maybe relpath) in its LockContention errors (and/or other lock-related errors).

I'd probably be inclined to go with the first option, although for perfect correctness I think perhaps we should do both. In general all paths returned by the server probably need to go through some sort of explicit translation back to client-usable relpaths, but that's out-of-scope for this particular bug report.

On Sun, May 03, 2009 at 02:16:26AM -0000, Andrew Bennetts wrote:
> > Somewhere deep down some file-like object gets written to with the following bytes:
> > "d16:Software version7:1.15deves6l17:Branch.lock_write25:~larstiq/+junk/bug250451/0:0:ee"
>
> Ignoring headers and whatnot, this is just a simple HPSS call to
> Branch.lock_write:
>
> >>> bdecode("l17:Branch.lock_write25:~larstiq/+junk/bug250451/0:0:e")
> ['Branch.lock_write', '~larstiq/+junk/bug250451/', '', '']

Right.

> The -Dhpss logging would be an easier way to observe the HPSS traffic.
> As you notice, the useless URL is not coming from the client. The two
> most likely fixes are:
>
> a) make the client ignore whatever URL the server returns when generating the break-lock message, or
> b) make the server send usable URL (or maybe relpath) in its LockContention errors (and/or other lock-related errors).

I tried to do a), but I fail to see how to do that. At the point where
the message gets dumped to screen, it happens magically to me, without a
point where I can intervene.

I can not find the point in the code to apply a)

Wouter van Heyst

Wouter van Heyst wrote:
> On Sun, May 03, 2009 at 02:16:26AM -0000, Andrew Bennetts wrote:
[...]
> > a) make the client ignore whatever URL the server returns when generating the break-lock message, or
[...]
>
> I tried to do a), but I fail to see how to do that. At the point where
> the message gets dumped to screen, it happens magically to me, without a
> point where I can intervene.
>
> I can not find the point in the code to apply a)

In bzrlib/remote.py, RemoteBranch._remote_lock_write makes the RPC call, and
the global _translate_error function in that module converts error responses
from the wire into regular bzrlib exceptions, e.g. LockFailed and
LockContention. It's the LockContention translation that produces this part
of the output: “Could not acquire lock "(remote lock)"”

The message in the bug report seems to be printed by the wait_lock method in
bzrlib/lockdir.py though. I'm not sure how that's getting invoked here,
maybe stick a "import pdb; pdb.set_trace()" or "import traceback;
traceback.print_stack()" or similar in there just before it calls
self._report_function? Oh, _report_function is bzrlib.trace.note: that text
is actually being emitted *server-side*, and SSH is faithfully conveying it
to the local terminal. If you run a local TCP bzr server (with “bzr serve
--allow-writes”) you'll see that the server process does the printing!

So actually a) has a few parts:

 a.1) make the client ignore the URL from the server (except perhaps if it is
      a relpath).
 a.2) fix bzrlib.lockdir to not spew over stdout. It's mixed UI code with
      low-level logic.
 a.3) make the text that bzrlib.lockdir get spewed in a more appropriate
      place so that the client can do it (and substitute in its own URLs).

Brief reproduction notes:

  $ cd /tmp
  $ bzr init locked-branch
  $ python -c "from bzrlib.bzrdir import BzrDir; BzrDir.open('/tmp/locked-branch').open_branch().lock_write()"
  $ bzr --no-plugins serve --allow-writes

  [now launch a separate terminal]

  $ bzr push -d [path-of-any-branch] bzr://localhost/locked-branch

-Andrew.

On Mon, May 04, 2009 at 12:43:53AM -0000, Andrew Bennetts wrote:

...

> The message in the bug report seems to be printed by the wait_lock method in
> bzrlib/lockdir.py though. I'm not sure how that's getting invoked here,
> maybe stick a "import pdb; pdb.set_trace()" or "import traceback;
> traceback.print_stack()" or similar in there just before it calls
> self._report_function? Oh, _report_function is bzrlib.trace.note: that text
> is actually being emitted *server-side*, and SSH is faithfully conveying it
> to the local terminal. If you run a local TCP bzr server (with “bzr serve
> --allow-writes”) you'll see that the server process does the printing!

Aha, it is not bzr but ssh that is doing the printing locally, now my
difficulty finding the code where it happened makes sense :)

> So actually a) has a few parts:
>
> a.1) make the client ignore the URL from the server (except perhaps if it is
> a relpath).
> a.2) fix bzrlib.lockdir to not spew over stdout. It's mixed UI code with
> low-level logic.
> a.3) make the text that bzrlib.lockdir get spewed in a more appropriate
> place so that the client can do it (and substitute in its own URLs).

Looks good. Now just need someone to implement your analysis. I won't
make the mistake of volunteering for that again.

> Brief reproduction notes:
>
> $ cd /tmp
> $ bzr init locked-branch
> $ python -c "from bzrlib.bzrdir import BzrDir; BzrDir.open('/tmp/locked-branch').open_branch().lock_write()"
> $ bzr --no-plugins serve --allow-writes
>
> [now launch a separate terminal]
>
> $ bzr push -d [path-of-any-branch] bzr://localhost/locked-branch

Very clear illustration of the lock message being printed on stdout,
thank you.

Wouter van Heyst

wing-commander libnih% bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~scott/libnih/trunk/
Unable to obtain lock lp-140021887518416:///~scott/libnih/trunk/.bzr/branch/lock
held by <email address hidden> on host crowberry [process #9552]
locked 3 minutes, 22 seconds ago
Will continue to try until 21:17:52, unless you press Ctrl-C
If you're sure that it's not being modified, use bzr break-lock lp-140021887518416:///~scott/libnih/trunk/.bzr/branch/lock
bzr: ERROR: Could not acquire lock "(remote lock)":
zsh: exit 3 bzr push

wing-commander libnih% bzr break-lock lp-139898621036240:///~scott/libnih/trunk/.bzr/branch/lock
bzr: ERROR: Unsupported protocol for url "lp-139898621036240:///~scott/libnih/trunk/.bzr/branch/lock"
zsh: exit 3 bzr break-lock lp-139898621036240:///~scott/libnih/trunk/.bzr/branch/lock

I'm re-filing it against launchpad-bazaar, because it shouldn't be sending the id.

The ID is just a Python id that's part of the underlying transport URL. If we were publishing directly from disk, it would be a file:/// url.

Another example

$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~maria-captains/maria/5.1.39-oqgraph/
Unable to obtain lock lp-46129552:///~maria-captains/maria/5.1.39-oqgraph/.bzr/branch/lock
held by <email address hidden> on host crowberry [process #5258]
locked 4 minutes, 30 seconds ago
Will continue to try until 01:25:56, unless you press Ctrl-C
If you're sure that it's not being modified, use bzr break-lock lp-46129552:///~maria-captains/maria/5.1.39-oqgraph/.bzr/branch/lock
bzr: ERROR: Could not acquire lock "(remote lock)":

(trying with the suggested URL)
$ bzr break-lock lp-46129552:///~maria-captains/maria/5.1.39-oqgraph/.bzr/branch/lock
bzr: ERROR: Unsupported protocol for url "lp-46129552:///~maria-captains/maria/5.1.39-oqgraph/.bzr/branch/lock"

(this works, using the url from bzr info)
$ bzr break-lock bzr+ssh://bazaar.launchpad.net/~maria-captains/maria/5.1.39-oqgraph/
Break lock bzr+ssh://bazaar.launchpad.net/~maria-captains/maria/5.1.39-oqgraph/.bzr/branch/lock
held by <email address hidden> on host crowberry [process #5258]
locked 9 minutes, 57 seconds ago? [y/n]: y

In a nutshell, the url shown is wrong/invalid/unusable.
Solution: change it ;-)

I'm not sure why I marked this as invalid. The bug does affect launchpad-code, since we'd have to do the reverse URL translation there. OTOH, bzrlib gives us no good way of doing this.

I got my repo locked via the same conditions here:
https://bugs.launchpad.net/bzr/+bug/495011
OSX 10.6, Bazaar 2.1.0, ctrl+c durring a commit, locked me up

I tried all three of the following:
bzr break-lock sftp://<email address hidden>/~exp-2-dev/exponent-cms/mvc-trunk/
bzr break-lock bzr+ssh://bazaar.launchpad.net/~exp-2-dev/exponent-cms/mvc-trunk/
bzr break-lock lp:exponent-cms

All run, but never prompt for a y/n to confirm the break lock. Attempts to commit after still result in:
Unable to obtain lock file:///Users/phillip/projects/_trunk/exp/.bzr/branch/lock
held by <my email> on host phillip.local [process #89248]

Any further thoughts on this?

yes, your lock is on your local disk : bzr break-lock
file:///Users/phillip/projects/_trunk/exp

A simple workaround is to just do the following:

$ bzr break-lock '.bzr/branch/lock'

Bazaar will then ask you if you want to break the lock, to which you answer yes.