[sru] exceptions.AssertionError: second push failed to complete a fetch set

Just got the same error while pulling from lp:~verterok/lalita/hal

Same here:

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1243, in run
    source_branch=br_from)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1183, in sprout
    result_repo.fetch(source_repository, revision_id=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1695, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 3413, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 81, in __init__
    self.__fetch()
  File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 107, in __fetch
    self._fetch_everything_for_search(search)
  File "/usr/lib/python2.5/site-packages/bzrlib/fetch.py", line 148, in _fetch_everything_for_search
    missing_keys,))
AssertionError: second push failed to complete a fetch set([('texts', '64bit.m4-20090614010449-cn3n492v3lgx3k5m-1', '<email address hidden>'), ('texts', 'pandora_canonical.m4-20090703184306-arh3f06b44z1rzhy-3', '<email address hidden>'), ('texts', 'pandora_platform.m4-20090726211404-wlzq1f8jwxrgn0u4-1', '<email address hidden>'), ('texts', 'changelog-20090327194452-mh732hbogx91c4q1-1', '<email address hidden>'), ('texts', 'drizzle_field.c-20081111211028-6fvm63287dma43co-59', '<email address hidden>'), ('texts', 'drizzle_column_st.c-20090311051952-ryuzr4cmw7pdgbhl-1', '<email address hidden>'), ('texts', 'pandora_warnings.m4-20090703184306-arh3f06b44z1rzhy-7', '<email address hidden>'), ('inventories', '<email address hidden>'), ('texts', 'pandora_canonical.m4-20090703184306-arh3f06b44z1rzhy-3', '<email address hidden>'), ('texts', 'drizzle_con_st.c-20090311051952-ryuzr4cmw7pdgbhl-2', '<email address hidden>'), ('texts', 'configure.ac-20081111211028-6fvm63287dma43co-16', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('texts', 'authors-20081111211028-6fvm63287dma43co-2', '<email address hidden>')]).

bzr 2.0.0 on python 2.5.4 (Linux-2.6.26-2-amd64-x86_64-with-debian-squeeze-sid)
arguments: ['/usr/bin/bzr', 'branch', 'lp:~eday/libdrizzle/eday-dev']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UT...

Also, fwiw, I was able to work around this by:

bzr branch lp:libdrizzle
bzr branch libdrizzle eday-dev
cd eday-dev
bzr pull --remember lp:~eday/libdrizzle/eday-dev

I cannot reproduce with any of the branches mentioned so far, which suggests it's either:
 a) something transient (e.g. a particular revision graph shape that later pushes changes, or a race with a pack-names update?), or
 b) something to about the local repositories?

I'm not sure how to proceed, as I don't have any good theories.

Did anyone experience this without a local shared repository (or with a pristine, freshly init-repo'd one)? That would suggest it is, or was, an issue with the data or logic on the server...

If it's a race with a concurrent write to a repo then perhaps there would be something in the server-side log? I'm not sure how feasible it is to track down the relevant bzr serve logs for Launchpad.

(Also, we probably should at least fix that "second push" error message to be clearer, as that code is used for fetching in both directions, not just pushing. All the reports in this bug are in fact for "bzr branch" or "bzr pull", not "bzr push".)

Hmm. That error is raised by the target repository (so, the local one in these cases), which essentially checks:

 a) that all compression bases for the new data are present, and
 b) if the target is stacked, that the parent inventories required for stacking correctly are present.

It's that first condition that appears to be triggering, as the missing records are (mostly) texts, rather than all inventories. But it did make me wonder if stacking is involved on the client... perhaps mixing stacked & unstacked branches in one repository might produce this...

Bug reporters: were/are you using stacked branches locally?

I haven't managed to trigger this bug yet, even mixing stacked/unstacked branches locally. I'd love to find a way to reproduce this.

My branches were created with bzr branch lp...; I don't know whether that will create a local stacked branch, I'd expect not.

Loïc, no, that would not create a stacked branch. (Unless you have some extraordinary local configuration, but it seems safe to assume not.)

Ok, so suspecting local stacked branches is probably a red herring. Drat.

If it's reproducible for one of you, can you give us a tarball (privately if you wish) of the destination repository where fetching fails?

So the test case which led me to file this bug (bzr branch lp:~ubuntu-core-dev/upstart/ubuntu) is not failing for me anymore.

Putting this to incomplete until we have enough data to proceed.

I get this after trying to branch a stacked branch into an empty repository. On the remote side:

repo/ <- shared repo (1.6, no trees)
repo/trunk <- branch
repo/user <- shared repo (1.6 no trees)
repo/user/feature-branch <- branch using repo/user stacked on repo/trunk

On my side:

bzr init-repo --no-trees --1.6 tmp
cd tmp
bzr branch bzr+ssh://remote/repo/user/feature-branch

=> ERROR: exceptions.AssertionError: second push failed to complete a fetch

Now, if I do the following I can get my branch:

rm -rf feature-branch
bzr branch bzr+ssh://remote/repo/trunk
bzr branch bzr+ssh://remote/repo/user/feature-branch

Tested with bzr-2.0.0 on both sides.

mwhudson has found a branch (lp:~ubuntu-core-doc/gnome-user-docs/ubuntu-karmic) that reproduces this, but only LP's hosted copy, not the read-only mirrored copy that most users see. He's put a snapshot of the hosted copy at lp:~spiv/+junk/u-k for me. Again, it's likely that Launchpad's mirroring may interfere for most people, but it's certainly reproducible by me now.

The recipe to reproduce is simply "bzr get lp:~spiv/+junk/u-k" — no shared repos or anything else involved on the client! The source branch is stacked on lp:~vcs-imports/gnome-user-docs/master.

This looks like a bug in the client. In the case of the gnome-user-docs branch the missing keys are claimed to be [('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>'), ('inventories', '<email address hidden>')], but when I instrument the client every single one of these keys is present in the first stream (from the stacked branch, before it fetches the rest from the stacked-on branch).

The bug is in KnitVersionedFiles.insert_record_stream, or at least its interaction with add_records(..., missing_compression_parents=True). It doesn't pass all the buffered records in one batch, but in several (one per key being depended on). This in turn breaks an assumption in add_records, so that depending on ordering of keys it's possible for it to think a key has a missing parent, when in fact it just has a buffered parent (because that parent, or one of that parents' parents, etc., has a missing parent). Probably this bug is so finicky in part because it depends on the order a Python dict is iterated, which in turn is perhaps partly dependent on the order records are received from the source.

The simple fix is to accumulate all buffered_index_entries into one list before calling add_records at the end of insert_record_stream.

This wouldn't affect 2a repositories, so that's another reason for people to upgrade...

I think this bug has been lurking for a long time, although it requires stacking and HPSS to provoke, as well as some bad luck with how Python orders a dict. Hmm, perhaps before this was masked because we weren't using 'unordered' fetches as often?

This bug has now been fixed in the 2.0.2 release of bzr, which is now in <http://packages.debian.org/search?keywords=bzr> Debian squeeze/testing and sid. It would be nice if this was synced into Lucid.

We would also like to address this, and a set of other bugs, by an SRU of bzr 2.0.2 into Karmic, as an upgrade from 2.0.0.

The lists of bugs fixed are in <https://launchpad.net/bzr/2.0/2.0.1> and <https://launchpad.net/bzr/2.0/2.0.2>. There are several high-impact bugs that can interrupt operation. Our 2.0.0 series has been designed to have minimal conservative bug fixes, and not to change plugin APIs, disk or network formats, or UI, and generally to be something that would be compatible with Ubuntu's update policy.

The diff is attached; it is fairly large because 20 bugs have been fixed, but readable. More than half the changes are to tests.

2.0.2 has been available in our PPA and as a source package for a bit over three weeks with no critical bugs or regressions reported.

Hi,

Here is a package targeted at karmic-updates that fixes this bug by bringing in the new upstream release of 2.0.2. Could somebody from ~ubuntu-devel please check it and either sponsor its upload for me, or advise me of what to do next?

The full changes in this upstream release are as follows:

bzr 2.0.2
#########

:Codename: after the scare
:2.0.2: 2009-11-02

The second in our "let's keep the stable bugfixes flowing" series. As
expected this has a few (~9) bugfixes relative to 2.0.1, and no major api
changes or features.

Bug Fixes
*********

* Avoid "NoneType has no attribute st_mode" error when files disappear
  from a directory while it's being read. (Martin Pool, #446033)

* Content filters are now applied correctly after revert.
  (Ian Clatworthy)

* Diff parsing handles "Binary files differ" hunks. (Aaron Bentley, #436325)

* Fetching from stacked pre-2a repository via a smart server no longer
  fails intermittently with "second push failed to complete".
  (Andrew Bennetts, #437626)

* Fix typos left after test_selftest refactoring.
  (Vincent Ladeuil, Matt Nordhoff, #461149)

* Fixed ``ObjectNotLocked`` errors during ``bzr log -r NNN somefile``.
  (Andrew Bennetts, #445171)

* PreviewTree file names are not limited by the encoding of the temp
  directory's filesystem. (Aaron Bentley, #436794)

Improvements
************

* ``bzr log`` now read-locks branches exactly once, so makes better use of
  data caches. (Andrew Bennetts)

Documentation
*************

* Filtered views user documentation upgraded to refer to format 2a
  instead of pre-2.0 formats. (Ian Clatworthy)

bzr 2.0.1
#########

:Codename: Stability First
:2.0.1: 2009-10-14

The first of our new ongoing bugfix-only stable releases has arrived. It
includes a collection of 12 bugfixes applied to bzr 2.0.0, but does not
include any of the feature development in the 2.1.0 series.

Bug Fixes
*********

* ``bzr add`` in a tree that has files with ``\r`` or ``\n`` in the
  filename will issue a warning and skip over those files.
  (Robert Collins, #3918)

* bzr will attempt to authenticate with SSH servers that support
  ``keyboard-interactive`` auth but not ``password`` auth when using
  Paramiko. (Andrew Bennetts, #433846)

* Fixed fetches from a stacked branch on a smart server that were failing
  with some combinations of remote and local formats. This was causing
  "unknown object type identifier 60" errors. (Andrew Bennetts, #427736)

* Fixed ``ObjectNotLocked`` errors when doing some log and diff operations
  on branches via a smart server. (Andrew Bennetts, #389413)

* Handle things like ``bzr add foo`` and ``bzr rm foo`` when the tree is
  at the root of a drive. ``osutils._cicp_canonical_relpath`` always
  assumed that ``abspath()`` returned a path that did not have a trailing
  ``/``, but that is not true when working at the root of the filesystem.
  (John Arbash Meinel, Jason Spashett, #322807)

* Hide deprecation warnings for 'final' releases for python2.6.
  (John Arbash Meinel, #440062)

* Improve the time for ``bzr log DIR`` for 2a format repositories.
  We had been using the same code path as for <2a formats, which required
  iterating over all o...

The proposed changelog entry is this:

bzr (2.0.2-1ubuntu1) karmic-proposed; urgency=low

  * Proposed SRU, taking all changes from Bazaar's 2.0.2 new bugfix-only
    upstream release, including LP: #437626.

 -- Martin Pool <email address hidden> Mon, 14 Dec 2009 16:34:45 +1100

Please let me know if this should be more verbose.

Lucid has 2.0.2. SRU looks good, but changelog needs to mention all other affected bugs as well, preferably with a short description.

Here's an updated diff with a better description of all changes. I have not yet checked all the bug descriptions (and probably won't tonight.)

@pitti Thanks for your comments, could you please have a look again and give me some more feedback or sponsor this into the SRU?

Martin, you just took the new Debian package, which has packaging changes (a new bzr-doc package). This is not appropriate for a stable update.

I'll clean this up, and upload. Thanks for preparing the detailled changelog!

Accepted bzr into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

2009/12/22 Martin Pitt <email address hidden>:
> Martin, you just took the new Debian package, which has packaging
> changes (a new bzr-doc package). This is not appropriate for a stable
> update.

Oops, sorry.

> I'll clean this up, and upload. Thanks for preparing the detailled
> changelog!

Thankyou.

> Accepted bzr into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

I've tested it, and it seems fine. I've asked our community for more
testing and to post here.

--
Martin <http://launchpad.net/~mbp/>

I just installed 2.0.2-0ubuntu1 from karmic-proposed, and it appears to be working just fine.

This bug was fixed in the package bzr - 2.0.2-0ubuntu1

---------------
bzr (2.0.2-0ubuntu1) karmic-proposed; urgency=low

  * Proposed SRU, taking all changes from Bazaar's 2.0.2 new bugfix-only
    upstream release.
  * Avoid "NoneType has no attribute st_mode" error when files disappear
    from a directory while it's being read. LP: #446033
  * Content filters are now applied correctly after revert.
  * Diff parsing handles "Binary files differ" hunks. LP: #436325
  * Fetching from stacked pre-2a repository via a smart server no longer
    fails intermittently with "second push failed to complete".
    LP: #437626
  * Fix typos left after test_selftest refactoring.
    LP: #461149
  * Fixed ``ObjectNotLocked`` errors during ``bzr log -r NNN somefile``.
    LP: #445171
  * PreviewTree file names are not limited by the encoding of the temp
    directory's filesystem. LP: #436794
  * ``bzr log`` now read-locks branches exactly once, so makes better use of
    data caches.
  * Filtered views user documentation upgraded to refer to format 2a
    instead of pre-2.0 formats.
  * ``bzr add`` in a tree that has files with ``\r`` or ``\n`` in the
    filename will issue a warning and skip over those files.
    LP: #3918
  * bzr will attempt to authenticate with SSH servers that support
    ``keyboard-interactive`` auth but not ``password`` auth when using
    Paramiko. LP: #433846
  * Fixed fetches from a stacked branch on a smart server that were failing
    with some combinations of remote and local formats. This was causing
    "unknown object type identifier 60" errors. LP: #427736
  * Fixed ``ObjectNotLocked`` errors when doing some log and diff operations
    on branches via a smart server. LP: #389413
  * Handle things like ``bzr add foo`` and ``bzr rm foo`` when the tree is
    at the root of a drive. ``osutils._cicp_canonical_relpath`` always
    assumed that ``abspath()`` returned a path that did not have a trailing
    ``/``, but that is not true when working at the root of the filesystem.
    LP: #322807
  * Hide deprecation warnings for 'final' releases for python2.6.
    LP: #440062
  * Improve the time for ``bzr log DIR`` for 2a format repositories.
    We had been using the same code path as for <2a formats, which required
    iterating over all objects in all revisions.
    LP: #374730
  * Make sure that we unlock the tree if we fail to create a TreeTransform
    object when doing a merge, and there is limbo, or pending-deletions
    directory. LP: #427773
  * Occasional IndexError on renamed files have been fixed. Operations that
    set a full inventory in the working tree will now go via the
    apply_inventory_delta code path which is simpler and easier to
    understand than dirstates set_state_from_inventory method. This may
    have a small performance impact on operations built on _write_inventory,
    but such operations are already doing full tree scans, so no radical
    performance change should be observed. LP: #403322
  * Retrieving file text or mtime from a _PreviewTree has good performance when
    there are many changes.
  * The CHK index pages now use an unlimited cache size. With a limited
    cache and a ...