BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

I wouldn't be shocked to find that some other failures are caused or at least related, e.g. <https://bugs.launchpad.net/bzr/+bug/772935>

I added a bug task for LP due to OOPS-2021BZR207511 which looks like a similar error. Could someone confirm this is the same or if it requires a new bug?

@matsubara unfortunately for some reason there's no actual traceback but it looks similar.

https://bugs.launchpad.net/launchpad/+bug/812106 for the lack of a traceback in the oops.

The linked branch, lp:~spiv/bzr/stacked-fetch-same-chks, adds a failing test for this case (or at least a case *like* this). The challenge is figuring out what the fix should be.

This is a pretty fiddly problem, but I'll try to explain it as clearly and concisely as possible. When fetching multiple heads from a stacked repository via a smart server the assumptions we use to stream only O(changes) data rather O(tree) data aren't necessarily correct. The assumptions is basically “we can omit transferring records that are unchanged in the parent revisions of the set we are sending”, and the justification is that the parent will be present anyway: we assume another fetch in the chain of stacked fetches will include it (or that the target has already got that fetch anyway).

The problem is that justification isn't true: the other fetches in the stacked chain, following the same assumption, can exclude the same CHK records, so it is possible they are never sent. It is possible to construct fetches that always have ‘present parent inventories’ at every step in the stacking chain, by starting with a fetch with multiple heads. e.g. a revision graph like:
 * D has parents: C, A
 * C has parents: B
 * B has parents: A
 * A has parents: (none)
And then fetch [D,A] from the one repo (with C as present parent inventory), and then [C,B] from the next (and A is present, so is present parent inventory). If A and C share some records, both fetches will conclude they can omit those records. That then triggers BzrCheckError during commit_write_group (thank goodness for that integrity check!).

Examples of records that can be omitted by this erroneous logic:
  * the parent_id_basename_to_file_id map can easily be shared between multiple revisions ('bzr revert -r -2 && bzr commit'), which shares the CHK root key
  * an id_to_entry map with some LeafNodes in common (e.g. it's possible to have all revisions sharing a LeafNode with just the tree root, and have that leaf be the same in all revisions).
  * text to sends are determined by the id_to_entry records we send, so if any id_to_entry nodes are missing, then so are some texts.

The challenge is how we can fix this while still sending only O(changes) data (which is a key benefit of using stacking). What doesn't work:
  * simply re-requesting those inventories in the 'fix missing inventories' step, because that doesn't cause texts to be transferred (perhaps if we allowed a *third* fetch round to fixup those missing texts then maybe that would work? Ugh.)
  * discarding known-interesting CHK roots from the uninteresting-roots set before calling iter_interesting_nodes: that still allows shared non-root CHK nodes to be omitted.
  * dropping the filtering of “uninteresting” CHK nodes: that would regress to sending O(tree)-sized data.

Ideas welcome!

I should note that another aspect here is that so far it appears that this situation only occurs when fetching multiple heads (as now commonly happens because we fetch missing tags in recent bzr versions). I'm not certain that fetching one head at a time can't hit a similar problem, but if that is safe it suggests that perhaps a fix could be:
 * when calculating which records to stream do so independently for each head and then send the union of those streams.

That feels too expensive to me though (both in computation effort, and in possibly sending redundant data).

I don't know what's changed but this seems to be happening to an increasing number of branches in <http://package-import.ubuntu.com/status/>, so I'm going to escalate this in bzr.

Martin Pool wrote:
> I don't know what's changed but this seems to be happening to an
> increasing number of branches in <http://package-
> import.ubuntu.com/status/>, so I'm going to escalate this in bzr.

The hypothesis is that when jubany upgraded to lucid it also upgraded to
a version of bzr that fetches tags by default… and fetching multiple
tips simultaneously seems to be a key (or maybe just common?) trigger
for this edge case.

I just requeued all of the following packages now that we're on bzr 2.3.4 (instead of 2.4b4):

anjuta bin-prot binfmt-support binutils binutils-z80 bird c++-annotations cacti cheese cl-sql compiz-fusion-plugins-extra conkeror crash cupt d-shlibs dbus debhelper dh-make-perl dialog dkms dma doxygen dpkg-cross feh fglrx-driver firebug firmware-nonfree fizmo freebsd-libs freetalk gawk gecode giplet gitpkg glpi gmerlin gnome-icon-theme gnumed-client gnumed-server gnupg-pkcs11-scd gource gpodder gpsd gpsk31 gsmartcontrol gst0.10-python gstreamer0.10-ffmpeg gtest gtk+2.0 gtkwave hardening-wrapper hatari hdf5 heimdal hplip hugin iceweasel imagej iptables iso-codes jags javatools jqueryui kdeaccessibility kdeadmin kdegames kdegraphics kdeplasma-addons kdewebdev klibc ldb libaqbanking libcompress-raw-zlib-perl libconfig-model-perl libdatetime-timezone-perl libdevel-declare-perl libdist-zilla-perl libffi libgadu libhtml-template-pro-perl libimager-perl libio-async-perl libisoburn libisofs libkarma liblocale-gettext-perl libmail-box-perl libmodule-corelist-perl libmoose-perl libnet-ssh2-perl libpng libsvn-hooks-perl libtest-www-mechanize-perl libtorrent-rasterbar libusb-1.0 linux-2.6 liquidsoap live-manual lua5.1 massxpert merkaartor meta-kde metacity miro mlt mozilla-devscripts mozilla-noscript nbd neon27 nginx nilfs-tools nodm nss openmsx-debugger org-mode pastebinit pbzip2 pciutils pidgin pilot-link pkg-kde-tools pstoedit puppet pyparted python-application python-networkx pywebdav qlandkartegt qsynth r-base r-zoo rcpp rxvt-unicode sane-backends serf sgt-puzzles shorewall-doc socat sphinx sphinxsearch squid3 sugar-hulahop swt-gtk tangerine thunderbird tinymce tomcat6 torbutton tortoisehg trac tryton-client tryton-modules-account tryton-modules-account-de-skr03 tryton-modules-account-statement tryton-modules-analytic-invoice tryton-modules-country tryton-modules-purchase tryton-modules-sale tryton-modules-stock tryton-modules-stock-location-sequence tryton-server tzdata unbound usermode valgrind varnish viking vinagre vlc w3m wdm webgui websvn wireshark wvdial xml-security-c xserver-xorg-input-mouse zendframework zsh-beta

Moving jubany back to bzr 2.3 did seem to avoid these problems, so fixing bug 771184 will probably largely fix this.

Not having versioned tags makes it very hard to get rid of tags; therefore very hard to get rid of tags that point at irrelevant history; so that keeps floating around.

As a quick regression avoidance, bug #771184 is relevant.

Is there any practical difference between this bug 485601?

bug 485601 is about bzr-svn triggering this bug - creating slightly inconsistent text revision metadata. That issue has been fixed.

I hit this just now attempting to checkout lp:maria
Although my stack trace more resembles that in one of the duplicate bugs - bug 890373
(My stack trace is effectively the same as that bug, with some very minor differences in lines numbers)

It also deleted everything. It would have been useful if it left the downloaded repo at the most recent version it got to so it was possible to attempt to resume... This after about ~400 MBytes and 1/2 hour ... X-(

bzr 2.5.1 on python 2.6.6 on squeeze

joni@mpi1:/mpi3/S3/maria$ bzr branch lp:maria trunk
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps
missing chk node(s) for parent_id_basename_to_file_id maps

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 920, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1131, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 673, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 695, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 136, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1493, in run
    source_branch=br_from)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 366, in sprout
    create_tree_if_local=create_tree_if_local)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 132, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 416, in _sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 1267, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.7/dist-packages/bzrlib/decorators.py", line 218, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2584, in fetch
    find_ghosts=find_ghosts)
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 77, in __init__
    self.__fetch()
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 104, in __fetch
    self._fetch_everything_for_search(search_result)
  File "/usr/lib/python2.7/dist-packages/bzrlib/fetch.py", line 132, in _fetch_everything_for_search
    stream, from_format, [])
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 2050, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repository.py", line 633, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1727, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/usr/lib/python2.7/dist-packages/bzrlib/repofmt/pack_repo.py", line 1607, in _commit_write_group
    "Cannot add revision(s) to repository: " + problems_summary)
BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps
missing chk node(s) for ...