missing chk node(s) for id_to_entry maps

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Geoff Jacobsen wrote:
> Public bug reported:
>
> This occurs in conjunction with the bzr-svn plugin (version 1.0.0dev)
> and bzr versions 2.0.0 and 2.0.1
>
> I have an original bzr repository which is push into a subversion
> repository. This is done half hourly if there are any changes on the
> original branch. No changes are done on the svn repo other than the
> push.
>
> Eventually there is a branch and merge on the original which results not
> being able to push to svn since a rebase is needed.
>
> At this stage if I "bzr branch" the svn repo and try to pull the
> original into it I get the error (both on Ubuntu and Solaris) also if I
> try a rebase on a "bzr co svn-repo" I get the same error.
>
> ** Affects: bzr
> Importance: Undecided
> Status: New
>
>
> ** Tags: bzr-svn
>

 affects bzr-svn

I don't know that it is bzr-svn's fault, but as it is provoking this
situation, I'm including bzr-svn in the bug report.

 affects bzr
 status confirmed
 importance medium

My guess is that there is a bad interaction with ghosts introduced by
bzr-svn that are(n't) getting filled in when fetching from the original
bzr branch.

Just shooting out a possibility, I would guess that you end up with a
revision graph like:

 A
 |\
 B C
 |/|
 D | # pushed to svn
 |\|
 E F
 |/
 G

And fetching from SVN gives you something like:

 A
 |
 B ? #ghost
 |/
 D

Then fetching from the bzr branch tries to fill in only new children,
such as:

 A
 |
 B ? #still a ghost
 |/|
 D |
 |\|
 E F
 |/
 G

And at that point, we fall over. The stream request did not request the
chk pages for C, so the server specifically excludes them.

However, those pages end up also present in F (and not present in D), so
we don't have them locally.

The various failures:

1) The ancestry search sees that we have D, and stops searching. Thus we
never see that we are missing revision "C" locally.

2) The fetch requests revision E F G versus revision D.

3) The remote side translates that into E F G versus C D.

I haven't proven any of this. But basically it is a failure because
bzr-svn introduces a ghost which we then don't fetch. And the current
codebase expects that if you have a revision, you have all of its
ancestry. (Which ghosts violate.) And the chk map code is especially
sensitive to this.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksGucEACgkQJdeBCYSNAAODuACfaKJ085K7Y3dgWrPTZq4vZnf5
UN0AnjVtFsbMON1N0fw9ghHQ+EBpfaJD
=hSov
-----END PGP SIGNATURE-----

Geoff, is your repository public?

On 8 April 2010 11:24, Jelmer Vernooij <email address hidden> wrote:

> Geoff, is your repository public?
>
>
Sorry, no but the problem is easy to re-produce. All you need to do is push
a change to subversion which will cause a re-org of the tree.

On Wed, 2010-04-07 at 23:59 +0000, Geoff Jacobsen wrote:
> On 8 April 2010 11:24, Jelmer Vernooij <email address hidden> wrote:
>
> > Geoff, is your repository public?
> Sorry, no but the problem is easy to re-produce. All you need to do is push
> a change to subversion which will cause a re-org of the tree.
I haven't encountered this problem before and I've pushed quite a few
repository reorganizations using bzr-svn. It's definitely not any
reorganization that triggers this bug but something more specific.

  status incomplete

Cheers,

Jelmer

I'll try and publish the steps to create it on an ubuntu box but I am tied
up for the next week

On 8 April 2010 12:30, Jelmer Vernooij <email address hidden> wrote:

> ** Changed in: bzr
> Status: Confirmed => Incomplete
>
> --
> missing chk node(s) for id_to_entry maps
> https://bugs.launchpad.net/bugs/485601
> You received this bug notification because you are a direct subscriber
> of the bug.
>

I have suffered the same problem in an annoyingly unreproducible way. I have tried many combinations of merging, pushing and pulling between bzr branches of the same svn repo project, and cant find a reproducible way.

Geoff... were you using any shared bzr repositories when you suffered this problem?

Yes the repos are shared and it happens when two merges
occur independently and then merged to parent forcing one tree to be
reordered.

On 5 May 2010 17:29, Philip Peitsch <email address hidden> wrote:

> I have suffered the same problem in an annoyingly unreproducible way. I
> have tried many combinations of merging, pushing and pulling between bzr
> branches of the same svn repo project, and cant find a reproducible way.
>
> Geoff... were you using any shared bzr repositories when you suffered
> this problem?
>
> --
> missing chk node(s) for id_to_entry maps
> https://bugs.launchpad.net/bugs/485601
> You received this bug notification because you are a direct subscriber
> of the bug.
>

That's what I am experiencing as well. I have two developers with independant branches of the same svn code in two separate shared repositories with no working trees, and if one dev tries to bzr pull from the other dev's branch, this crash is always experienced.

Is there a way to get more information out about the branch history etc to help with debugging this problem? I unfortunately can't simply release the source as it is a propriatary app (and around 2Gb worth of histroy at this point I might add), but I am happy to summarize most data and send to specific emails if that would be able to provide any extra information.

We're experiencing this problem as well ... any solution in the pipeline?

This bug has pretty much sunk all our bzr branches now (we are/were trying to migrate from svn).

Is there likely to be any time soon to get some help looking at this? Though I can't open up our source control access to outside access.... I can make myself available on IRC between 9-5pm Melbourne, Australia time if that's helpful.

Geoff & jeffrey... I am wondering if this is possibly caused by Bug #587819. Can you both see if you're able to create clean checkouts of the svn repo using a non-shared-repository branch? (i.e., outside the shared repo, do a bzr co <svn repo url>). If you had the race condition bug, it should fail in theory...

I am the only one accessing the svn repository and the bug is
100% reproducible so I don't believe it has to do with bug #587819

Ahh. Fair enough. How many revisions do you have in your branch at the moment if I may ask? I'm interested in fixing this one, but am just about to lose my test repo due to the previously mentioned issue. Do you have any idea on reproduction steps yet?

Thanks,

Philip

Marking as incomplete as I haven't encountered this myself and I don't have a way to reproduce this.

I just had this problem, here are the steps that brought this error message. I will try to reproduce later with a public svn repository. Here are the logs to help me reproduce it later.

Note that we use subversion in a braindead manner (i.e. we commit inside the tags).

Subversion side :
We have tagged twice our trunk, first with tag "r3", then with tag "r4". Then, we did several commits in both tags, both from svn and from bzr (by pulling changes then committing).

Bazaar side:
1) Branch from the subversion tag r3
repo$ bzr branch svn://host/repo/tags/r3
2) Branch from the subversion tag r4
repo$ bzr branch svn://host/repo/tags/r4
3) Try to merge or pull a revision from the bzr branch "r3" into the branch "r4"
repo/r4$ bzr merge -r1218 ../r3/

=> Crash with the following message

bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3794, in run
  File "bzrlib\builtins.pyo", line 3923, in _get_merger_from_branch
  File "bzrlib\merge.pyo", line 363, in from_revision_ids
  File "bzrlib\merge.pyo", line 505, in set_other_revision
  File "bzrlib\merge.pyo", line 522, in _maybe_fetch
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\branch.pyo", line 593, in fetch
  File "bzrlib\repository.pyo", line 1704, in fetch
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\repository.pyo", line 3432, in fetch
  File "bzrlib\fetch.pyo", line 83, in __init__
  File "bzrlib\fetch.pyo", line 109, in __fetch
  File "bzrlib\fetch.pyo", line 137, in _fetch_everything_for_search
  File "bzrlib\repository.pyo", line 4274, in insert_stream
  File "bzrlib\repository.pyo", line 4377, in _locked_insert_stream
  File "bzrlib\repository.pyo", line 1563, in commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2314, in _commit_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2152, in _commit_write_group
BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

bzr 2.1.0 on python 2.5.4 (Windows-XP-5.1.2600-SP2)
arguments: ['c:\\Program Files\\Bazaar\\bzr.exe', 'merge', '-r1218', '../2.42r3/']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.1.0]
  explorer C:\Program Files\Bazaar\plugins\explorer [1.0.0rc1]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.1.0]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store [2.1.0]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.18.1]
  rebase C:\Program Files\Bazaar\plugins\rebase [0.5.5]
  svn C:\Program Files\Bazaar\plugins\svn [1.0.2]
 ...

This bug just sunk all the devs again when upgrading to 2.2.1

Could this bug potentially be caused by https://bugs.launchpad.net/bzr/+bug/522637 ?

Unfortunately I still can't do much about this, as there is no clear way of reproducing it.

The ghost handling in bzr-svn 1.0.4 has changed significantly, so it'd be interesting to hear if anybody can still reproduce this with that version of bzr-svn and a clean clone (into a new shared repository) from their svn repository.

This bug appears to happen using the following workflow:

* dev a & dev b both share a network repo, and both have a local repo as well
* dev a does some work and commits to their branch (this is immediately pushed up to the local repo via a bound branch)
* dev a merges this work into svn trunk
* system admin updates remote repo
* dev b updates local svn trunk VIA SVN
* dev b then attempts to merge changes from svn trunk into their local feature and commit (which is also a bound branch to the remote repo)

Explosions happen!

At a glance bzr-svn seems to be using bzr APIs correctly. It adds inventory objects to a bzr repo via add_inventory_by_delta, which is a public API that is intended to be used for exactly that. So that seems fine.

But I wonder if perhaps it's possible that bzr-svn isn't roundtripping bzr inventories perfectly? Is it possible that it adds a bzr revision to a SVN repo, and then another user fetches that revision and gets an inventory with the same ID but slightly different contents? That might explain this behaviour.

Update from IRC: we've found a revision in Philip Peitsch's scenarios that is causing this trouble. When fetched from the SVN repo into bzr repo A it has different CHK root keys than after fetching from A into B. But in both A and B the CHK maps seem to be in canonical form! So apparently A and B have different content for the same inventory?

Philip has sent me dumps of the CHK maps of some of the revisions converted from the same SVN repo at different times. The inventories for those revisions are the same in every detail *except* some of the inventory entries have 'revision' attributes. It appears the 'revision' attributes of inventory-entries generated by bzr-svn from SVN are changing every time a new bzr revision is added to the SVN repository!

The original bzr repository was created after the svntrunk suffered from https://bugs.launchpad.net/bzr-svn/+bug/515850. To recover from this bug, manually deleted all bzr:XXX revision properties from the repository. I am quite sure we cleaned out all svn-caches on all PC's as well, but it's worth considering if that history has lead to this current situation somehow.

I've checked dumps of the CHK maps for the rev just prior to our first "new" bzr checkin being landed in SVN (post bug 515850), and these differ slightly. No svn checkins prior to that first bzr commit seem to differ from what I've seen however.

Just to clarify the previous comment: To recover from this bug, manually deleted all bzr:XXX revision properties from the SUBVERSION repository, and threw away the old BZR repository. I then created a brand new checkout of svntrunk into a clean BZR repository, and this is what we've been using since then (early June, 2010)

I just reproduced this on bzr-svn 1.0.4.

I reproduced it fairly simply: I have 2 repositories, one on a desktop and one on a laptop.

desktop$ bzr init-repository .
desktop$ bzr get svn+http:.../trunk
laptop$ bzr init-repository .
laptop$ bzr get svn+http:.../trunk
laptop$ bzr branch trunk feature-branch
laptop$ hack hack hack
laptop$ bzr ci -m "stuff"
desktop$ bzr get bzr+ssh://laptop/~/feature-branch
... and that causes the traceback.

There is literally only one bzr revision in my feature branch which hasn't been pushed to SVN which seems to be causing the problem.

I'm still having trouble trying to reproduce this, but I'm running into it quite frequently. I really can't take Philip's approach (I don't own the repository). :-(

On Thu, 2010-12-23 at 08:45 +0000, John Szakmeister wrote:
> I'm still having trouble trying to reproduce this, but I'm running into
> it quite frequently. I really can't take Philip's approach (I don't own
> the repository). :-(
Since this the solution to this is non-obvious to me at this point I
hope to find some time at the Bazaar Rodeo early January to discuss and
fix it.

i am bit by this too. i've been tracking chromium source repo for a few months and only have lagged behind recently. a pull from svn will need more memory than i have so i decided to pull from lp:~vcs-imports/chromium-browser/trunk then the glorious crash came ... so sad

Just got hit by this. I have a shared repo (colo workspace) into which I have branched a svn trunk. I worked with this for months, creating local feature branches and merging them back into trunk and vice versa, and pushing up to svn trunk.

Then yesterday I branched the svn trunk on the svn server directly using TortoiseSVN and then branched that into my local bzr shared repo. I cherry-picked a few changes from the bzr trunk into the bzr branch, committed and pushed up to teh svn branch without trouble. Today, I tried to do the same thing again, cherry-pick a revision from trunk into the branch and it gave me the BzrCheckError.

I'm running bzr 2.3.0 and bzr-svn 1.0.5dev (as bundled with bzr 2.3.0).

Was also struck by this.

Where M1 is my desktop and M2 is my laptop

M1 :
  merged some changes from bzr topic branch into SVN trunk
  pushed these to SVN

# somewhat later
M2 :
  pulled this merged revision from SVN
  also pulled some later revisions committed to SVN by other SVN clients

M1 :
  attempt to pull branch from M2
  error occurs

M1 :
  in attempt to fix missing ghosts, tried push --overwrite of this branch to M2 (over bzr:// smart server)
  this does not fix the problem ; presumably M2 just reports that it already has the tip revision and nothing actually happens
  would this work if you tried over file:// ?

Resolution :
M2 :
  delete repository
  create new empty repository
M1 :
  push SVN trunk branch to M2 repository
M2 :
  pull new SVN revisions
  # only M2 has access to SVN at the moment because the VPN concentrator is prejudiced against non-Windows VPN clients
M1 :
  pull new SVN revisions from bzr branch on M2
  success!

I can imagine that where multiple persons are merging and then pushing to SVN, this would be a real showstopper.

You should be able to prevent this by only pushing Bazaar revisions to SVN from one machine, or by using rebase exclusively. This has not happened to my colleagues who only use rebase. It might be simplest to set up a bzr repository on the SVN server that pulls desired SVN branches via SVN post_commit hooks and pushes back any revisions it receives to these branches.

Would a command to deliberately lock a bzr branch assist this use case? Some SVN commits can be quite prolonged and it would be annoying to push a revision, only to discover it had then been overwritten by a pull from the SVN branch (--overwrite being pretty much a given, since you cannot rewrite SVN history). The lock command could be issued by the SVN pre-commit hook, then released before the post-commit pull. Although a quick and dirty echo of a lock file might work.

Bazaar (bzr) 2.3.1 bzr-svn 1.0.5dev # standard Natty packages

This should now *finally* be fixed. bzr-svn will now always store the right text revisions and text parents, and has a test framework to test the various scenarios.

This means that bzr-svn should now always write correct data to svn repositories. There might still be a few out there with incorrect metadata that make bzr-svn blow up; bug 628354 is about that.

I just got this error when attempting to merge from another local branch:

$ bzr merge ../trunk-bzr-svn
bzr: ERROR: bzrlib.errors.BzrCheckError: Internal check failed: Cannot add revision(s) to repository: missing chk node(s) for id_to_entry maps

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 927, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 1127, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 692, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib64/python2.7/site-packages/bzrlib/commands.py", line 714, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/builtins.py", line 3941, in run
    location, revision, remember, possible_transports, None)
  File "/usr/lib64/python2.7/site-packages/bzrlib/builtins.py", line 4074, in _get_merger_from_branch
    other_revision_id, base_revision_id, other_branch, base_branch)
  File "/usr/lib64/python2.7/site-packages/bzrlib/merge.py", line 410, in from_revision_ids
    merger.set_other_revision(other, other_branch)
  File "/usr/lib64/python2.7/site-packages/bzrlib/merge.py", line 551, in set_other_revision
    self._maybe_fetch(other_branch, self.this_branch, self.other_rev_id)
  File "/usr/lib64/python2.7/site-packages/bzrlib/merge.py", line 568, in _maybe_fetch
    target.fetch(source, revision_id)
  File "/usr/lib64/python2.7/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/branch.py", line 687, in fetch
    pb=pb)
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 1781, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib64/python2.7/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 3437, in fetch
    find_ghosts=find_ghosts)
  File "/usr/lib64/python2.7/site-packages/bzrlib/fetch.py", line 73, in __init__
    self.__fetch()
  File "/usr/lib64/python2.7/site-packages/bzrlib/fetch.py", line 99, in __fetch
    self._fetch_everything_for_search(search)
  File "/usr/lib64/python2.7/site-packages/bzrlib/fetch.py", line 127, in _fetch_everything_for_search
    stream, from_format, [])
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 4105, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/lib64/python2.7/site-packages/bzrlib/repository.py", line 1640, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib64/python2.7/site-...

Hi Patrick, I suspect this is fallout from data pushed before this bug was fixed. See also bug 628354.

Thanks, Jelmer. Is there any way to work around it? I'm trying to work around bug 702870, which I was able to do before by creating a fresh local branch from the svn repo, merging my changes from my bzr branch (which was failing to push) into the fresh local branch, and then push the "fresh" branch back to the svn repo. But now I can't do that because of this problem.

I'd go back to plain svn, but svn doesn't track changes through renames, so I need bzr with svn to track the changes in the bzr branch.

I'm really willing to help debug, if you can offer some advice of how to do so.

The easiest way to get rid of the issues, post this bug is to simply remove the revision properties from the revisions pushed by bzr. This will break your existing bzr branches though. I can walk you through that if it helps.

Thanks, but unfortunately, I cannot afford to lose my existing bzr branches. Is there a way to tell what information is missing in the rev properties, and to correct it? If I have to do this manually, I can.

fdf

(Whups. Sorry about the last one. My web page wasn't working...)

Just wanted to re-open this. I've just encountered this between a system running bzr 2.4.1 and bzr 2.5.1, running bzr-svn 1.1.0 and 1.1.1dev.

The workflow *seems* to be as follows:

1.
user1 => bound to svn trunk
user2 => bound to svn trunk

2.
user1 commits a change to svn trunk
user2 updates their copy of svn trunk

3.
at a later time, user1 tries to branch directly from another of user2's feature branches made after the checkins occurred and experiences this error.

Hrm... the problem isn't as simple as presented.

After switching and branching various things, the problem seems to have gone away... go figure. The exact working setup we have here is:

- Bazaar Server with a Shared Repository contains the master branch from SVN
- Individual devs create a local shared repo on their machines, and branch from the master branch
- Individual devs then bind their master branch directly to the SVN source branch in order to commit things to trunk
- Individual dev checkouts are bound to feature branches created on the Bazaar Server (in order to allow network backups)

The dev experienced the crash just after checking in to trunk. They had then just created a new feature branch, and the crash occured during the bzr switch command in their checkout to the new feature branch on the Bazaar Server

@peitsche, I suggest you just open a new bug.

Hi Philip,

On 09/11/11 03:45, Philip Peitsch wrote:
> (Whups. Sorry about the last one. My web page wasn't working...)
>
> Just wanted to re-open this. I've just encountered this between a
> system running bzr 2.4.1 and bzr 2.5.1, running bzr-svn 1.1.0 and
> 1.1.1dev.
>
> The workflow *seems* to be as follows:
>
> 1.
> user1 => bound to svn trunk
> user2 => bound to svn trunk
>
Sorry to see you're hitting this still.

Are both the branches that these users have created with bzr-svn 1.1.0
or 1.1.1 ?

This bug is caused by inconsistent data being pushed into svn by
bzr-svn. A clone that was made by an older version of bzr-svn could
still be problematic.

Cheers,

Jelmer

Just to note here, I've replied to Jelmer's questions in bug #887880