'BTreeBuilder' object has no attribute '_find_ancestors'

Probably a fastimport bug, re-affect to bzr if needed.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vincent Ladeuil wrote:
> Probably a fastimport bug, re-affect to bzr if needed.
>

...

> + AttributeError: 'BTreeBuilder' object has no attribute '_find_ancestors'

Maybe not entirely a fast-import bug. This looks like fast import is
using one of the 'fast' apis for doing heads checks, etc. However part
of that load is done while a pack is in-progress, which means it gets
read from the Builder that hasn't been written to disk yet. I think the
answer is that BTreeBuilder should implement _find_ancestors so that
KnitGraphIndex can use it.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkulzfQACgkQJdeBCYSNAAPxsACfb4Xp3prMF10lhMS6DDieCNCt
vDIAoJRApCE8UHXdPn7/AoY27o0c5GMA
=yx30
-----END PGP SIGNATURE-----

Hello!

Just to report that after pulling latest code from bzr-fastimport and running tests, I get the same error which, at the moment makes ther plugin unusable. :-(

(We would like to use it to do 2-way sync between darcs <---> bzr and to be able to use LP for bug tracking.)

Sincerely,
Gour

I am also getting this error while attempting to import from git. Attached is the script I am using that generates the error and the error log.

Of note is that the first import works correctly, but any after that (even disregarding import-marks) fail with this error.

And the script... since it would only let me add one attachment

Upgrading to bzr 2.2b4 fixed this for me. I suggest closing this bug.

Hmm, apparently the fix did not stick, I still get this error intermittently and I have not yet figured out a cause

Upon more investigation it appears simply duplicating the _find_ancestors method from BTreeGraphIndex to BtreeBuilder solves this issue, it also works without this change when using pyrex, reopening the bug in bzr

Scratch the comment about pyrex, that doesn't seem to affect this bug, I just have too many versions of bzr installed.

Any further thoughts on this issue? Duplicating the code for _find_ancestors fixes the issue, is there something else people would prefer to do?

This bug is really a PITA, any fix planned?

One more affected person here.

I can't reproduce this issue with bzr 2.3 or 2.4 and bzr-fastimport, neither manually nor using the script provided by termie.

I am also having trouble reproducing the bug, I added tests for git-bzr-ng using a variety of versions of bzr-fastimport and a variety of versions of bzr, but nothing has triggered the error so far, the tests are based on the same test script I originally attached.

https://github.com/termie/git-bzr-ng is where that code lives, the tests/test_basic.py file has the tests but, again, as it all passes it won't be of much use to you unless you want to use it as a framework for writing additional tests.

All that said, I do have a current case that appears to be successfully triggering the issue, I will attach the appropriate files along with debug output

Additional attachments, marks file

additional attachments, marks file

Walking through the traceback a bit, in find_ancestry in /Users/termie/.venv/nova/lib/python2.6/site-packages/bzrlib/index.py, it appears to only attempt to access that protected method when a key is missing, which in this case seems like it would be triggered by either (a) a dropped commit somewhere, (b) a dropped mapping somewhere, or (c) an expected condition where a key is considered missing. A key in this case is, I think, a commit id.

My gut instinct is that it is due to (a), somewhere along whatever conversion process or upstream manipulations a commit id has either changed or is no longer present (or if, for example, the conversion process, the export from git, produces a parent dependency on something that was never there in bzr, or vice-versa) and triggers a code path that is otherwise untested.

The brunt of it is that the code is trying to access a member method that doesn't exist, copying the implementation from the other class seems to work however I don't understand the code and data model well enough to say for certain that it would be the correct operation for this index type. Either way there is at very least a bug in that there is a code path that calls a non-implemented method and that should be resolved.

Any luck tracking this down? I've gotten this a few times now, and also a similar "AttributeError: 'InMemoryGraphIndex' object has no attribute '_find_ancestors'"

On Sun, 2011-07-31 at 14:44 +0000, Aman Gupta wrote:
> Any luck tracking this down? I've gotten this a few times now, and also
> a similar "AttributeError: 'InMemoryGraphIndex' object has no attribute
> '_find_ancestors'"
I'm not yet familiar enough with this code, so I was going to ask for
some help from one of the other developers. This hasn't happened yet
because of some more critical issues, but it's still on my todo list.

Cheers,

Jelmer

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 8/1/2011 11:24 AM, Jelmer Vernooij wrote:
> On Sun, 2011-07-31 at 14:44 +0000, Aman Gupta wrote:
>> Any luck tracking this down? I've gotten this a few times now, and
>> also a similar "AttributeError: 'InMemoryGraphIndex' object has no
>> attribute '_find_ancestors'"
> I'm not yet familiar enough with this code, so I was going to ask for
> some help from one of the other developers. This hasn't happened yet
> because of some more critical issues, but it's still on my todo
> list.
>
> Cheers,
>
> Jelmer
>

I thought this was fixed in newer releases.

Otherwise, it shouldn't be too hard to just reproduce the api on
BTreeBuilder.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk47/UUACgkQJdeBCYSNAAOg2QCfWxct9EPshATWEVGYh9xhLyEi
hgQAn3mf8mXlMMuNI9ngHLQ08oq79TL6
=AZKf
-----END PGP SIGNATURE-----

On Fri, 2011-08-05 at 14:25 +0000, John A Meinel wrote:
> On 8/1/2011 11:24 AM, Jelmer Vernooij wrote:
> > On Sun, 2011-07-31 at 14:44 +0000, Aman Gupta wrote:
> >> Any luck tracking this down? I've gotten this a few times now, and
> >> also a similar "AttributeError: 'InMemoryGraphIndex' object has no
> >> attribute '_find_ancestors'"
> > I'm not yet familiar enough with this code, so I was going to ask for
> > some help from one of the other developers. This hasn't happened yet
> > because of some more critical issues, but it's still on my todo
> > list.
> >
> > Cheers,
> >
> > Jelmer
> >
>
> I thought this was fixed in newer releases.
>
> Otherwise, it shouldn't be too hard to just reproduce the api on
> BTreeBuilder.
I can confirm it's still open.

Cheers,

Jelmer

I'm using git-bzr-ng, that uses bzr and bzr-fastimport.
I'm using last commit version of bzr (2.5.0dev2, rev no: 6174) and bzr-fastimport (0.12, rev no: 332) as of 28 September 2011.

git-bzr-ng will issue 2 command in one of its process. The first will run flawlessly, the second will trigger the exception.

Here are both commands:

   bzr fast-import --import-marks=MY_PYTHON_DIR/bzrprojectrepo/.git/bzr/map/master-bzr --export-marks=MY_PYTHON_DIR/bzrprojectrepo/.git/bzr/map/mybranch-name-bzr - MY_PYTHON_DIR/bzrprojectrepo/.git/bzr/repo/mybranch-name
   git fast-export -M --import-marks=MY_PYTHON_DIR/bzrprojectrepo/.git/bzr/map/master-git --export-marks=MY_PYTHON_DIR/bzrprojectrepo/.git/bzr/map/mybranch-name-git mybranch-name

Last command triggers the exception with this trackback:

Traceback (most recent call last):
  File "bzrlib/commands.py", line 924, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "bzrlib/commands.py", line 1124, in run_bzr
    ret = run(*run_argv)
  File "bzrlib/commands.py", line 677, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "bzrlib/commands.py", line 699, in run
    return self._operation.run_simple(*args, **kwargs)
  File "bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "bzrlib/plugins/fastimport/cmds.py", line 314, in run
    user_map=user_map)
  File "bzrlib/plugins/fastimport/cmds.py", line 40, in _run
    return proc.process(p.iter_commands)
  File "bzrlib/plugins/fastimport/processors/generic_processor.py", line 311, in process
    super(GenericProcessor, self)._process(command_iter)
  File "fastimport/processor.py", line 76, in _process
    handler(self, cmd)
  File "bzrlib/plugins/fastimport/processors/generic_processor.py", line 536, in commit_handler
    handler.process()
  File "fastimport/processor.py", line 159, in process
    self.post_process_files()
  File "bzrlib/plugins/fastimport/bzr_commit_handler.py", line 673, in post_process_files
    self._get_inventories)
  File "bzrlib/plugins/fastimport/revision_store.py", line 376, in load_using_delta
    rev.parent_ids)
  File "bzrlib/decorators.py", line 154, in read_locked
    result = unbound(self, *args, **kwargs)
  File "bzrlib/vf_repository.py", line 1843, in get_known_graph_ancestry
    known_graph = self.revisions.get_known_graph_ancestry(revision_keys)
  File "bzrlib/versionedfile.py", line 1440, in get_known_graph_ancestry
    parent_map, missing_keys = self._index.find_ancestry(keys)
  File "bzrlib/groupcompress.py", line 2103, in find_ancestry
    return self._graph_index.find_ancestry(keys, 0)
  File "bzrlib/index.py", line 1527, in find_ancestry
    search_keys = index._find_ancestors(search_keys,
AttributeError: 'BTreeBuilder' object has no attribute '_find_ancestors'

I can confirm that copying the method BTreeGraphIndex._find_ancestor(..) on BTreeBuilder will REMOVE the exception and solve the bug.

Unfortunately simply copying _find_ancestors isn't a fix for this bug; this will only make _find_ancestors() work if key_count is False.

The rest of the method can't work since (as far as I can tell) BTreeBuilder doesn't have a node_ref_lists attribute.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 9/28/2011 5:41 PM, Jelmer Vernooij wrote:
> Unfortunately simply copying _find_ancestors isn't a fix for this
> bug; this will only make _find_ancestors() work if key_count is
> False.
>
> The rest of the method can't work since (as far as I can tell)
> BTreeBuilder doesn't have a node_ref_lists attribute.
>
> ** Changed in: bzr Assignee: Jelmer Vernooij (jelmer) =>
> (unassigned)
>
> ** Changed in: bzr Status: In Progress => Triaged
>

I don't think the method can just be copied, but the conceptual
interface is perfectly available. We just return
~get_parent_map(values) in the worst case.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6DfiQACgkQJdeBCYSNAAO37QCg1qo1HOwmMopS3Tuu1D1/jk7x
fW0AoJqoOitjGUKdhOzXKh8xsjiJx+va
=mwjo
-----END PGP SIGNATURE-----

Same Problem here as #23. Please fix if possible, this is is SOO important for true collaboration with other DVCS.

So I keep running into this as well, using git-bzr-ng

I have noticed that it seems the map files are being corrupted, or at least no longer ordered, and that new entry revision numbers don't follow the sequence

Eg.
  .git/bzr/map/master-git file starts in a clean ordered state (tail below)

:1921 f6ae3ad3fb09386e84af64aa362073a8a149f399
:1922 76b042bdbf71ed243f297840b8c83017b117c342
:1923 e78101ca78ef017763b2dd410a3773bc895696e2
:1924 5b9f36d19950415ca15be87ed1d73a31263fed31
:1925 2d12d6de3eb9fd5ef38d50883c6c6f9e950682aa
:1926 cb7760b89181f28ff43b14694ab19d60dcc4ceb7
:1927 b08724ebfb908a56a9ec2dcb46c4205810c5e76d
:1928 960a3fb12dab066e7ca4050f4c92583d12229d24
:1929 e527e97be98ce8be92611b5085c4dc4d0db0575e
:1930 6e64fcdadd08c884fa630d1f2157927435c1dcaa

but after trying to do
  git bzr push

the map file is completely out of order (tail below)

:745 6b654fb354ea1fdef9f1fb7109c682917427567d
:1403 0f56b35105744659e62d9a81d6d1141372aa1fd7
:816 af5abab61940ea2b02ca92164e530fe3949b4442
:1256 75f90156567dff18523a2245d5bb2dea2140fb4a
:1760 dd26b2d93c31f87e84a07c12a8ffeda486b170a8
:986 e080917546f62260fc9e0e3f98b9d79768053df9
:162 3aecde75203c436bd92d35de6d1ba44033e6e17a
:378 701d356d835d89611dbf846aa2d0bf86600ccbd5
:1777 f0fa6f42de7129634424d2df6b4162fc379b6e73
:1503 c1bd59e54332b7d021eee3e7bd44f4a8afb76673

and after sorting I find 2 new entries, that seem to
correspond to the two commits I am pushing but their
revision # seem to be incrementing by 3 instead of 1

:1927 b08724ebfb908a56a9ec2dcb46c4205810c5e76d
:1928 960a3fb12dab066e7ca4050f4c92583d12229d24
:1929 e527e97be98ce8be92611b5085c4dc4d0db0575e
:1930 6e64fcdadd08c884fa630d1f2157927435c1dcaa
:1933 46fa4e95ef25a411c2ab87f91c9603809828597b
:1936 b4424da76c71e4a5ff60a53c15093d78b44ad6be

The .git/bzr/map/master-bzr file sees the same type of reordering but does not gain the 2 additional entries

related bug in git-bzr-ng https://github.com/termie/git-bzr-ng/issues/38

Peter Waller boiled the problem down to a minimal case on irc and attached a script to the github issue. I edited it to push to local branches instead of lp. It needs git-bzr to work, which I cloned from https://github.com/termie/git-bzr-ng.git and then symlinked into ~/bin/ so git would pick it up. Git-bzr uses bzr-fastimport to do the heavy lifting. I expect this can be further reduced to a bzr-fastimport test.

Versions used: bzr 2.6.0dev2 (r6523), bzr-fastimport r356, git-bzr-ng 011a644ccadae9bb577c47d5ac9a5f782a9cca1c

I'll see if I can add a minimal BTreeBuilder._find_ancestors.

lp:~larstiq/bzr/bug541626 takes the approach of implementing _find_ancestors nearly the same way as CombinedGraphIndex.get_parent_map. It seems to work for bug541626.sh, but that might be accidental.

Jam: could you give it a look and say if this is the right direction to go?

I just tried out Wouter's branch, and it seems to work. I can again collaborate and keep my sanity, yay! (The insane part having to copy files between version control systems as a workaround..)

I faced the same issue using bzr 2.5 (currently on precise, using ubuntu default package). I downloaded the latest version possible of bzr (branching the trunk, rev.6553, bzr 2.6.0dev3), and _find_ancestors seem to be already implemented in bzrlib/btree_index.py (as Wouter's branch kinda does). Using the fresh version of bzr, I wasn't able to reproduce the issue. Should we consider this bug fixed in bzr upstream?

This is still present with bzr 2.6.0b2 from the beta ppa on precise

Could we please reopen this issue:

on python-bzrlib 2.6.0~beta2-0ubuntu1 and trunk (or released, the same) bzr-fastimport:

bzr: ERROR: exceptions.AttributeError: 'BTreeBuilder' object has no attribute '_find_ancestors'

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 930, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1141, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 673, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 697, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 136, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/home/zyga/.bazaar/plugins/fastimport/cmds.py", line 312, in run
    user_map=user_map)
  File "/home/zyga/.bazaar/plugins/fastimport/cmds.py", line 42, in _run
    return proc.process(p.iter_commands)
  File "/home/zyga/.bazaar/plugins/fastimport/processors/generic_processor.py", line 310, in process
    super(GenericProcessor, self)._process(command_iter)
  File "/usr/lib/python2.7/dist-packages/fastimport/processor.py", line 75, in _process
    handler(self, cmd)
  File "/home/zyga/.bazaar/plugins/fastimport/processors/generic_processor.py", line 535, in commit_handler
    handler.process()
  File "/usr/lib/python2.7/dist-packages/fastimport/processor.py", line 158, in process
    self.post_process_files()
  File "/home/zyga/.bazaar/plugins/fastimport/bzr_commit_handler.py", line 672, in post_process_files
    self._get_inventories)
  File "/home/zyga/.bazaar/plugins/fastimport/revision_store.py", line 375, in load_using_delta
    rev.parent_ids)
  File "/usr/lib/python2.7/dist-packages/bzrlib/decorators.py", line 155, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 1896, in get_known_graph_ancestry
    known_graph = self.revisions.get_known_graph_ancestry(revision_keys)
  File "/usr/lib/python2.7/dist-packages/bzrlib/versionedfile.py", line 1441, in get_known_graph_ancestry
    parent_map, missing_keys = self._index.find_ancestry(keys)
  File "/usr/lib/python2.7/dist-packages/bzrlib/groupcompress.py", line 2084, in find_ancestry
    return self._graph_index.find_ancestry(keys, 0)
  File "/usr/lib/python2.7/dist-packages/bzrlib/index.py", line 1529, in find_ancestry
    search_keys = index._find_ancestors(search_keys,
AttributeError: 'BTreeBuilder' object has no attribute '_find_ancestors'

By some reason 2.6.0 beta2 is the default and only available bzr version in Ubuntu Quantal. If suffer from this bug as well and there is no obvious downgrade possibility (ppa has no packages for Quantal other than bzr-git).

Is there any other possibility of downgrading than using the sources?

Btw. putting beta release packages into stable Ubuntu releases seems to be a bug for me as well...

Here's an even more minimal version to reproduce the issue 100%.

I can reproduce on v2.5.1 and latest trunk: 6571. The issue is definitely still there.

No, the bug is not in bzr, it's in bzr-fastimport, a it happens since versions 2.4 of bzr because the name of one class changed.

Here's the fix.

Forget my last patch. While something along those lines might or might not make sense, it doesn't fix the issue completely.

Here's a patch that does. Basically, we can't call repo.get_known_graph_ancestry() while in the middle of a write group, so let's disable the whole known graph feature.

Here's a very small patch to the bzr test framework to reproduce the issue 100% without any bzr-fastimport interaction. Whether or not this is actually a bug is up to bzr developers.

thanks, felipes fix to disable graphing helped me, test from git-bzr-ng passes as well

i used git-bzr-ng to sync, and what i noticed when i looked my mapfiles, that at some point id in first column was duplicate, afaik in bzr mapfile (i don't have the files around anymore to show them)

Patch is attached in comments.

The attachment "Proposed fix" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Status changed to 'Confirmed' because the bug affects multiple users.

Thanks Filipe, I uploaded bzr-fastimport 0.13.0-4 to Debian unstable with your patch included.

This bug was fixed in the package bzr-fastimport - 0.13.0-4

---------------
bzr-fastimport (0.13.0-4) unstable; urgency=low

  * QA upload.
  * Disable know graph feature to fix a crash. (LP: #541626)
  * Switch to debhelper 9.

 -- Benjamin Drung <email address hidden> Tue, 26 Nov 2013 17:18:12 +0100