Bazaar Git Plugin

zero-padded file modes break git->bzr imports

Bug #581064 reported by Simone Margaritelli
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Bazaar Git Plugin
Confirmed
Low
Unassigned
Breezy
Triaged
Low
Unassigned
Dulwich
Fix Released
Low
Augie Fackler
Dulwich 0.7.0
Launchpad code imports
Triaged
Low
Unassigned

Bug Description

When i try to start a job to import my git repo into the bzr branch, the job fails and i get the following message in the logs :

2010-05-15 11:16:45 INFO Starting job.
2010-05-15 11:16:45 INFO Getting exising bzr branch from central store.
2010-05-15 11:16:45 INFO 35 bytes transferred |
2010-05-15 11:16:46 INFO Importing foreign branch.
2010-05-15 11:16:48 INFO Counting objects: 370
Counting objects: 377, done. 0
2010-05-15 11:16:49 INFO finding revisions to fetch:generating index 0/377
2010-05-15 11:16:49 INFO finding revisions to fetch 1/356
2010-05-15 11:16:49 INFO
Traceback (most recent call last):
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/scripts/code-import-worker.py", line 82, in <module>
    sys.exit(script.main())
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/scripts/code-import-worker.py", line 77, in main
    return import_worker.run()
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/lib/lp/codehosting/codeimport/worker.py", line 437, in run
    return self._doImport()
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/lib/lp/codehosting/codeimport/worker.py", line 575, in _doImport
    overwrite=True, **self.getExtraPullArgs())
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/branch.py", line 453, in pull
    stop_revision, overwrite=overwrite, graph=graph, limit=limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/branch.py", line 403, in _update_revisions
    determine_wants, self.source.mapping, limit=limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 535, in fetch_objects
    objects_iter, store, recorded_wants, pb, limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 420, in import_git_objects
    trees_cache)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 335, in import_git_commit
    allow_submodules=getattr(repo._format, "supports_tree_reference", False))
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 222, in import_git_tree
    base_tree = lookup_object(base_hexsha)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/fetch.py", line 371, in lookup_object
    return target_git_object_retriever[sha]
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/object_store.py", line 479, in __getitem__
    unusual_modes, expected_sha=sha)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/object_store.py", line 389, in _reconstruct_tree
    _check_expected_sha(expected_sha, tree)
  File "/srv/importd.launchpad.net/production/launchpad-rev-9329/optionalbzrplugins/git/object_store.py", line 107, in _check_expected_sha
    expected_sha))
AssertionError: Invalid sha for <Tree 4a698785d734daf98f3aee3a3318f5b32d94de98>: 97f1ccb990d78f6b7a1c9aec69c367594c8fa56f
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended with exit code 1.

Related branches

lp://staging/~xaav/bzr-git/bugfix
Rejected for merging into lp://staging/bzr-git
Jelmer Vernooij: Disapprove
Diff: 15 lines (+4/-1)
1 file modified
object_store.py (+4/-1)
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Which import is this?

Revision history for this message
Simone Margaritelli (evilsocket) wrote :

An import from the git repo on git://github.com/evilsocket/hybris.git .
Sometimes the import goes as it should, but most of the times it fails with this kind of error.

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Easiest way to reproduce this issue:

bzr branch git://github.com/evilsocket/hybris.git
cd hybris.git
rm -rf .bzr/repository/git
bzr git-objects

Changed in bzr-git:
status: New → Triaged
assignee: nobody → Jelmer Vernooij (jelmer)
importance: Undecided → Medium
Tim Penhey (thumper)
Changed in launchpad-code:
status: New → Triaged
importance: Undecided → Medium
tags: added: code-import
Revision history for this message
Jelmer Vernooij (jelmer) wrote : Re: Support for zero-padded file modes

This is caused by zero-padded file modes in the git repository. To see the warnings from git:

$ git clone git://github.com/evilsocket/hybris.git
$ cd hybris.git
$ git fsck --full
warning in tree 0a25d5f22d8677b38b14e3b3f8d6ad815782e0f8: contains zero-padded file modes
warning in tree 438c91c596ec31f0a4d42ff7d361db8c505ebd2a: contains zero-padded file modes
warning in tree 662f9059b67ba7f95565b694286ff336362890d1: contains zero-padded file modes
warning in tree a40b6cb8252533e3bf3fa9d19366db5bbb3b2cd7: contains zero-padded file modes
warning in tree c170762f3e4be04f5202ce37e4993ce334e361d8: contains zero-padded file modes
warning in tree ce6c5fd1b863da641cabaaaad4b1c16014126288: contains zero-padded file modes
warning in tree ea01d2b07b93a61f1cd9c56a9c0d8103d25914ae: contains zero-padded file modes

summary: - AssertionError: Invalid sha for <Tree
- 4a698785d734daf98f3aee3a3318f5b32d94de98>:
- 97f1ccb990d78f6b7a1c9aec69c367594c8fa56f
+ Support for zero-padded file modes
Changed in bzr-git:
importance: Medium → Low
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Apparently this is caused by a bug in github's "gist" feature, which causes it to write out incorrect data. c git silently ignores this error but dulwich and jgit both error out.

summary: - Support for zero-padded file modes
+ Cope with zero-padded file modes
Changed in dulwich:
importance: Undecided → Low
status: New → Triaged
Jelmer Vernooij (jelmer)
Changed in dulwich:
assignee: nobody → Augie Fackler (durin42)
milestone: none → 0.7.0
status: Triaged → Fix Committed
Jelmer Vernooij (jelmer)
Changed in dulwich:
status: Fix Committed → Fix Released
Jelmer Vernooij (jelmer)
Changed in bzr-git:
assignee: Jelmer Vernooij (jelmer) → nobody
Revision history for this message
Robert Collins (lifeless) wrote : Re: Cope with zero-padded file modes

@jelmer, what is needed in LP here?

Changed in launchpad:
status: Triaged → Incomplete
summary: - Cope with zero-padded file modes
+ zero-padded file modes break git->bzr imports
Revision history for this message
Jelmer Vernooij (jelmer) wrote : Re: [Bug 581064] Re: Cope with zero-padded file modes

On Sat, 2011-04-09 at 06:12 +0000, Robert Collins wrote:
> @jelmer, what is needed in LP here?
 lp:bzr-git needs to be merged into lp:~launchpad-pqm/bzr-git/devel and
sourcedeps.conf needs to be updated to use a new bzr-git revno.

This bug is mainly here since it affects launchpad code imports (that's
why it was originally reported) and to give users an ability to track
its progress in Launchpad.

Cheers,

Jelmer

Revision history for this message
Robert Collins (lifeless) wrote :

@jelmer is that merge doable now? Would you like to put it forward if it is?

Changed in launchpad:
status: Incomplete → Triaged
importance: Medium → High
tags: added: easy
Revision history for this message
xaav (xaav) wrote :

This is now affecting my project; linking question.

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

@lifeless The merge for the launchpad imports can't happen until the bug is fixed in bzr-git.

Revision history for this message
xaav (xaav) wrote :

Since this is really annoying, and since the fix really isn't that hard, I'm going to try to fix this.

Line: http://bazaar.launchpad.net/~bzr-git/bzr-git/trunk/view/head:/object_store.py#L146

Changed in bzr-git:
assignee: nobody → Geoff (geoffreyfishing)
xaav (xaav)
Changed in bzr-git:
status: Triaged → In Progress
Revision history for this message
xaav (xaav) wrote :

Sorry, the fix appears to be more complex than I thought it was going to be, and I'm not equipped to deal with that.

Changed in bzr-git:
assignee: Geoff (geoffreyfishing) → nobody
status: In Progress → Confirmed
Jelmer Vernooij (jelmer)
tags: removed: easy
Revision history for this message
Robert Collins (lifeless) wrote :

Marking low for LP, as it was suffering starvation from the low priority of the dependent tasks.

Changed in launchpad:
importance: High → Low
Curtis Hovey (sinzui)
tags: added: git
Jelmer Vernooij (jelmer)
Changed in brz-git:
status: New → Triaged
importance: Undecided → Low
Jelmer Vernooij (jelmer)
affects: brz-git → brz
Colin Watson (cjwatson)
affects: launchpad → lp-codeimport
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.