Importing revisions with submodules is not supported

This is blocked by by-reference nested tree support in bzrlib.

I think I'm affected by the same problem. When trying to branch pitivi:

jeff@testbed:~$ bzr branch git://git.pitivi.org/git/pitivi.git
Escaped 1 XML-invalid characters in ce745af077aeda4f8d22cae84d5b1f0baf8eb1b7. Will be unable to regenerate the SHA map.
bzr: ERROR: exceptions.NotImplementedError: <function import_git_submodule at 0xa803f7c>

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 835, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1030, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 647, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 1245, in run
    source_branch=br_from)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 1177, in sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1544, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 378, in fetch
    mapping=mapping, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 403, in fetch_refs
    self.fetch_objects(determine_wants, mapping, pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 437, in fetch_objects
    store, recorded_wants, pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 335, in import_git_objects
    parent_invs, target_git_object_retriever._idmap, lookup_object)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 246, in import_git_tree
    file_id, revision_id, parent_invs, shagitmap, lookup_object)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 174, in import_git_submodule
    raise NotImplementedError(import_git_submodule)
NotImplementedError: <function import_git_submodule at 0xa803f7c>

bzr 1.17 on python 2.6.2 (linux2)
arguments: ['/usr/bin/bzr', 'branch', 'git://git.pitivi.org/git/pitivi.git', 'pitivi-bazaar']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'fr_CA.UTF-8'
plugins:
  bzrtools /usr/lib/python2.6/dist-packages/bzrlib/plugins/bzrtools [1.17]
  dbus /usr/lib/python2.6/dist-packages/bzrlib/plugins/dbus [0.1dev]
  git /usr/lib/python2.6/dist-packages/bzrlib/plugins/git [0.4]
  gtk /usr/lib/python2.6/dist-packages/bzrlib/plugins/gtk [0.96.2.final]
  launchpad /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [1.17]
  netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [1.17]
  svn /usr/lib/python2.6/dist-packages/bzrlib/plugins/svn [0.6.3]

This is probably what is affecting https://code.launchpad.net/~vcs-imports/pitivi/trunk-git too, I presume.

A proper error message is now printed.

Launchpad could automatically recognize and perhaps better format this error by explicitly catching objects of the class bzrlib.plugins.git.fetch.SubmodulesNotSupported

This is blocked by nested tree support in Bazaar. I've marked it as affecting Bazaar, but I'm not sure if that's the right thing to do. I couldn't find any existing bug reports in Bazaar about missing nested tree support.

Of course, this doesn't mean you can use submodules now - bzr-git can import them and converts them to nested trees, but nested tree support in bzr itself is still incomplete.

The git import of xmms2 [1-2] failed due to missing submodules support:

bzrlib.plugins.git.fetch.SubmodulesNotSupported: Submodules can not yet be imported (requires nested tree support in Bazaar).

[1] https://code.launchpad.net/~vcs-imports/xmms2/stable
[2] https://code.launchpad.net/~vcs-imports/xmms2/trunk

A version of bzr-git that supports submodules should land on launchpad within the next two weeks.

Launchpad import still fails:

2010-01-09 12:30:49 INFO 51 bytes transferred |
2010-01-09 12:30:50 INFO Counting objects: 5920, done. 0
2010-01-09 12:30:52 INFO finding revisions to fetch:generating index 0/5920
2010-01-09 12:30:53 INFO finding revisions to fetch 1/1808
2010-01-09 12:31:18 INFO
Traceback (most recent call last):
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/scripts/code-import-worker.py", line 63, in <module>
    script.main()
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/scripts/code-import-worker.py", line 58, in main
    import_worker.run()
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/lib/lp/codehosting/codeimport/worker.py", line 409, in run
    self._doImport()
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/lib/lp/codehosting/codeimport/worker.py", line 526, in _doImport
    bazaar_tree.branch.pull(foreign_branch, overwrite=True)
  File "<string>", line 4, in pull_write_locked
  File "/home/pqm/for_rollouts/production/eggs/bzr-2.1.0b3-py2.5-linux-x86_64.egg/bzrlib/branch.py", line 949, in pull
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/branch.py", line 416, in pull
    graph=graph)
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/branch.py", line 379, in update_revisions
    interrepo.fetch_objects(determine_wants, self.source.mapping)
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/fetch.py", line 493, in fetch_objects
    store, recorded_wants, pb)
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/fetch.py", line 363, in import_git_objects
    allow_submodules=getattr(repo._format, "supports_tree_reference", False))
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/fetch.py", line 261, in import_git_tree
    raise SubmodulesRequireSubtrees()
bzrlib.plugins.git.fetch.SubmodulesRequireSubtrees: The repository you are fetching from contains submodules. To continue, upgrade your Bazaar repository to a format that supports nested trees, such as 'development-subtree'.
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended with exit code 1.

Launchpad fails for me too, same error. I no longer have submodules in the repository, but there was a time in the past when there were submodules. The branch that's failing is "lp:astrid"

This is fairly important to me - more important than 'Wishlist'. We
can't import GDB or QEMU into Launchpad without this support.

Also affects pitivi and gstreamer.

Any update on this? This is really holding us back enabling daily builds.

Sorry, I should have been more clear, this is holding back our ability to evangelize daily builds to existing upstreams who would be interested in using dailies and PPAs.

For a more complete list of failing imports because of this issue, see https://dev.launchpad.net/FailingBzrSvnImports

Sorry, of course I mean https://dev.launchpad.net/FailingBzrGitImports

And also affects banshee

Are there any concrete plans to add this to bzr stable in some specific future version? If not, how about making this higher priority, as it seems to render a valuable bzr feature only half workable.

affects emesene

Affects qutim

Has been affecting Mumble for a long time now. Even being able to import without the submodules would be better than being completely unable to do so.

Any ETA on releasing a fix for this bug?

Affects tomahawk too

affects hasciicam and more software

I updated the title because it was misleading.

@Geoff this bug is for the specific git issue.

Added affected software to description :)

Bug #769629 is not a duplicate then. Can someone please un-mark it as a duplicate.

how do you fix this problem E:: command not found
root@kpulaifer-desktop:~# Go to the repository dialog to correct the problem.
Go: command not found
root@kpulaifer-desktop:~# E: _cache->open() failed, please repo
bash: syntax error near unexpected token `('
root@kpulaifer-desktop:~# /etc/apt/sources.list.d/ferramroberto
bash: /etc/apt/sources.list.d/ferramroberto: No such file or directory
root@kpulaifer-desktop:~# sudo remove /etc/apt/sources.list.d/ferramroberto
sudo: remove: command not found
root@kpulaifer-desktop:~# /etc/apt/sources.list.d/ferramroberto
bash: /etc/apt/sources.list.d/ferramroberto: No such file or directory
root@kpulaifer-desktop:~# sudo/etc/apt/sources.list.d/ferramroberto
bash: sudo/etc/apt/sources.list.d/ferramroberto: No such file or directory
root@kpulaifer-desktop:~# /etc/apt/sources.list.d/ferramroberto-
bash: /etc/apt/sources.list.d/ferramroberto-: No such file or directory
root@kpulaifer-desktop:~# sudo 'b-src
> sudo deb-src
> sudo E:Type 'b-src' is not known on line 1 in source list /etc/apt/sources.list.d/ferramroberto-java-natty.list

Hi Karen,

That's a different problem; please go to
<http://askubuntu.com/questions/ask> and describe what it is you want
to accomplish, what you're doing, and what goes wrong, and someone
will help you soon.

Martin

This bug is impacting our work with CloudFoundry. Any likelihood of it being worked-on or solved in the near future?

Thanks,
Dustin

This bug is affecting:

lp:zile
lp:dico
lp:gnu-cssc
lp:mailutils
lp:diffutils

as a workaround, i used `git filter-branch` to remove the submodules from the project -- of course, this only works if the submodules are purely optional or otherwise not relevant to the purposes intended (in my case, i need the branch to build daily packages; upstreams might keep library dependencies in a subtree to build and ship them, while in debian/ubuntu, this is patched out anyway to use package dependencies).

the very line i used for filtering was

  git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch .gitmodules libraries/MCAD'

(libraries/MCAD is the only submodule i had). for reference, this affects the lp:openscad project.

chrysn, this workaround doesn't help, because a machine running this command is needed.

These projects/branches are also affected by this bug:

lp:anarchnet
lp:anubis
lp:autoconf-archive
lp:bison
lp:cflow
lp:coreutils
lp:cpio (lp:~rbose-vcs-imports/cpio/master)
lp:findutils
lp:gnu-radius
lp:grep (lp:~rbose-vcs-imports/grep/master)
lp:mumble
lp:patch
lp:sed

The GDB project you can remove from the list, the branch is fixed, as you can see here: lp:gdb

Also affected by this bug:

lp:gzip
lp:parted

sorry, lp:parted does not exist, i mean: lp:~rbose-vcs-imports/parted/master

Hi DNS,

We appreciate this bug is important and is blocking an increasing
number of projects. We have enough test data, so adding a list of
every affected branch is not especially going to help.

Ok :-)

Then can we expect some work to be done, Martin? It's more than a year now and many are affected. Something has to be done.

Prevents Qt5 base module from being imported from gitorious.. isn't that reason enough to fix this pronto.

What does the change of the milestone to "none" mean? Does it mean that the bug will never be fixed? That'd be a pity.

@Eli: No, it means whatever was in progress was not finished when 2.6b1 was frozen, nothing more.

2012-05-19 11:17:49 INFO Unable to import branch because of limitations in Bazaar.
2012-05-19 11:17:49 INFO The repository you are fetching from contains submodules. To continue, upgrade your Bazaar repository to a format that supports nested trees, such as 'development-subtree'.

We're no longer using git submodules, is there a way to tell bzr/vcs-imports to just ignore history and focus on current code?
We'd like to setup a daily ppa for our software without using anything else than github and launchpad.

Thanks

Hi c1oud,

There is no way to tell the import system to just ignore history. You could provide a custom repository that has the submodules filtered out of the history or filter them out of your history on github.

I'm not sure why this is marked as affecting cloudfoundry? I don't see what changes would be needed in cloudfoundry for this to be fixed.

CloudFoundry uses github. Getting sub modules supported would allow the
package to be built directly from their repository.

Thanks,

Juan

Sent from my mobile.
Apologies for the brevity of this message.
On Jun 28, 2012 6:05 AM, "Jelmer Vernooij" <email address hidden>
wrote:

> I'm not sure why this is marked as affecting cloudfoundry? I don't see
> what changes would be needed in cloudfoundry for this to be fixed.
>
> --
> You received this bug notification because you are a member of
> Canonical-SIG, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/402814
>
> Title:
> Importing revisions with submodules is not supported
>
> Status in Bazaar Version Control System:
> Triaged
> Status in bzr git support plugin:
> Fix Released
> Status in CloudFoundry:
> Confirmed
> Status in Launchpad itself:
> Triaged
>
> Bug description:
> bzr-git converts submodules into nested trees, but nested trees aren't
> very well supported by bzr yet.
>
> Affected projects:
>
> lp:anarchnet
> lp:anubis
> astrid
> lp:autoconf-archive
> banshee
> lp:bison
> lp:cflow
> lp:coreutils
> lp:cpio (lp:~rbose-vcs-imports/cpio/master)
> lp:dico
> lp:diffutils
> emesene
> lp:findutils
> f-spot
> lp:gnu-cssc
> lp:gnu-radius
> lp:grep (lp:~rbose-vcs-imports/grep/master)
> gstreamer
> haasciicam
> lp:mailutils
> lp:mumble
> lp:patch
> pitivi
> QEMU
> qutim
> lp:sed
> tomahawk
> lp:vlc
> lp:xmms2
> lp:zile
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/bzr/+bug/402814/+subscriptions
>

hmmm, many of the affected packages seems to be way old in recent ubuntu releases due to this bug, e.g. coreutils.

On Thu, 2012-10-04 at 19:43 +0000, Jarl wrote:
> hmmm, many of the affected packages seems to be way old in recent ubuntu
> releases due to this bug, e.g. coreutils.
This shouldn't have any effect on what version of these packages is in
Ubuntu. New versions can be packaged without working VCS imports

Cheers,

Jelmer

How can we workaround this bug?

I see vlc team is workarounding it [1], but I don't see how

[1] https://code.launchpad.net/~videolan/vlc/master-manual

LocutusOfBorg, I myself created cron script (for lp:qutim) with the following contents:
#!/bin/bash
cd /path/to/qutim-replica
git pull origin master
git submodule update --init --recursive
bzr stat -S|grep ^?|sed 's/^? //'| while read LINE; do bzr add "$LINE"; done
bzr ci -m "Cron update"
bzr push

You also have to add ..git* to your .bzrignore.

I wrote a small hacky script [1] and run it every 15 minutes via Cron.

[1] https://bazaar.launchpad.net/~videolan/vlc/manual-bzr-import/view/head:/manual-bzr-import

So... any ETA on the fix for the bug?

On Tue, Apr 16, 2013 at 01:16:25AM -0000, ?????????? ???????????????????? wrote:
> So... any ETA on the fix for the bug?
Nobody is actively working on this bug at the moment, as far as I am aware.

> CloudFoundry uses github. Getting sub modules supported would allow the
> package to be built directly from their repository.
Fixing this doesn't require any changes to cloudfoundry though, which is what that field is about (I agree the naming is a bit confusing).

Also, with a github repo with no submodules at all, LP import fails complaining of submodules:
https://code.launchpad.net/~dnscrypt-proxy/dnscrypt-proxy/master

pataquets: That's because there are submodules at some point in the commit history, which will trigger the issue as well.

arand: So you're stating that removing submodules isn't a workaround? Anyway it would be extremely beneficial to getting things running if submodules could be skipped as an option for imports, as in many cases any recipes may not even need to care about submodules.

hexafraction:
The only "workaround" (as far as I can tell, provided the way I guess the importer currently works) would be to go through the whole repository history and replace all commits that contained submodules with commits that didn't, and then run the importer on that.

Thanks. Unfortunately for two repositories I use their design makes this untenable.

The only workaround I found was to install bazaar WITHOUT the git plugin, bzr init inside the existing repo, fix up ignores, and then just push that. The only pitfall is an extra modification to the git repo's state.

I read about workarounds here, but I don't know how to apply them to lp:openra The web-interface seems very limited in that regard. Is there another way or did some post a step-by-step guide somewhere. Not that the recent commits don't use git submodules anymore. It was many years ago, but still bzr complains because somewhere in the history git submodules have been detected. :(

On Sun, Dec 08, 2013 at 10:27:11PM -0000, Matthias Mailänder wrote:
> I read about workarounds here, but I don't know how to apply them to
> lp:openra The web-interface seems very limited in that regard. Is there
> another way or did some post a step-by-step guide somewhere. Not that
> the recent commits don't use git submodules anymore. It was many years
> ago, but still bzr complains because somewhere in the history git
> submodules have been detected. :(

The workarounds require you to do the import yourself and then
push the result bzr branch up to Launchpad; Launchpad can't
do them for you.

Is there any progress on this issue?

On Thu, Oct 09, 2014 at 08:10:53PM -0000, Sergey Belov wrote:
> Is there any progress on this issue?
Nobody is currently working on this issue.

As a workaround we could use github.com svn gateway, but it is not working.
Don't know why:

Example: http://launchpadlibrarian.net/186966468/saltstack-salt-develop.log

svn co https://github.com/saltstack/salt/trunk

It works on my laptop.

On Fri, Oct 10, 2014 at 05:08:33AM -0000, Sergey Belov wrote:
> As a workaround we could use github.com svn gateway, but it is not working.
> Don't know why:
>
> Example: http://launchpadlibrarian.net/186966468/saltstack-salt-
> develop.log
>
> svn co https://github.com/saltstack/salt/trunk
>
> It works on my laptop.

Launchpad imports don't just do a checkout, they import the full
history.

GitHubs SVN protocol implementation appears to be behave differently
from "regular" SVN servers. E.g. visiting
https://github.com/saltstack/salt/trunk gives me a 404.

Github SVN url don't work because it uses the User-agent string to determine whether to serve SVN protocol... And bzr-svn (unlike the official svn client) does not set the User-agent string.

I have requested to Github Support to provide a solution. Because bzr-svn is EOL (for several years now). And it can't be improved / updated anymore.

Even if bzr-svn wasn't unmaintained, it would be wrong for us to change the user agent. Servers should not have different behaviour based on user agent.

On 24 April 2015 5:44:29 pm BST, dreamcat4 <email address hidden> wrote:
>Github SVN url don't work because it uses the User-agent string to
>determine whether to serve SVN protocol... And bzr-svn (unlike the
>official svn client) does not set the User-agent string.
>
>I have requested to Github Support to provide a solution. Because bzr-
>svn is EOL (for several years now). And it can't be improved / updated
>anymore.
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/402814
>
>Title:
> Importing revisions with submodules is not supported
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/bzr/+bug/402814/+subscriptions

Even if bzr-svn wasn't EOL, it would be wrong for us to change the user agent. Servers should not have different behaviour based on user agent.

On 24 April 2015 5:44:29 pm BST, dreamcat4 <email address hidden> wrote:
>Github SVN url don't work because it uses the User-agent string to
>determine whether to serve SVN protocol... And bzr-svn (unlike the
>official svn client) does not set the User-agent string.
>
>I have requested to Github Support to provide a solution. Because bzr-
>svn is EOL (for several years now). And it can't be improved / updated
>anymore.
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/402814
>
>Title:
> Importing revisions with submodules is not supported
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/bzr/+bug/402814/+subscriptions

The bug is also preventing Numix Circle daily builds. Someone accidentally submitted a submodule and then it was removed, but it was many many commits later before we realised that our Launchpad builds had broken. It's now such a tangled mess that we can't really unpick the error.

A fix for this would be much appreciated.

Joshua,

Listen to me: you MUST rewind the git repo(s) for your official priject back to a point before the submodule commit. It's your only hope to recover from this situation on Launchpad.net. Then carefully re-apply any commits since then cleanly. If it's not too late, that's your least - cost option.

You can try messing around with git-->svn-->bzr import but that is simply flat out not gonna work if you're source repo is on github.com. (due to other bug in bzr / launchpad)

The entire bzr toolset is now defunct, including the bzr-git helper tool. So those bugs are never getting fixed / going away. The launchpad team are in a transition period right now, and it's a priority for them to get proper git hosting going (including native git-git imports). But they don't have any solid timeframe by which you can reasonably expect that to become available.

I say that coming from another project, where we already investigated every other possible workaround, including importing via git->svn->bzr, and asking github to help us work around the blocking launchpad bugs / issues. It was all dead ends. Eventually I just had to set up a brand new build system on dockerhub (tvheadend project). With a 'maybe when' can return back to launchpad.net.

For reference: our project the tvheadend builds had been broken on launchpad for about 2 years or something. Way before I got involved with it. So it was already far too late to be rewinding / picking through the commits.

Launchpad beta now supports native git. Although, I think, it can not be used for ppa build.

Bump on this issue. I hope launchpad supports native git submodules in the project.

Perhaps adding a simple line `git submodule update --init --recursive` solves the problem. Why is this issue not fixed yet? I think the fix is fairly easy.

On Wed, Aug 31, 2016 at 04:54:20AM -0000, Synapse Developer Team wrote:
> Perhaps adding a simple line `git submodule update --init --recursive`
> solves the problem. Why is this issue not fixed yet? I think the fix is
> fairly easy.

This bug is about importing revisions with submodules from git into bazaar.
Support for submodules in git->git imported repositories is a separate (Launchpad) issue.

We accidentally used a git submodule which was removed later on but against now stuck with this bug as the import failed for the last couple of weeks. Is there any ETA on this bug? I created a git import which worked, but then I needed nest-part from another project which was bzr-hosted so that didn't work and I couldn't build my recipe.

On Sun, Oct 02, 2016 at 01:09:14PM -0000, Jasem Mutlaq wrote:
> We accidentally used a git submodule which was removed later on but
> against now stuck with this bug as the import failed for the last couple
> of weeks. Is there any ETA on this bug? I created a git import which
> worked, but then I needed nest-part from another project which was bzr-
> hosted so that didn't work and I couldn't build my recipe.

Nobody is working on this bug at the moment, and as far as I am aware
there are no plans to work on it in the near future.

Jelmer

Your best bet is probably to use the git imports, and use daily builds
from git.

@jelmer: How to use that git imports? Can launchpad import/mirror git repository foreign git repository?

On Sun, Oct 23, 2016 at 10:09:45AM -0000, Pali wrote:
> @jelmer: How to use that git imports? Can launchpad import/mirror git
> repository foreign git repository?
The launchpad folks should be able to help you with that; I'm just
involved in bzr-git/bzr these days.

Launchpad git-to-git imports are nearly done; see bug 1469459 if you want to follow that work.

Launchpad git-to-git imports are available now: http://blog.launchpad.net/code/git-to-git-imports

I can confirm that using git-to-git import works as expected. We annoyingly had old submodules usage in the git history that prevented importing before now. Big thanks Colin! :)

Cloning git repositories with submodules now works in Breezy, and it's possible to pull Git branches with submodules into Breezy branches.

Working tree updates don't work yet - this requires changes to tree transform.

The support for nested trees in transform is being tracked in bug 267770.