Support lightweight checkouts

Bug #500483 reported by Paul Natsuo Kishimoto
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
bzr-builder
Triaged
Low
Unassigned

Bug Description

I am using bzr-builder to package GnuCash unstable for my own use. Initially I used a branch I had created with only one revision—the 72 MB source from the 12.3 MB tarball. I discovered that there was an automatic bzr import of the GnuCash svn trunk and wanted to take advantage of the fresher code for my builds. I am now using the following recipe:

# bzr-builder format 0.2 deb-version 2.3.8+r{revno}~ppa{revno:packaging}
lp:gnucash
nest packaging lp:~khaeru/+junk/gnucash-2.3-packaging debian

"bzr dailydeb" is now pulling more than 200 MB of data each time it runs—over 16 times the size of the source tarball!

On the other hand, the following command involves about a fifth (<40 MB) as much network traffic and runs much, much more quickly:

bzr checkout --lightweight lp:gnucash

Since it seems that the revision history is unnecessary baggage, especially when a recipe involves no merging, I would like to add a wishlist item for a way to specify lightweight checkouts in bzr-builder recipe files.

description: updated
Revision history for this message
Paul Natsuo Kishimoto (khaeru) wrote :

Also, this recipe is currently failing with the output:

Building the source package
bzr: ERROR: [Errno 2] No such file or directory

...which is very opaque :( Adding --verbose to the command-line doesn't actually produce more output.

Revision history for this message
James Westby (james-w) wrote : Re: [Bug 500483] Re: Support lightweight checkouts

On Wed, 30 Dec 2009 20:02:57 -0000, Paul Kishimoto <email address hidden> wrote:
> Also, this recipe is currently failing with the output:
>
> Building the source package
> bzr: ERROR: [Errno 2] No such file or directory
>
> ...which is very opaque :( Adding --verbose to the command-line doesn't
> actually produce more output.

Could you run with -Derror to get a full backtrace?

Thanks,

James

Revision history for this message
Paul Natsuo Kishimoto (khaeru) wrote :

Here you go. I omitted four GPG passphrase prompts.

Building the source package
bzr: ERROR: exceptions.OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/builder/__init__.py", line 292, in run
    self._build_source_package(package_dir)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/builder/__init__.py", line 350, in _build_source_package
    stdin=subprocess.PIPE)
  File "/usr/lib/python2.6/subprocess.py", line 621, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1126, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Revision history for this message
James Westby (james-w) wrote :

On Tue, 05 Jan 2010 17:59:36 -0000, Paul Kishimoto <email address hidden> wrote:
> Here you go. I omitted four GPG passphrase prompts.
>
> Building the source package
> bzr: ERROR: exceptions.OSError: [Errno 2] No such file or directory
>
> Traceback (most recent call last):
> File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
> return the_callable(*args, **kwargs)
> File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
> ret = run(*run_argv)
> File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/builder/__init__.py", line 292, in run
> self._build_source_package(package_dir)
> File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/builder/__init__.py", line 350, in _build_source_package
> stdin=subprocess.PIPE)
> File "/usr/lib/python2.6/subprocess.py", line 621, in __init__
> errread, errwrite)
> File "/usr/lib/python2.6/subprocess.py", line 1126, in _execute_child
> raise child_exception
> OSError: [Errno 2] No such file or directory

This is the obscure message you get from python when you run something
with subprocess that isn't installed.

There's a bug that we don't depend on everything we should as well.

I think if you install "devscripts" this will probably go away.

Thanks,

James

Revision history for this message
Paul Natsuo Kishimoto (khaeru) wrote :

As it wound up, I had to install dpatch as well, but after that things went swimmingly. Thanks for the tip! It appears that I was bit before by https://bugs.launchpad.net/bzr-builder/+bug/428722, since I was the one who marked it confirmed—two months ago. Long term memory failure :/

The recipe I am running is now:

$ cat gnucash.recipe
# bzr-builder format 0.2 deb-version 2.3.8+{revno}~ppa{revno:packaging}
lp:gnucash
nest packaging lp:~khaeru/+junk/gnucash-2.3-packaging debian
run ./autogen.sh

...and it's still pulling way too much data. As I said earlier, this should have 'Wishlist' importance.

Revision history for this message
James Westby (james-w) wrote :

Hi,

I don't see a problem with adding this. It may be that it turns out not to be much
faster, depending on the operations it ends up doing remotely, as merging will require
pulling down extra data as well.

The best way to find out is to do it though.

To add this you would need to add:

  * An Option to the commands to specify "checkout".
  * checkout=None added to both run() methods in __init__.py.
  * This variable passed down to build_tree()
  * The changes in build_tree() to make checkouts not branches (see cmd_checkout
     in bzrlib/builtins.py in the bzr source for inspriration).
  * Tests, and I would be happy to help work out how to write those.

Thanks,

James

Changed in bzr-builder:
status: New → Triaged
importance: Undecided → Low
Changed in bzr-builder:
assignee: nobody → Paul Kishimoto (khaeru)
Revision history for this message
jasherai (phatforge) wrote :

I would say the first step would be to create/use a local copy of the branch referenced by the recipe. On subsequent runs this copy can be updated and then copied (hard linked?) to the build dir for the actual build process.

The lightweight option makes a lot of sense but only for non-merge recipes. (This would actually be really helpful for me too! )

Opinions?

Changed in bzr-builder:
assignee: Paul Kishimoto (khaeru) → nobody
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Have you tried running the recipe in a shared repository? That should prevent most of the traffic.

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

In other words, specify a build location that is under a shared repository:

bzr init-repo /tmp/gnucash
bzr dailydeb gnucash.recipe /tmp/gnucash

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.