Bazaar

Large memory usage on upgrade

Bug #490770 reported by Rune Philosof
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
Undecided
Unassigned
Breezy
Triaged
Low
Unassigned

Bug Description

I ran upgrade (pack-0.92 -> 2a) on a repository and got this error:
rune@hilbert:~/biostat_local/bzr.repo$ bzr upgrade
starting upgrade of file:///home/rune/biostat_local/bzr.repo/
making backup of file:///home/rune/biostat_local/bzr.repo/.bzr
  to file:///home/rune/biostat_local/bzr.repo/backup.bzr
starting repository conversion
bzr: ERROR: Must end write group before releasing write lock on CHKInventoryRepository('file:///home/rune/biostat_local/bzr.repo/.bzr/repository/')

rune@hilbert:~/biostat_local/bzr.repo$ bzr version
Bazaar (bzr) 2.0.0
  Python interpreter: /usr/bin/python 2.6.4
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.31-14-generic-x86_64-with-Ubuntu-9.10-karmic
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib
  Bazaar configuration: /home/rune/.bazaar
  Bazaar log file: /home/rune/.bzr.log

Log:
fre 2009-11-27 13:45:07 +0100
0.032 bzr arguments: [u'upgrade']
0.043 looking for plugins in /home/rune/.bazaar/plugins
0.043 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.084 encoding stdout as sys.stdout encoding 'UTF-8'
3264.515 creating repository in file:///home/rune/biostat_local/bzr.repo/.bzr/.
3268.055 Resizing the inventory entry cache from 10240 to 30926
3269.572 Resizing the inventory entry cache from 30926 to 69320
3286.566 Resizing the inventory entry cache from 69320 to 100166
8125.502 checking remap as 15 deletions
8126.356 checking remap as 16 deletions
8144.451 Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3162, in run
    upgrade(url, format)
  File "/usr/lib/python2.6/dist-packages/bzrlib/upgrade.py", line 85, in upgrade
    Convert(url, format)
  File "/usr/lib/python2.6/dist-packages/bzrlib/upgrade.py", line 40, in __init__
    self.convert()
  File "/usr/lib/python2.6/dist-packages/bzrlib/upgrade.py", line 79, in convert
    self.bzrdir = converter.convert(self.bzrdir, self.pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 3023, in convert
    converter.convert(repo, pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4084, in convert
    converted.unlock()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 2355, in unlock
    % self)
BzrError: Must end write group before releasing write lock on CHKInventoryRepository('file:///home/rune/biostat_local/bzr.repo/.bzr/repository/')

I did a commit and some other commands in between the check and the upgrade, so I moved the 'backup.bzr' to '.bzr' and did the check after this error. The check did not report any problems.

rune@hilbert:~/biostat_local/bzr.repo$ bzr info
Shared repository (format: pack-0.92)
Location:
  shared repository: .

Maybe my system ran out of memory?
rune@hilbert:~/biostat_local/bzr.repo$ free -m
             total used free shared buffers cached
Mem: 7874 3456 4418 0 119 2417
-/+ buffers/cache: 920 6954
Swap: 8918 547 8371

rune@hilbert:~/biostat_local/bzr.repo$ du -sh .bzr
16G .bzr

Patrick Regan (patrick-regan)
Changed in bzr:
status: New → Confirmed
Revision history for this message
Andrew Bennetts (spiv) wrote :

Unfortunately the original error has been obscured by bug 429747, which has been fixed in trunk (and the 2.1 beta releases) but isn't fixed in 2.0. So it might be a memory error (especially considering the size of your repo), but it's hard to tell.

Another change in the 2.1 series is that we've cut memory consumption roughly in half for many operations, so if it is memory related that *might* help.

So, is it possible for you to try installing 2.1.0b3 and report what happens? (your traceback says you're using karmic, we have a PPA with the beta releases at https://launchpad.net/~bzr-beta-ppa/+archive.)

Revision history for this message
Rune Philosof (olberd) wrote :

at https://launchpad.net/~bzr-beta-ppa/+archive there is no version 2.1 listed

Revision history for this message
Rune Philosof (olberd) wrote :

I have just attempted to upgrade again.
rune@hilbert:~/data$ bzr upgrade
starting upgrade of file:///home/rune/data/
making backup of file:///home/rune/data/.bzr
  to file:///home/rune/data/backup.bzr
starting repository conversion
Doing on-the-fly conversion from <RepositoryFormatKnitPack1> to <RepositoryFormat2a>.
This may take some time. Upgrade the repositories to the same format for better performance.

Segmentation faultTransferring revisions:repacking texts:texts 63900/88212

Just before the segmentation fault I collected some information on memory consumption.
rune@host:~/data$ free -m
             total used free shared buffers cached
Mem: 7874 7826 47 0 0 96
-/+ buffers/cache: 7729 144
Swap: 8918 6129 2789

From top:
  PID USER PR NI VIRT RES SHR S
14714 rune 20 0 13.5g 7.1g 504 D

 %CPU %MEM TIME+ COMMAND
    2 92.3 105:00.67 bzr

So it is using swap heavily, resulting in a lot of waiting time for the cpu.
Earlier in the process it peaked at 4 different times with 10GB usage and then a drop.
Few computers have more than 8GB ram.
Of course, few also has such repository sizes to work on.
But nevertheless, I think you should work a bit more on having algorithms that only need to work on fragments of the data instead (if the problem is because it load everything into memory)

Revision history for this message
Rune Philosof (olberd) wrote :

rune@hilbert:~/test.bzr$ ulimit -v 5000000
rune@hilbert:~/test.bzr$ bzr branch ../data test5GB
Doing on-the-fly conversion from <RepositoryFormatKnitPack1> to <RepositoryFormat2a>.
This may take some time. Upgrade the repositories to the same format for better performance.

/ Transferring revisions 0/29

Then:
bzr: out of memory

Shouldn't this bug have importance "high" since bug #367545 has "high" importance.
Bug #367545 is about not being able to use branch/checkout/check commands on large pack-0.92 repositories.
The workaround suggested in bug #367545 is to upgrade. Which is stopped by this bug.

Rune Philosof (olberd)
summary: - Must end write group before releasing write lock on
- CHKInventoryRepository
+ Large memory usage on upgrade
Revision history for this message
Rune Philosof (olberd) wrote :

I just tried with a limit of 8GB memory and it reached almost 20000 in:
| Build phase:Adding file contents 20000/50172

I wanted to try again without restraining the memory, so I tried
rune@hilbert:~/test.bzr$ bzr branch ../data testallGB
And since test.bzr is a shared repository it resumed from the 20000/50172, but as far as I could see it did not try to allocate more than 8GB memory when resuming (total system usage was around 2GB), so maybe it would have been able to resume even with the 8GB restraint.
How come?

Can I expect a broken repository when resuming a broken branch this way?

Revision history for this message
Rune Philosof (olberd) wrote :

The process completed now.

Branched 29 revision(s).

And seems functional.
rune@hilbert:~/test.bzr$ bzr check
Checking working tree at '/home/rune/test.bzr/testallGB'.
Checking branch at 'file:///home/rune/test.bzr/testallGB/'.
Checking repository at 'file:///home/rune/test.bzr/'.
checked repository <bzrlib.transport.local.LocalTransport url=file:///home/rune/test.bzr/> format <RepositoryFormat2a>
    29 revisions
 50188 file-ids
checked branch file:///home/rune/test.bzr/testallGB/ format Branch format 6

Could something be missing because of the "bzr: out of memory" error, or would that get caught by the check?

Revision history for this message
Neil Martinsen-Burrell (nmb) wrote : Re: [Bug 490770] Re: Large memory usage on upgrade

On 2010-04-06 09:24 , Olberd wrote:
> The process completed now.
>
> Branched 29 revision(s).
>
> And seems functional.
> rune@hilbert:~/test.bzr$ bzr check
> Checking working tree at '/home/rune/test.bzr/testallGB'.
> Checking branch at 'file:///home/rune/test.bzr/testallGB/'.
> Checking repository at 'file:///home/rune/test.bzr/'.
> checked repository <bzrlib.transport.local.LocalTransport url=file:///home/rune/test.bzr/> format <RepositoryFormat2a>
> 29 revisions
> 50188 file-ids
> checked branch file:///home/rune/test.bzr/testallGB/ format Branch format 6
>
> Could something be missing because of the "bzr: out of memory" error, or
> would that get caught by the check?

That is precisely the sort of thing that check checks. The above looks
like exactly right for the results of ``bzr check``.

Revision history for this message
Rune Philosof (olberd) wrote :

I just checked the info in comment #5.
I am able to branch this into the new format with a restriction on memory throughout the process.
$ ulimit -v 8000000
$ bzr init-repository test
$ cd test
test$ bzr branch ../data try1
bzr: out of memory
test$ bzr branch ../data try2

So it is obvious that some memory is locked that is not needed.

Revision history for this message
Roel Van de Paar (roel11) wrote :

ref bug#191731 and bug#367545

Jelmer Vernooij (jelmer)
tags: added: check-for-breezy
Jelmer Vernooij (jelmer)
Changed in brz:
status: New → Triaged
importance: Undecided → Medium
importance: Medium → Low
tags: added: memory perforamnce upgrade
removed: check-for-breezy
tags: added: performance
removed: perforamnce
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.