MemoryError in pack.bytes_record during fetch of enormous repository

Bug #336890 reported by GiuseppeVerde
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
Low
Unassigned
Breezy
Triaged
Low
Unassigned
bzr (Ubuntu)
Triaged
Low
Unassigned

Bug Description

Binary package hint: bzr

When using sftp to check out my ginormous repository of PDFs (physics articles I need to keep synchronized between desktops, netbook, and n810), bzr eats all of my available RAM (gets up to 1.4-1.5GB) and then, of course, runs out of RAM and gets killed:

$ bzr branch sftp://<workstation>./home/<user>/Documents
/usr/lib/python2.6/dist-packages/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  from sha import *
/usr/lib/python2.6/dist-packages/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  from md5 import *
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 896, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 542, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 1045, in run
    source_branch=br_from)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 1116, in sprout
    result_repo.fetch(source_repository, revision_id=revision_id)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1118, in fetch
    find_ghosts=find_ghosts)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 2985, in fetch
    return self._pack(self.source, self.target, revision_ids)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 2992, in _pack
    revision_ids).pack()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 655, in pack
    return self._create_pack_from_packs()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 809, in _create_pack_from_packs
    self._copy_text_texts()
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 786, in _copy_text_texts
    self.new_pack.text_index, readv_group_iter, total_items))
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 900, in _copy_nodes_graph
    write_index, output_lines, pb, readv_group_iter, total_items):
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 944, in _do_copy_nodes_graph
    pos, size = writer.add_bytes_record(raw_data, names)
  File "/usr/lib/python2.6/dist-packages/bzrlib/pack.py", line 154, in add_bytes_record
    serialised_record = self._serialiser.bytes_record(bytes, names)
  File "/usr/lib/python2.6/dist-packages/bzrlib/pack.py", line 105, in bytes_record
    return ''.join(byte_sections)
MemoryError

bzr 1.12 on python 2.6.1 (linux2)
arguments: ['/usr/bin/bzr', 'branch', 'sftp://<workstation>/home/<user>/Documents']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  avahi /usr/lib/python2.6/dist-packages/bzrlib/plugins/avahi [0.3dev]
  builddeb /usr/lib/python2.6/dist-packages/bzrlib/plugins/builddeb [2.1dev]
  bzrtools /usr/lib/python2.6/dist-packages/bzrlib/plugins/bzrtools [1.12]
  dbus /usr/lib/python2.6/dist-packages/bzrlib/plugins/dbus [unknown]
  gtk /usr/lib/python2.6/dist-packages/bzrlib/plugins/gtk [0.95.0.final.1]
  launchpad /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [unknown]
  loom /usr/lib/python2.6/dist-packages/bzrlib/plugins/loom [1.4dev]
  netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [unknown]
  rebase /usr/lib/python2.6/dist-packages/bzrlib/plugins/rebase [0.4.2]
  search /usr/lib/python2.6/dist-packages/bzrlib/plugins/search [1.7dev]
  stats /usr/lib/python2.6/dist-packages/bzrlib/plugins/stats [unknown]
  svn /usr/lib/python2.6/dist-packages/bzrlib/plugins/svn [0.5]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

This occurred while it was "Adding file contents"

This occurs both when I do a lightweight checkout (bzr checkout --lightweight) and when doing a branch (bzr branch). I'd much prefer the lightweight checkout; then I only need rev history in one location and don't have to duplicate the files all over.

Tags: memory
Revision history for this message
GiuseppeVerde (launchpad-digitasaru) wrote :
Revision history for this message
GiuseppeVerde (launchpad-digitasaru) wrote :

Oh, I should mention that this is with jaunty, up to date as of mid morning today (CST)

Martin Pool (mbp)
summary: - bzr branch, co of a ginormous repository of PDFs eats all available ram
- and then gets killed
+ MemoryError during fetch of enormous repository
Martin Pool (mbp)
summary: - MemoryError during fetch of enormous repository
+ MemoryError in pack.bytes_record during fetch of enormous repository
Changed in bzr:
importance: Undecided → Low
status: New → Confirmed
Jelmer Vernooij (jelmer)
Changed in bzr (Ubuntu):
status: New → Confirmed
Jelmer Vernooij (jelmer)
tags: added: memory
Jelmer Vernooij (jelmer)
Changed in bzr (Ubuntu):
importance: Undecided → Low
status: Confirmed → Triaged
Jelmer Vernooij (jelmer)
tags: added: check-for-breezy
Jelmer Vernooij (jelmer)
tags: removed: check-for-breezy
Changed in brz:
status: New → Triaged
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.