Ubuntu
bzr package

failure to extract content > 2^31 bytes (OverflowError: signed integer is greater than maximum)

Bug #531612 reported by Adam D. Horden on 2010-03-03

This bug affects 3 people

	Status	Importance	Assigned to
Bazaar	Confirmed	Low	Unassigned
Breezy	Triaged	Low	Unassigned
bzr (Ubuntu)	Confirmed	Low	Unassigned

Bug Description

Binary package hint: bzr

When trying to commit a 3.8 gig database dump into revision control I get ERROR: exceptions.OverflowError: signed integer is greater than maximum. The system has more than enough memory, around 8 gig. The crash report is here http://dev.biological-hazard.net/~adhorden/bzr-20100303195926-10714.crash.

ProblemType: Bug
Architecture: amd64
Date: Wed Mar 3 20:06:01 2010
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: nvidia
Package: bzr 2.0.2-0ubuntu1
ProcEnviron:
PATH=(custom, no user)
LANG=en_GB.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-19.56-server
SourcePackage: bzr
Uname: Linux 2.6.31-19-server x86_64

Tags:

Revision history for this message

Adam D. Horden (adhorden) wrote on 2010-03-03:

Dependencies.txt Edit (919 bytes, text/plain; charset="utf-8")
XsessionErrors.txt Edit (11.8 KiB, text/plain; charset="utf-8")

Revision history for this message

John A Meinel (jameinel) wrote on 2010-03-03:

Download full text (4.3 KiB)

Including the traceback:
PythonVersion: 2.6.4
Traceback:
Traceback (most recent call last):
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
     return the_callable(*args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3058, in run
     exclude=safe_relpath_files(tree, exclude))
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 197, in commit
     result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/mutabletree.py", line 229, in commit
     *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 393, in commit
     self.branch.repository, new_revno, self.rev_id)
   File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 909, in import_last_revision_info
     self.repository.fetch(source_repo, revision_id=revid)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1695, in fetch
     find_ghosts=find_ghosts, fetch_spec=fetch_spec)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 3413, in fetch
     pb=pb, find_ghosts=find_ghosts)
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 81, in __init__
     self.__fetch()
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 107, in __fetch
     self._fetch_everything_for_search(search)
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 135, in _fetch_everything_for_search
     stream, from_format, [])
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4238, in insert_stream
     return self._locked_insert_stream(stream, src_format, is_resume)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4271, in _locked_insert_stream
     self.target_repo.texts.insert_record_stream(substream)
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1608, in insert_record_stream
     for _ in self._insert_record_stream(stream, random_id=False):
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1714, in _insert_record_stream
     bytes = record.get_bytes_as('fulltext')
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 454, in get_bytes_as
     self._manager._prepare_for_extract()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 554, in _prepare_for_extract
     self._block._ensure_content(self._last_byte)
   File "/usr/lib/pyt...

Including the traceback:
PythonVersion: 2.6.4
Traceback:
 Traceback (most recent call last):
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
     return the_callable(*args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3058, in run
     exclude=safe_relpath_files(tree, exclude))
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/workingtree_4.py", line 197, in commit
     result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/mutabletree.py", line 229, in commit
     *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/commit.py", line 393, in commit
     self.branch.repository, new_revno, self.rev_id)
   File "/usr/lib/python2.6/dist-packages/bzrlib/branch.py", line 909, in import_last_revision_info
     self.repository.fetch(source_repo, revision_id=revid)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1695, in fetch
     find_ghosts=find_ghosts, fetch_spec=fetch_spec)
   File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
     result = unbound(self, *args, **kwargs)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 3413, in fetch
     pb=pb, find_ghosts=find_ghosts)
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 81, in __init__
     self.__fetch()
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 107, in __fetch
     self._fetch_everything_for_search(search)
   File "/usr/lib/python2.6/dist-packages/bzrlib/fetch.py", line 135, in _fetch_everything_for_search
     stream, from_format, [])
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4238, in insert_stream
     return self._locked_insert_stream(stream, src_format, is_resume)
   File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 4271, in _locked_insert_stream
     self.target_repo.texts.insert_record_stream(substream)
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1608, in insert_record_stream
     for _ in self._insert_record_stream(stream, random_id=False):
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 1714, in _insert_record_stream
     bytes = record.get_bytes_as('fulltext')
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 454, in get_bytes_as
     self._manager._prepare_for_extract()
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 554, in _prepare_for_extract
     self._block._ensure_content(self._last_byte)
   File "/usr/lib/python2.6/dist-packages/bzrlib/groupcompress.py", line 158, in _ensure_content
     self._z_content, num_bytes + _ZLIB_DECOMP_WINDOW)
 OverflowError: signed integer is greater than maximum

Basically, it seems that we are asking zlib to do the decompression, and it is telling us that the request is invalid. I'm a bit surprised that the compression succeeded, to have the decompression fail.

The immediate fix is that we could probably change the single 'decompress()' call into looping over a fixed size decompression (1GB should be more than reasonable, 1-10MB may be just as good.)

However, I'll note that bzr has a general policy that we may allocate at least 3x the size of a single file (so in your case, of a 3.8GB file, we might allocate 3.8*3=11.4GB.) We've tuned commit to not, (it should be close to 1x), but we also probably have some code paths that still need to be tuned (some probably allocate >5x the size of a single text).

Anyway, just to say that bzr isn't going to handle a 4GB file particularly well regardless. But yes, this specific failure could probably be worked around (I just expect you'll run into quite a few more problems).

summary:	- ERROR: exceptions.OverflowError: signed integer is greater than maximum + failure to extract content > 2^31 bytes (OverflowError: signed integer + is greater than maximum)
Changed in bzr:
importance:	Undecided → Low
status:	New → Confirmed

Revision history for this message

Martin Pool (mbp) wrote on 2010-03-08: Re: [Bug 531612] Re: failure to extract content > 2^31 bytes (OverflowError: signed integer is greater than maximum)

There have been several other failures like this recently (grep for
_ensure_content); could you assess if they're dupes?

Jelmer Vernooij (jelmer) on 2011-01-20

Changed in bzr (Ubuntu):
status:	New → Confirmed

Jelmer Vernooij (jelmer) on 2011-04-28

Changed in bzr (Ubuntu):
importance:	Undecided → Low

Jelmer Vernooij (jelmer) on 2017-11-09

tags:

added: check-for-breezy

Jelmer Vernooij (jelmer) on 2019-01-11

tags:

added: lfs
removed: check-for-breezy

Jelmer Vernooij (jelmer) on 2019-01-24

Changed in brz:
status:	New → Triaged
importance:	Undecided → Low

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #894051

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntubzr package

failure to extract content > 2^31 bytes (OverflowError: signed integer is greater than maximum)

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
bzr package