Bazaar

MemoryError in record_entry_contents during commit on win32

Bug #241938 reported by Igoryan on 2008-06-21

This bug report is a duplicate of: Bug #109114: [master] bzr holds whole files in memory; raises MemoryError on large files. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Bazaar	Confirmed	Medium	Unassigned

Bug Description

Environment:

- Bazaar version 1.5 upon Windows XP Professional SP2.
- 1024 Megabytes of RAM.
- 1344 Megabytes of paging file.

The test:

- About 512 Megabytes of RAM is allocated before test.
- About 512 Megabytes of page file is allocated before test.
- The video file size, which is to be commited via Bazaar, is about 1396 Megabytes.
- Actions taken:

> bzr init c:/temp/bzr
> bzr init-repo c:/temp/bzr2
> bzr branch c:/temp/bzr c:/temp/bzr2/bzr
> cd c:/temp/bzr
> bzr bind c:/temp/bzr2/bzr
> copy d:\video.avi c:\temp\bzr\video.avi
> bzr add
> bzr ci
Committing to: C:/temp/bzr2/bzr/
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "D:\programs\Python25\Lib\site-packages\bzrlib\commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\builtins.py", line 2364, in run
    author=author)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\workingtree_4.py", line 240, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\mutabletree.py", line 197, in commit
    revprops=revprops, *args, **kwargs)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\commit.py", line 355, in commit
    self._update_builder_with_changes()
  File "D:\programs\Python25\Lib\site-packages\bzrlib\commit.py", line 655, in _update_builder_with_changes
    self._populate_from_inventory(specific_files)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\commit.py", line 783, in _populate_from_inventory
    content_summary)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\commit.py", line 825, in _record_entry
    self.parent_invs, path, self.work_tree, content_summary)
  File "D:\programs\Python25\Lib\site-packages\bzrlib\repository.py", line 358, in record_entry_contents
    lines = tree.get_file(ie.file_id, path).readlines()
MemoryError

bzr 1.5 on python 2.5.2 (win32)
arguments: ['D:\\programs\\Python25\\Scripts\\bzr', 'ci']
encoding: 'cp1251', fsenc: 'mbcs', lang: None
plugins:
  launchpad D:\programs\Python25\lib\site-packages\bzrlib\plugins\launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Tags:

James Westby (james-w) on 2008-07-12

Changed in bzr:
importance:	Undecided → Medium
status:	New → Confirmed

Revision history for this message

Troy Cobb (troy-cobb) wrote on 2009-05-01:

Also affecting me with larger SQL files.

aborting commit write group: MemoryError()
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 716, in exception_to_return_code
  File "bzrlib\commands.pyo", line 911, in run_bzr
  File "bzrlib\commands.pyo", line 547, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 2789, in run
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\workingtree_4.pyo", line 226, in commit
  File "bzrlib\decorators.pyo", line 192, in write_locked
  File "bzrlib\mutabletree.pyo", line 228, in commit
  File "bzrlib\commit.pyo", line 367, in commit
  File "bzrlib\commit.pyo", line 667, in _update_builder_with_changes
  File "bzrlib\commit.pyo", line 814, in _populate_from_inventory
  File "bzrlib\commit.pyo", line 858, in _record_entry
  File "bzrlib\repository.pyo", line 445, in record_entry_contents
MemoryError

bzr 1.13 on python 2.5.2 (win32)
arguments: ['bzr', 'commit']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [1.13]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [unknown]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store
[unknown]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.9.8]
  svn C:\Program Files\Bazaar\plugins\svn [0.5.3]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Revision history for this message

John A Meinel (jameinel) wrote on 2009-05-04:

So if I understand the test correctly, you have:
- 1024 Megabytes of RAM.
- 1344 Megabytes of paging file.
And are committing a single file of size:
- The video file size, which is to be commited via Bazaar, is about 1396 Megabytes.

So you are committing a 1.4GB file, with only 2GB of memory + page, and 1GB of that is already accounted for.

I can just change this bug to "cannot commit files > memory". Though internally we've actually stated that we don't plan (in short-to-medium term) on supporting files that we cannot fit 3 copies in memory. (old, new, merged_content).

As for "larger SQL files" what size is that?

I'm tempted to mark this as Won't Fix for very large files, though we do have some plans to be tighter about that '3 copies'... There are also some discussion points about possibly internally fragmenting large files into pages, and versioning them at the page level, but I think that is at least a medium-term change.

Revision history for this message

Igoryan (igor-v-mazur) wrote on 2009-05-11:

Hi John,

Yes, you understand the test correctly.

It's not a critical issue as for me, since I rather put small text files under the version convtrol. But it's interesting to know is there any workaround in Bazaar in order to commit such a big file with such amount of memory which is used in the test?

Thanks!

Jelmer Vernooij (jelmer) on 2017-11-08

tags:

added: check-for-breezy