MemoryError when generating indexes

Bug #265802 reported by Jmg-users
2
Affects Status Importance Assigned to Milestone
GNU Mailman
New
Medium
Unassigned

Bug Description

Vernier Networks has a relatively small/medium mailing
list archive (only ~238megs) that can not be
regenerated in 512megs of memory. (Once the machine
needs a reboot, this will increase to 1gig.) It dies
with a MemoryError supplied below:
Updating index files for archive [2002-May]
Traceback (most recent call last):
  File "/home/mailman/bin/arch", line 129, in ?
    main()
  File "/home/mailman/bin/arch", line 119, in main
    archiver.close()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 304, in close
    self.update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 934, in update_dirty_archives
    self.__super_update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 513, in update_dirty_archives
    self.update_archive(i)
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 937, in update_archive
    self.__super_update_archive(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 417, in update_archive
    self.__set_parameters(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 332, in __set_parameters
    firstdate = self.database.firstdate(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
207, in firstdate
    self.__openIndices(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
238, in __openIndices
    self.__closeIndices()
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
259, in __closeIndices
    index.close()
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
186, in close
    fp.write(marshal.dumps(self.dict))
MemoryError

and I also have an IOError on a lock file.

I will be looking at it, but since I do not know
Pipermail, I won't progress very far.

[http://sourceforge.net/tracker/index.php?func=detail&aid=647102&group_id=103&atid=100103]

Tags: pipermail
Revision history for this message
Jmg-users (jmg-users) wrote :

Found that in pipermail.T.processUnixMailbox, it will read
in ALL articles w/ their bodies attached. I have fixed
pipermail.Article to use an instance of a new class I wrote
for body. The class will emulate a simple sequence and an
interator object and just store the offsets to the lines,
and reread the lines as necessary. I'm waiting the nightly
run to see how things will go.

The patches currently work for small archives.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.