MD5 check implementation is inefficient
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ubiquity (Ubuntu) |
Fix Released
|
Low
|
Evan |
Bug Description
Binary package hint: ubiquity
Ubiquity's new MD5 checking implementation uses shutil.copyfileobj to copy the file (which reads and writes 16KB chunks at a time), and then seeks back to the start of the source file, reads the whole thing into memory in one go, and computes its MD5. This is inefficient in two ways: firstly, it must read the file twice; secondly, it uses memory linear in the size of the largest file being copied.
hashlib.md5 allows you to create a hash object with no data, and feed data to it block by block. Therefore, I would suggest expanding out the (trivial) code for shutil.copyfileobj, and feeding each block to md5obj.update as you go along. Then you just have to close and reopen the target file and compute its MD5; I would suggest also doing that block-by-block to keep memory requirements down.
Related branches
Changed in ubiquity: | |
assignee: | nobody → evand |
importance: | Undecided → Low |
Changed in ubiquity: | |
status: | New → Confirmed |
Changed in ubiquity: | |
status: | Confirmed → Fix Committed |
This bug was fixed in the package ubiquity - 1.8.2
---------------
ubiquity (1.8.2) hardy; urgency=low
[ Evan Dandrea ] too_small (LP:
* Treat the dbfilter returning nonzero as a fatal error in the
noninteractive frontend.
* Clean up the reboot message in the noninteractive frontend.
* Optimize the md5 check on file copy (LP: #198019).
* Copy the locale over to the target system when using oem-config.
This was previously fixed for the alternate CD in LP 181291.
* Use + instead of : for a replacement character for m-a as it is
explicitly allowed by debconf policy.
* Break out of resizing the partition in cases where partman fed us bad
boundary values (LP: #197838).
* Don't miscalculate with nested partitions in 03partition_
#198039).
* Automatic update of included source packages: apt-setup
1:0.31ubuntu7, choose-mirror 2.19ubuntu5, clock-setup 0.92ubuntu3,
console-setup 1.21ubuntu6, localechooser 1.42ubuntu4, migration-
assistant 0.6.1, partman-auto 73ubuntu7, tzsetup 1:0.20.
[ Mario Limonciello ]
* Don't unnecessarily hardcode the return of get_hostname()
in the noninteractive frontend.
* Turn off console blanking in Ubiquity init script to prevent
confusion on the noninteractive frontend.
* Update mythbuntu ubiquity icon.
[ Colin Watson ] setup/restricte d is false. postinst can check its dimensions (LP: #188764).
* Fix crash if partitioning failed when partitioning was automated (see LP
#206113).
* Remove packages in the restricted section from the installed system if
apt-
* Fix crash with the "don't use" partitioning option (LP: #132611).
* Use localised, human-readable names for partitioning methods.
* Reorganise the wrapper script to cope with kdesu's broken argument
handling.
* Add a "Format the partition?" checkbox to the partition edit dialog
(LP: #184838).
* Give reconfigured packages access to our X display, so that
usplash.
-- Evan Dandrea <email address hidden> Tue, 08 Apr 2008 02:02:23 -0400