Memory leak while hashing

Bug #517948 reported by Max Khon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
LinuxDC++
Invalid
Medium
Unassigned

Bug Description

Looks like memory mappings are destroyed on Linux when the file descriptor is closed. This is not the case on FreeBSD.

Please see attached patch.

Revision history for this message
Max Khon (fjoe) wrote :
Revision history for this message
Razzloss (razzloss) wrote :

The last mapping isn't destroyed. And according to the manpage this seems to be true with Linux also. Though I doubt it is 16M every time, only in the worst case scenario. Since if that one client was leaking 16M every file hashed, it would have run out of memory long ago.)

Upstream changes to the HashManager seem to have fixed this already, commit only needed to the Linuxdcpp-trunk.

--RZ

Changed in linuxdcpp:
importance: Undecided → Medium
milestone: none → 1.1.0
status: New → Confirmed
tags: added: core hashing
Revision history for this message
Razzloss (razzloss) wrote :

Lovely, I should reread these comments before sending.

The one client part refers to certain instance of linuxdcpp that has been running for the last three weeks and there hasn't been that significant memory problems...

--RZ

Revision history for this message
Steven Sheehy (steven-sheehy) wrote :

I mentioned this memory leak in https://bugs.launchpad.net/linuxdcpp/+bug/473173/comments/12, but guess I forgot to do something about it in trunk and just fixed it upstream. Either that or I was too lazy and hoped Razzloss would fix it. :D

tags: added: memoryleak
summary: - [patch] huge memory in file hasher on FreeBSD (16M per every hashed
- file)
+ Memory leak while hashing
Revision history for this message
Steven Sheehy (steven-sheehy) wrote :

After looking at the code and discussing with Razzloss, we don't see a memory leak here. Your patch will actually cause it to try to unmap the same memory twice. The one thing that was wrong with the code was an extra iteration in the fast hash loop that I fixed in revision 351. This extra iteration would not cause an extra mmap though due to the if(size_left > 0) check.

Are you sure you're getting a memory leak? If so, are you able to verify it (say using a memory leak detector)? Can you try with the latest revision?

Changed in linuxdcpp:
milestone: 1.1.0 → none
status: Confirmed → Incomplete
Revision history for this message
Max Khon (fjoe) wrote :

My patch prevents exit from loop (due to size_left <= 0) without doing munmap. And it does not cause munmap to be called twice.
The patch was made against 1.0.3.

However, the code in trunk (rev 1.351) does not have this problem and is ok.

Revision history for this message
Steven Sheehy (steven-sheehy) wrote :

Sorry, you're right it doesn't munmap twice, I was mistaken. However, before rev 351 the fast hash loop had an extra, useless iteration in which size_left == 0, pos == size and no memory is mmapped as a result due to the if(size_left > 0) check. Your patch stops it from entering that extra iteration similar to how rev 351 does. Our point is that there was no memory being allocated during this last iteration so we're not sure why you experience leaking. If you were to put a printf of the buf address and size_read when it was mmap'd and munmap'd, you would see that there was a munmap for every mmap even before rev 351.

The only thing that occurs on that extra iteration is a tth.update and a crc calculation. So perhaps those were causing leaks for you. Does your system happen to be big endian? Tiger hash behaves differently on big endian.

Revision history for this message
Max Khon (fjoe) wrote :

Yes, you seem to be right.

No, my system is x86_64.

Revision history for this message
Steven Sheehy (steven-sheehy) wrote :

If I'm right, as you say, does that mean that there was no memory leaking and this bug can be marked as invalid? Or are you saying that you confirmed it was leaking but revision 351 now stops it from leaking and we should mark it as fix committed?

Changed in linuxdcpp:
status: Incomplete → Invalid
tags: added: memory-leak
removed: memoryleak
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Patches

Remote bug watches

Bug watches keep track of this bug in other bug trackers.