Excessive memory usage in python-subversion

Also reproduced by Erik Bågfors.

Can you pull the latest bzr.dev and see what happens?

We fixed one of our big memory consumers. But that was for _KnitData, so it may have no bearing on what you are doing.

I used latest bzr.dev from this morning, revno 1885, so I guess not.
(1886 which is the latest now, doesn't seem to have anything to do with
memory stuff :) )

/Erik

tor 2006-07-27 klockan 15:28 +0000 skrev John A Meinel:

> Can you pull the latest bzr.dev and see what happens?
>
> We fixed one of our big memory consumers. But that was for _KnitData, so
> it may have no bearing on what you are doing.
>

Does the branch complete? If so you should be able to branch from the resulting bzr branch, and see how that performs, which should help eliminate whether it's a bzr-svn problem?

fre 2006-07-28 klockan 00:31 +0000 skrev Michael Ellerman:
> Does the branch complete? If so you should be able to branch from the
> resulting bzr branch, and see how that performs, which should help
> eliminate whether it's a bzr-svn problem?

Mine didn't complete.
I will try a smaller svn repo and see. A really small works just fine.

/Erik

fre 2006-07-28 klockan 00:31 +0000 skrev Michael Ellerman:
> Does the branch complete? If so you should be able to branch from the
> resulting bzr branch, and see how that performs, which should help
> eliminate whether it's a bzr-svn problem?

So I tried this with a smaller branch. This one only has 1598 revisions.
During the branching from svn it took 215MB of RAM. When branching the
bzr branch it took about 20MB. So this seams to be in bzr-svn.

Regards,
Erik

It looks like this has to do with the APR memory management routines that Subversion uses. Somewhere a reference isn't being dropped.

in particular the revision information fetch code seems leaky. no idea why.

For now I'm rejecting this as part of bzr, if Jelmer thinks it is our fault, I'm happy to re-open it.

(I know we had a bug in 'bzr log' that consumed too much memory, but that has also been fixed)

The Mercurial folks found a memory leak in svn.ra.get_log() that is at
least partially responsible for the memory usage:

http://hg.intevation.org/mercurial/crew/file/e8f4e40f285a/hgext/convert/subversion.py

  affects /products/subversion

Newer versions of Subversion don't appear to leak as much memory.
Especially 1.3.x seems to use a lot of memory.
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

I was importing one project from KDE SVN, which has currently 733k revisions, most of them obviously unrelated to the project. ^C and restart every few minutes was no fun, so I wrote this ugly workaround - http://users.musicbrainz.org/~luks/tmp/bzr-svn-log.diff - it has no error checking and is definitely not meant for inclusion in bzr-svn, but it did the job for me.

Thanks for the workaround! Will try it out. Would be nice if something like this could get into the official bzr-svn.

After some digging, I found out where the leak is (bug 3052 filed in Subversions Issue Tracker). I've added a workaround to bzr-svn that frees most of the memory python-subversion leaks.

python-svn specific. I'll attach a proposed fix.

Actually a leak in the python-subversion bindings.

Kees: can you take a look at it?

This debdiff looks good. Thanks for getting it tested! I'll upload it shortly.

This bug was fixed in the package subversion - 1.4.4dfsg1-1ubuntu5

---------------
subversion (1.4.4dfsg1-1ubuntu5) hardy; urgency=medium

  * Add debian/patches/python-memleak: Fix major memory leak in the
    Python bindings. Taken from Debian SVN head. (LP: #54253)

 -- Jelmer Vernooij <email address hidden> Fri, 18 Jan 2008 01:28:28 +0100