Comment 3 for bug 812970

Revision history for this message
Dillon Sadofsky (dillonsadofsky) wrote :

I would like to add to this bug, as I believe I experienced the issue and found a workaround. Firstly, as context, I have a python app that reads in MySQL blobs based on modified hashes and replicates them to another db. So, that means I have loops that work with very large result sets. I quickly noticed that python.exe could start using up gigs of space, despite that I do things in small chunks.

What I noticed is that the MySQLConnection has a list of all cursors that have ever been created through it. This container keeps cursors from every being cleaned up (due to reference counting). There are functions for removing cursors from the list, but it didn't appear to be happening automatically.

I tried several things, including wrapping the cursor in my own class that held a weakref to the real cursor. I have a custom subclass for the MySQLConnection, and it contains a function that accepts a query to be run and returns a result set. I was able to solve my memory issues by adding this line to the bottom of my execute function:
del self.cursors[:]

This cleans out the MySQLConnection cursor list. I have no idea why the connection holds onto these cursors. Because it does, their shared data never gets cleaned up (I believe MySQLConnection::close empties this list). IMO, if this list were not there, the result sets would get cleaned up automatically when the last instance of the caller using it goes out of scope. Maybe I'm wrong about that.