[SRU] Memory leaks in rados.py

Bug #1425164 reported by Billy Olsen
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ceph (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Billy Olsen
Utopic
Fix Released
High
Billy Olsen
Vivid
Fix Released
High
Unassigned

Bug Description

[Impact]

Using Ceph with Cinder causes memory to grow due to memory leaks in librados.py which holds on to native C callback function references. These references cause a circular dependency which prevents the python garbage collector from freeing the memory used by the Ioctx object. Over time with repeated volume and snapshot creation/deletion, this can cause the memory used by the Cinder process to grow continuously without retraction.

The fix is to not store a reference to the native c callback functions in the Ioctx object and instead pass the references around so they can live only as long as necessary to fulfill the request. This avoids the circular dependency issue.

[Test Case]

1. Deploy OpenStack using Ceph as a block storage for Cinder.
2. Repeatedly create and delete volumes.
3. Observe memory usage by the cinder-volume backend services.

[Regression Potential]

From a regression standpoint, there exists the possibility of a small performance regression due to creating multiple C native callback references and passing those around rather than keeping a single class level reference to a native C callback function. This may be more noticeable when using the librados library to actually perform I/O to a volume rather than using it to manage objects, which is done within the Cinder library.

[Other Info]

This has been filed and fix in upstream bug http://tracker.ceph.com/issues/10723, and merged into the hammer release stream. This should include commits 60b019f69aa0e39d276c669698c92fc890599f50 and 36d37aadbbbece28d70e827511f1a473d851463d.

tags: added: cts
James Page (james-page)
Changed in ceph (Ubuntu Vivid):
status: New → Fix Released
Changed in ceph (Ubuntu Trusty):
importance: Undecided → High
Changed in ceph (Ubuntu Utopic):
importance: Undecided → High
Changed in ceph (Ubuntu Vivid):
importance: Undecided → High
Changed in ceph (Ubuntu Utopic):
status: New → Triaged
Changed in ceph (Ubuntu Trusty):
status: New → Triaged
summary: - Memory leaks in librados.py
+ [SRU] Memory leaks in librados.py
Revision history for this message
Billy Olsen (billy-olsen) wrote : Re: [SRU] Memory leaks in librados.py
Revision history for this message
Billy Olsen (billy-olsen) wrote :
David Medberry (med)
summary: - [SRU] Memory leaks in librados.py
+ [SRU] Memory leaks in rados.py
James Page (james-page)
Changed in ceph (Ubuntu Trusty):
status: Triaged → In Progress
Changed in ceph (Ubuntu Utopic):
status: Triaged → In Progress
Changed in ceph (Ubuntu Trusty):
assignee: nobody → Billy Olsen (billy-olsen)
Changed in ceph (Ubuntu Utopic):
assignee: nobody → Billy Olsen (billy-olsen)
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Billy, or anyone else affected,

Accepted ceph into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/0.80.9-0ubuntu0.14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ceph (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in ceph (Ubuntu Utopic):
status: In Progress → Fix Committed
Revision history for this message
Chris J Arges (arges) wrote :

Hello Billy, or anyone else affected,

Accepted ceph into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/0.80.9-0ubuntu0.14.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Takashi Aosawa (t-aosawa) wrote :

Hello Chris-san and Billy-san,

Thank you for your continuous support.

I have confirmed on trusty (14.04) the package fixes the issue.

Ubuntu: 14.04 LTS
python: 2.7.5-5ubuntu3
cinder: 1:2014.1.2-0ubuntu1

tags: added: verification-done-trusty
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Verification completed on utopic as well.

tags: added: verification-done
removed: cts verification-done-trusty verification-needed
tags: added: cts
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 0.80.9-0ubuntu0.14.04.2

---------------
ceph (0.80.9-0ubuntu0.14.04.2) trusty; urgency=medium

  * Fix memory leak in rados.py (LP: #1425164):
    - d/p/fix-python-rados-memleak.patch: patches the rados.py Ioctx
      class to remove circular dependencies which prevent python from
      garbage collecting object references.
 -- Billy Olsen <email address hidden> Tue, 14 Apr 2015 09:34:42 +0100

Changed in ceph (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 0.80.9-0ubuntu0.14.10.2

---------------
ceph (0.80.9-0ubuntu0.14.10.2) utopic; urgency=medium

  * Fix memory leak in rados.py (LP: #1425164):
    - d/p/fix-python-rados-memleak.patch: patches the rados.py Ioctx
      class to remove circular dependencies which prevent python from
      garbage collecting object references.
 -- Billy Olsen <email address hidden> Tue, 14 Apr 2015 09:38:13 +0100

Changed in ceph (Ubuntu Utopic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.