Memory leaked per connection

Not a bug in eucalyptus

About "other rampart configurations", we added rampart for eucalyptus so if the patch is needed to work properly with the way eucalyptus uses rampart, I'd say it takes precedence over other unknown ways that may rely on the buggy behavior.

But knowing *why* this was commented out would certainly help in the discussion.

Asked upstream, since they explicitly commented this out in a "memory leak fix" revision:
http://svn.apache.org/viewvc?view=revision&revision=709349

the patch required a small patch to eucalyptus that initializes axis2c in the CC for each NC client connection (most already were handled this way, except to StartNetwork/RunInstances).

in r946

Rampart upstream is looking at the patch.

Comments on the patch from Rampart upstream (shankar):

The patch is having several problems. First of all, if the
rampart_context is freed in in_handler, server side functionalities
will be broken. rampart_context is actually freed when msg_ctx is
freed. So no need to free it.

Uncommenting the lines to free receiver_cert is fine, but it causes
crash in some scenarios due to a bug. I fixed it in trunk. If you are
working on 1.3.0 release, I have attached a patch. You have to apply
it as well to fix the crash.

Thierry, Shankar,

Thank you both for following up on this issue and finding a potential patch to the memory leak issue in rampart; I'm testing the rampartc-1.3.0 with the above patch now with a long term eucalyptus test, and will report back when the test completes (we usually run for .5 day to make sure that memory is not creeping up). So far, it looks great.

Dan: thanks for testing this.

Did you simply apply both patches (yours and shankar's), or did you also remove the rampart_context_free in in_handler from your patch ? In doubt, could you attach the combined patch (against rampart 1.3.0) that you are currently testing ?

@Dan:
Any news about how did your long term test go ?
Please also answer to question in comment 9, so that I know what you exactly tested :)

Eucalyptus part of the fix now committed to lp:~ubuntu-core-dev/eucalyptus/ubuntu-karmic

This bug was fixed in the package eucalyptus - 1.6.1~bzr1083-0ubuntu1

---------------
eucalyptus (1.6.1~bzr1083-0ubuntu1) lucid; urgency=low

  [ Dustin Kirkland ]
  * Merge upstream bzr revision 1082; the following bugs have been fixed
    upstream since the last merge:
    - LP: #378969 - private bug
    - LP: #404842 - init script fix
    - LP: #434283 - existing keys should be overwritten unconditionally
    - LP: #445990 - run instance will fail if no kernel or ramdisk specified
    - LP: #447457 - euca_conf --register-sc ... check the number of parameters
    - LP: #449874 - fix incorrect help text (--delete-nodes doesn't exist)
    - LP: #451795 - show registered images in elastic fox
    - LP: #454405 - return correct networkIndex values on describeInstances
    - LP: #456877 - init script fix
    - LP: #456878 - fix for libvirt xen driver
    - LP: #460085 - fix rampart memory leak
    - LP: #461156 - fix authentication problem w/ userdata
    - LP: #461394 - fix multiple concurrent snapshots on the same volume
    - LP: #461444 - fix memory leaks in NC getConsoleOutput and startup_thread
    - LP: #469984 - fix iptables rules issue
    - LP: #477776 - fix query string authentication
    - LP: #480783 - allow api connection over https
    - LP: #482249 - fix "Describe Regions"
    - LP: #484217 - create keypair should return an error if key exists
    - LP: #490623 - parse RFC 1123 formatted datetime
  * debian/control:
    - make all package lists one-per-line (makes changes henceforth more
      readable), sort lists
    - depend on rampart >= 1.3.0-0ubuntu6, which fixes some shared library
      installation issues
  * debian/patches/04-axis2c-1.6.0-rampart-1.3.0.patch: drop this patch,
    since Eucalyptus 1.6.1 natively supports axis2c 1.6.0 now
  * debian/eucalyptus-cloud.install,
    debian/eucalyptus-common.eucalyptus.upstart,
    debian/eucalyptus-java-common.install, debian/eucalyptus-sc.install,
    debian/eucalyptus-walrus.install: update static version number strings
    from "1.6-devel" to "1.6.1"; (we should really find a better way to do
    this)
  * debian/patches/03-DESTDIR.patch: ported forward for merge
 -- Dustin Kirkland <email address hidden> Tue, 01 Dec 2009 21:09:28 -0600

Dan/Thierry-

I just uploaded a package to karmic-proposed containing the eucalyptus r946 fix.

Could one of you guys help draft the SRU testing notes and put that into the bug description? Thanks.

Accepted eucalyptus into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Thierry-

Can you verify this package?

We need other fixes associated with this update here onsite at a partner.

Dustin,
We can't verify this fix without the associated Rampart fix, which is pending some tests on Dan/Eucalyptus side.

All,

I've verified that the upstream patch, on top of the patch we've provided, is both functional and does not exhibit the memory leak problem; the test was to run the CC/NC against patched rampart/c for approximately 18 hours, watching the memory footprint of the associated apache2 processes.

Minimal patch for rampart

This bug was fixed in the package rampart - 1.3.0-0ubuntu7

---------------
rampart (1.3.0-0ubuntu7) lucid; urgency=low

  * debian/patches/rampart-memleak.patch: Fix memory leak in rampart where
    for every connection receiver_cert is not freed (LP: #460085)
 -- Thierry Carrez <email address hidden> Mon, 14 Dec 2009 09:09:58 +0100

Accepted rampart into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

All,

I've been able to confirm that the newest euca/rampart packages, from proposed, are both functional and do not exhibit the memory leak. To be specific, i'm running with:

eucalyptus-cc: 1.6~bzr931-0ubuntu7.4
librampart0: 1.3.0-0ubuntu5.1

Thanks!
-Dan

This bug was fixed in the package eucalyptus - 1.6~bzr931-0ubuntu7.4

---------------
eucalyptus (1.6~bzr931-0ubuntu7.4) karmic-proposed; urgency=low

  [ Thierry Carrez ]
  * cluster/handlers.c: Cherrypick upstream r946: initialize axis2c in the CC
    for each NC client connection, to avoid rampart memory leak (LP: #460085)
  * clc/modules/wsstack/src/main/java/com/eucalyptus/ws/handlers/HmacV2Handler.java:
    Cherrypick upstream r1079: Fix authentication issue when using a euca2ools
    that doesn't double base64encode userdata (LP: #461156)

  [ Dustin Kirkland ]
  * debian/eucalyptus-cc.upstart, debian/eucalyptus-common.eucalyptus.upstart:
    support CLEAN=1 on start/stop/restart of eucalyptus/euclayptus-cc; export
    the CLEAN env variable in eucalyptus.init, and handle it in both the
    pre-start and post-stop sections of eucalyptus-cc, (LP: #491254)
 -- Dustin Kirkland <email address hidden> Wed, 02 Dec 2009 17:58:18 -0600

This bug was fixed in the package rampart - 1.3.0-0ubuntu5.1

---------------
rampart (1.3.0-0ubuntu5.1) karmic-proposed; urgency=low

  * debian/patches/rampart-memleak.patch: Fix memory leak in rampart where
    for every connection receiver_cert is not freed (LP: #460085)
 -- Thierry Carrez <email address hidden> Mon, 14 Dec 2009 08:43:06 +0100