OpenStack Compute (nova)

Cells connecting pool tracking

Bug #1417745 reported by Brian Elliott
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Brian Elliott
OpenStack Compute (nova) 2015.1.0 "kilo"
Juno
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2014.2.4

Bug Description

Cells has a rpc driver for inter-cell communication. A oslo.messaging.Transport is created for each inter-cell message.

In previous versions of oslo.messaging, connection pool references were maintained within the RabbitMQ driver abstraction in oslo.messaging. As of oslo.messaging commit f3370da11a867bae287d7f549a671811e8b399ef, the application must maintain a single reference to Transport or references to the connection pool will be lost.

The net effect of this is that cells constructs a new broker connection pool (and a connection) on every message sent between cells. This is leaking references to connections.

Brian Elliott (belliott)
Changed in nova:
assignee: nobody → Brian Elliott (belliott)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/152667

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/152667
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aac3b4b7e2e0bd20e8044f716068637329e48feb
Submitter: Jenkins
Branch: master

commit aac3b4b7e2e0bd20e8044f716068637329e48feb
Author: Brian D. Elliott <email address hidden>
Date: Tue Feb 3 20:26:33 2015 +0000

    Fix cells rpc connection leak

    Only create a Transport object once for each inter-cell hop. This
    prevents the cells rpc driver from creating a new connection pool
    (and connection) for each inter-cell message sent. This affects the
    RabbitMQ and qpid transports.

    This is a regression introduced by oslo.messaging commit
    f3370da11a867bae287d7f549a671811e8b399ef which got rid stateful
    tracking of connection pool references within oslo.messaging.
    It is now the responsibility of the caller to manage these
    references.

    See related bug:
    https://bugs.launchpad.net/oslo.messaging/+bug/1397925

    Change-Id: Id1e75f456d4c0ef5b87bf3efe810e9fcfa4cce1d
    Closes-Bug: #1417745

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Revision history for this message
Kris Lindgren (klindgren) wrote :

Can we get a backport of this patch into juno? When running with a newer version of oslo.messaging we are running into this frequently.

Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-2 → 2015.1.0
Revision history for this message
Matt Riedemann (mriedem) wrote :

It's not a clean backport to stable/juno but I'll work on it since it also came up in the Vancouver ops meetup on rabbitmq and the heartbeat patch in oslo.messaging 1.8.1 which people are running against Juno:

https://etherpad.openstack.org/p/YVR-ops-rabbitmq

tags: added: juno-backport-potential
Revision history for this message
Matt Riedemann (mriedem) wrote :

However, note that this is only required if you're running with oslo.messaging >= 1.5.0 but stable/juno global-requirements is capped at oslo.messaging < 1.5.0:

https://github.com/openstack/requirements/blob/stable/juno/global-requirements.txt#L65

So maybe not a good candidate for backport to stable/juno.

tags: removed: juno-backport-potential
tags: added: cells rpc
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/184339

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/184339
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=be8be3916fd65258b2dfa9ae158bddde1622c639
Submitter: Jenkins
Branch: stable/juno

commit be8be3916fd65258b2dfa9ae158bddde1622c639
Author: Brian D. Elliott <email address hidden>
Date: Tue Feb 3 20:26:33 2015 +0000

    Fix cells rpc connection leak

    Only create a Transport object once for each inter-cell hop. This
    prevents the cells rpc driver from creating a new connection pool
    (and connection) for each inter-cell message sent. This affects the
    RabbitMQ and qpid transports.

    This is a regression introduced by oslo.messaging commit
    f3370da11a867bae287d7f549a671811e8b399ef which got rid stateful
    tracking of connection pool references within oslo.messaging.
    It is now the responsibility of the caller to manage these
    references.

    See related bug:
    https://bugs.launchpad.net/oslo.messaging/+bug/1397925

    Conflicts:
            nova/tests/unit/cells/test_cells_rpc_driver.py

    NOTE(mriedem): This is due to the tests being moved in Kilo.

    Change-Id: Id1e75f456d4c0ef5b87bf3efe810e9fcfa4cce1d
    Closes-Bug: #1417745
    (cherry picked from commit aac3b4b7e2e0bd20e8044f716068637329e48feb)

tags: added: in-stable-juno
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.