Can't establish qpid connection with impl_qpid.py

I have followed the trace farther into qpid code and have this stack trace:

http://pastebin.com/k2ZKHN0C

I put a pdb.set_trace() in /usr/lib/python2.6/site-packages/qpid/compat.py(125)fileno and qpid seems to hang on:

          rd, wr, ex = select(self.reading, self.writing, (), timeout)

I did a local sync of Oslo-incubator's notifier module to Keystone, pulling in the latest notifier changes including the following commit:

https://review.openstack.org/#/c/32187/

Dims suggested I try with the latest notifier module. I set 'qpid_topology_version' to both 1 and 2 and observed the same outcome as before, qpid still hangs calling qpid_messaging.Connection(broker). It seems Glance had similar issues in Grizzly, according to the following bug:

https://bugs.launchpad.net/glance/+bug/1100317

Even though Glance doesn't use the notifier module (or the dependent rpc module) provided by Oslo-incubator I thought it might help shed some light on a possible solution for this as well.

Found the issue. Earlier in the dev cycle a commit was merged to start isolating eventlet code in Keystone (https://github.com/openstack/keystone/blob/master/keystone/common/environment/__init__.py). If a module in Keystone needs to be monkey patched, it is carried out here, when the service starts:

https://github.com/openstack/keystone/blob/master/keystone/common/environment/__init__.py#L81

Before, select was never being monkey patched. As documented in commit #1, qpid would hang on a select call, monkey patching select within Keystone allows select to execute in a monkey patched environment. This can be resolved be explicitly saying what to monkey patch and what to not in keystone/common/environment/__init__.py. This is helpful since it will avoid future issues similar to this and also makes the monkey patching code more readable. The following is a link to the patch:

http://paste.openstack.org/show/47270/

I understand I opened this under the wrong project and I will reassign to Keystone and push up a patch shortly.

Fix proposed to branch: master
Review: https://review.openstack.org/47444

Reviewed: https://review.openstack.org/47444
Committed: http://github.com/openstack/keystone/commit/c0c61ed60b356d581a5618a753f5a5a8939ab844
Submitter: Jenkins
Branch: master

commit c0c61ed60b356d581a5618a753f5a5a8939ab844
Author: Lance Bragstad <email address hidden>
Date: Thu Sep 19 18:22:03 2013 +0000

    Monkey patch select in environment

    Explicitly say what needs to be monkey patched in
    environment/__init__.py. By doing this we are making the code easier to
    read and select needs to be monkey patched in order for notifications to
    work with keystone/openstack/common/rpc/impl_qpid.py. Without monkey
    patching select, keystone will hang on sending a notification since
    qpid_messaging won't operate properly when establishing a connection
    with the qpid broker.

    Change-Id: I4cbdf29924473a76ce459d969e5e7077aac5beb8
    Closes-bug: #1224565