oslo.messaging

EventletContextManagerSpawnTest test cases fail

Bug #1282706 reported by Lance Bragstad
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.messaging
Fix Released
Low
Ihar Hrachyshka
oslo.messaging 1.3.0

Bug Description

When working on adding log_handler to oslo.messaging as part of https://wiki.openstack.org/wiki/Oslo/GraduationStatus#oslo.messaging

I noticed failures with tests.test_executor.EventletContextManagerSpawnTest.test_normal_run that I can't recreate locally (using tox --recreate -e py27,pep8) .

The patch to graduate log_handler.py to oslo.messaging is here: https://review.openstack.org/#/c/74804/ but keeps failing on http://logs.openstack.org/04/74804/8/check/gate-oslo.messaging-python27/afe5f4a/console.html which seems like an unrelated failure, and when looking at http://logs.openstack.org/04/74804/8/check/gate-oslo.messaging-python27/afe5f4a/testr_results.html.gz it almost seems like a race condition with Eventlet. This is just a guess and I'm open to suggestions.

I've checked the other open reviews for oslo.messaging but it doesn't look like it is effecting other patches.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Looks like it's just your patch...

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRkFJTFxcOiB0ZXN0cy50ZXN0X2V4ZWN1dG9yLkV2ZW50bGV0Q29udGV4dE1hbmFnZXJTcGF3blRlc3QudGVzdF9ub3JtYWxfcnVuXCIgQU5EIGZpbGVuYW1lOmNvbnNvbGUuaHRtbCIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MjkyMDc2MDUzM30=

Changed in oslo.messaging:
status: New → Invalid
Revision history for this message
Lance Bragstad (lbragstad) wrote :

Ok, figured it out.. I think:

https://review.openstack.org/#/c/74804/8..9/oslo/messaging/notify/log_handler.py

Apparently using CONF = cfg.CONF the way I was doing it leaded to the error. The py27 tests are passing now so Jenkins should be ok with it once it finishes.

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

It failed the same way for https://review.openstack.org/#/c/77273/
You can find test results at: http://logs.openstack.org/73/77273/4/check/gate-oslo.messaging-python26/f0feed9/

The reason is probably some other greenthreads are still running when eventlet.sleep(0) is issued, so an incorrect thread is switched to, making test run fail.

I don't know whether there's a way to wait for greenthreads to complete before proceeding with the test.

Changed in oslo.messaging:
status: Invalid → Confirmed
OpenStack Infra (hudson-openstack)
Changed in oslo.messaging:
assignee: nobody → Ihar Hrachyshka (ihar-hrachyshka)
status: Confirmed → In Progress
Revision history for this message
Mark McLoughlin (markmc) wrote :

Fix proposed here: https://review.openstack.org/#/c/82151/

Changed in oslo.messaging:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (master)

Reviewed: https://review.openstack.org/82151
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=d4c04255f429ae22a08600db19212178ee6156bf
Submitter: Jenkins
Branch: master

commit d4c04255f429ae22a08600db19212178ee6156bf
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Mar 21 18:57:39 2014 +0100

    Fixed inconsistent EventletContextManagerSpawnTest failures

    The test spawns a greenthread and expects it to complete after
    eventlet.sleep(0) call. The problem is that if there are other
    greenthreads currently running, sleep(0) may switch to unexpected
    thread, which may result in test thread not being invoked at all.
    Instead of assuming no other green threads are running at the moment
    of test execution, explicitly wait() for the test thread.

    Change-Id: Id45307860733658638b74ed3a713493df8c1080d
    Closes-Bug: #1282706

Changed in oslo.messaging:
status: In Progress → Fix Committed
Mark McLoughlin (markmc)
Changed in oslo.messaging:
status: Fix Committed → Fix Released
milestone: none → 1.3.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.