oslo-incubator

ZmqProxy's PUB sockets cause eventlet polling loop - 100% cpu utilization

Bug #1065532 reported by Erica Windisch
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
oslo-incubator
Fix Released
Low
Erica Windisch
oslo-incubator grizzly-3 "g3"
Grizzly
Fix Released
Low
Erica Windisch
oslo-incubator 2013.1 "grizzly"

Bug Description

In some(?) versions of ZeroMQ and/or Eventlet, the ZmqProxy (used by nova-rpc-zmq-receiver) can consume 100% cpu.

This seems to be caused by a bug or quirk in the ZMQ_FD polling, either in Eventlet or ZeroMQ itself.

Originally reported by Joe Gordon. Confirmed by Eric Windisch

See original description
Revision history for this message
Erica Windisch (ewindisch) wrote :

https://review.openstack.org/14337

Changed in openstack-common:
status: New → Confirmed
description: updated
Mark McLoughlin (markmc)
affects: openstack-common → oslo
Revision history for this message
Stuart Stent (stuart-stent) wrote :

Is there an update to this bug?

Has this been patched upstream , if so what version of zeromq fixes it

Mark McLoughlin (markmc)
Changed in oslo:
importance: Undecided → Low
Revision history for this message
Erica Windisch (ewindisch) wrote :

Not yet fixed upstream.

The following git-fork/branch includes a fix:
 https://bitbucket.org/zedshaw/eventlet

I have debs with this alternative eventlet.green.zmq code patched in. They'll be uploaded to
https://launchpad.net/~ewindisch/+archive/eventlet-zmq-et

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/19312

Changed in oslo:
assignee: nobody → Eric Windisch (ewindisch)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/19312
Committed: http://github.com/openstack/oslo-incubator/commit/ab043101589a9167357ca317ca610a3f4144747c
Submitter: Jenkins
Branch: master

commit ab043101589a9167357ca317ca610a3f4144747c
Author: Eric Windisch <email address hidden>
Date: Wed Jan 9 12:30:47 2013 -0500

    zmq-receiver msg forwarding in greenthreads

    Because PUSH sockets can block and there
    may not be PULL consumers, the oslo-zmq-receiver
    process may stall in certain senarios.

    Improves error handling in ZmqProxy.

    This addresses bug 1097856.

    Additionally, this bug seems to improve
    (but not necessarily fix) the behavior
    reported in bug 1065532.

    Change-Id: I6df6035a6676c5bcdddaec7a332ac77e621ba9f3

Changed in oslo:
status: In Progress → Fix Committed
Revision history for this message
Erica Windisch (ewindisch) wrote :

The previous patch was at best a partial workaround and should not have resolved this ticket.

Eventlet needs to be bumped to version 0.12.0

Changed in oslo:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/21012

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/21012
Committed: http://github.com/openstack/oslo-incubator/commit/4a9a7dc39c296455349a59346d22cc06fc5364c8
Submitter: Jenkins
Branch: master

commit 4a9a7dc39c296455349a59346d22cc06fc5364c8
Author: Eric Windisch <email address hidden>
Date: Fri Feb 1 11:56:42 2013 -0500

    Bump eventlet to 0.12.0

    This version has important fixes required
    for ZeroMQ (spurious wakeups).

    Fixes bug 1065532.

    Change-Id: I16b9220db1f5bf16a24644887963ce18e4e5ecbd

Changed in oslo:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in oslo:
milestone: none → grizzly-3
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.