Eventlet exception "do not call blocking functions from the mainloop"

Bug #1840780 reported by Romain LE DISEZ
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

swift: 2.22.0
eventlet: 0.25.0 (greenlet 0.4.15)

This exception happens on a regular basis. Just reporting the stack as I have no clue what could be the cause. There is no transaction-id in the logs to follow.

Traceback (most recent call last):
  File "/opt/swift/bin/swift-object-server", line 27, in <module>
    **options))
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/wsgi.py", line 1122, in run_wsgi
    run_server(conf, logger, sock)
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/wsgi.py", line 620, in run_server
    wsgi.server(sock, app, wsgi_logger, **server_kwargs)
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/wsgi.py", line 983, in server
    client_socket, client_addr = sock.accept()
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/greenio/base.py", line 230, in accept
    self._trampoline(fd, read=True, timeout=self.gettimeout(), timeout_exc=_timeout_exc)
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/greenio/base.py", line 210, in _trampoline
    mark_as_closed=self._mark_as_closed)
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 159, in trampoline
    return hub.switch()
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 298, in switch
    return self.greenlet.switch()
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 350, in run
    self.wait(sleep_time)
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 115, in wait
    self.squelch_exception(fileno, sys.exc_info())
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 301, in squelch_exception
    traceback.print_exception(*exc_info)
  File "/opt/python-2.7.16-ovh69/lib/python2.7/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/opt/python-2.7.16-ovh69/lib/python2.7/traceback.py", line 70, in print_tb
    if line: _print(file, ' ' + line.strip())
  File "/opt/python-2.7.16-ovh69/lib/python2.7/traceback.py", line 13, in _print
    file.write(str+terminator)
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/utils.py", line 1821, in write
    {'type': self.log_type, 'value': value})
  File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1471, in error
    self.logger.error(msg, *args, **kwargs)
  File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1200, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1293, in _log
    self.handle(record)
  File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1303, in handle
    self.callHandlers(record)
  File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1343, in callHandlers
    hdlr.handle(record)
  File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 764, in handle
    self.acquire()
  File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 715, in acquire
    self.lock.acquire()
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/utils.py", line 5549, in acquire
    trampoline(self.rfd, read=True)
  File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline
    assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
AssertionError: do not call blocking functions from the mainloop

/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/utils.py:5549 is

                # Tell eventlet to suspend the current greenthread until
                # self.rfd becomes readable. This will happen when someone
                # else writes to self.wfd.
                trampoline(self.rfd, read=True)

Revision history for this message
clayg (clay-gerrard) wrote :

As a debugging tool at a minimum we might consider trying to backout the stderr to syslog redirection.

"write critical failure to stderr" is a pretty standard last ditch approach to trying to get useful information out to the operator - and in our case we have a lot of machinary trying to make syslog and eventlet play together.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.