swift: 2.22.0
eventlet: 0.25.0 (greenlet 0.4.15)
This exception happens on a regular basis. Just reporting the stack as I have no clue what could be the cause. There is no transaction-id in the logs to follow.
Traceback (most recent call last):
File "/opt/swift/bin/swift-object-server", line 27, in <module>
**options))
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/wsgi.py", line 1122, in run_wsgi
run_server(conf, logger, sock)
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/wsgi.py", line 620, in run_server
wsgi.server(sock, app, wsgi_logger, **server_kwargs)
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/wsgi.py", line 983, in server
client_socket, client_addr = sock.accept()
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/greenio/base.py", line 230, in accept
self._trampoline(fd, read=True, timeout=self.gettimeout(), timeout_exc=_timeout_exc)
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/greenio/base.py", line 210, in _trampoline
mark_as_closed=self._mark_as_closed)
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 159, in trampoline
return hub.switch()
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 298, in switch
return self.greenlet.switch()
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 350, in run
self.wait(sleep_time)
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 115, in wait
self.squelch_exception(fileno, sys.exc_info())
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 301, in squelch_exception
traceback.print_exception(*exc_info)
File "/opt/python-2.7.16-ovh69/lib/python2.7/traceback.py", line 125, in print_exception
print_tb(tb, limit, file)
File "/opt/python-2.7.16-ovh69/lib/python2.7/traceback.py", line 70, in print_tb
if line: _print(file, ' ' + line.strip())
File "/opt/python-2.7.16-ovh69/lib/python2.7/traceback.py", line 13, in _print
file.write(str+terminator)
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/utils.py", line 1821, in write
{'type': self.log_type, 'value': value})
File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1471, in error
self.logger.error(msg, *args, **kwargs)
File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1200, in error
self._log(ERROR, msg, args, **kwargs)
File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1293, in _log
self.handle(record)
File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1303, in handle
self.callHandlers(record)
File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 1343, in callHandlers
hdlr.handle(record)
File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 764, in handle
self.acquire()
File "/opt/python-2.7.16-ovh69/lib/python2.7/logging/__init__.py", line 715, in acquire
self.lock.acquire()
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/utils.py", line 5549, in acquire
trampoline(self.rfd, read=True)
File "/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline
assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
AssertionError: do not call blocking functions from the mainloop
/opt/swift-2.22.0-ovh86-py2.7/lib/python2.7/site-packages/swift/common/utils.py:5549 is
# Tell eventlet to suspend the current greenthread until
# self.rfd becomes readable. This will happen when someone
# else writes to self.wfd.
trampoline(self.rfd, read=True)
As a debugging tool at a minimum we might consider trying to backout the stderr to syslog redirection.
"write critical failure to stderr" is a pretty standard last ditch approach to trying to get useful information out to the operator - and in our case we have a lot of machinary trying to make syslog and eventlet play together.