SSYNC logs tracebacks on unexpected hang-up

Bug #1889951 reported by Tim Burke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Committed
Undecided
Unassigned

Bug Description

This is likely to happen if the client pops a connection timeout, but you can repro it easily enough with curl:

  $ curl -I -X SSYNC http://localhost:6010/sdb1/1
  HTTP/1.1 200 OK
  Content-Type: text/html; charset=UTF-8
  Date: Sat, 01 Aug 2020 00:45:11 GMT
  Transfer-Encoding: chunked

...but then the server logs like

  Aug 1 00:45:11 saio object-6010: 127.0.0.1/sdb1/1 EXCEPTION in ssync.Receiver:
  Traceback (most recent call last):
    File ".../swift/obj/ssync_receiver.py", line 166, in __call__
      for data in self.missing_check():
    File ".../swift/obj/ssync_receiver.py", line 343, in missing_check
      'Looking for :MISSING_CHECK: START got %r' % line[:1024])
  Exception: Looking for :MISSING_CHECK: START got b''
  Aug 1 00:45:11 saio object-6010: STDERR: Traceback (most recent call last):
    File "...s/eventlet/wsgi.py", line 599, in handle_one_response
      write(b'')
    File ".../eventlet/wsgi.py", line 538, in write
      wfile.flush()
    File ".../socket.py", line 604, in write
      return self._sock.send(b)
    File ".../eventlet/greenio/base.py", line 397, in send
      return self._send_loop(self.fd.send, data, flags)
    File ".../eventlet/greenio/base.py", line 384, in _send_loop
      return send_method(data, *args)
  BrokenPipeError: [Errno 32] Broken pipe

That second one's going to be hard to avoid since it's all buried down in eventlet, but we ought to be able to do something for that first one.

Revision history for this message
Alistair Coles (alistair-coles) wrote :

Also seen manifesting as:

object-server: err <snip> EXCEPTION in ssync.Receiver: #012Traceback (most recent call last):#012 File "/opt/ss/lib/python2.7/site-packages/swift/obj/ssync_receiver.py", line 166, in __call__#012 for data in self.missing_check():#012 File "/opt/ss/lib/python2.7/site-packages/swift/obj/ssync_receiver.py", line 340, in missing_check#012 line = self.fp.readline(self.app.network_chunk_size)#012 File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 226, in readline#012 return self._chunked_read(self.rfile, size, True)#012 File "/opt/ss/lib/python2.7/site-packages/eventlet/wsgi.py", line 211, in _chunked_read#012 raise ChunkReadError(err)#012ChunkReadError: invalid literal for int() with base 16: ''

Revision history for this message
Alistair Coles (alistair-coles) wrote :
Changed in swift:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.27.0

This issue was fixed in the openstack/swift 2.27.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.