object-server logger can complain about missing txn_id

Bug #1922810 reported by Tim Burke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

Seen in prod:

Traceback (most recent call last):
  File ".../swift/obj/server.py", line 1338, in __call__
    res = getattr(self, req.method)(req)
  File ".../swift/common/utils.py", line 2016, in _timing_stats
    resp = func(ctrl, *args, **kwargs)
  File ".../swift/obj/server.py", line 1047, in PUT
    writer.put(metadata)
  File ".../swift/obj/diskfile.py", line 2941, in put
    super(DiskFileWriter, self)._put(metadata, True)
  File ".../swift/obj/diskfile.py", line 1912, in _put
    tpool.execute(self._finalize_put, metadata, target_path, cleanup)
  File ".../eventlet/tpool.py", line 129, in execute
    six.reraise(c, e, tb)
  File ".../eventlet/tpool.py", line 83, in tworker
    rv = meth(*args, **kwargs)
  File ".../swift/obj/diskfile.py", line 1868, in _finalize_put
    target_path, new_target_path)
  File ".../logging/__init__.py", line 1446, in debug
    msg, kwargs = self.process(msg, kwargs)
  File ".../swift/common/utils.py", line 2165, in process
    kwargs['extra'] = {'server': self.server, 'txn_id': self.txn_id,
  File ".../swift/common/utils.py", line 2135, in txn_id
    return self._cls_thread_local.txn_id
  File ".../eventlet/corolocal.py", line 45, in __getattribute__
    return object.__getattribute__(self, attr)
AttributeError: 'local' object has no attribute 'txn_id' (txn: txa5431800f4ad4a1da42b5-00606bcb94)

*Super* weird that the log line about not having a txn_id *has a txn_id*.

I think this comes down to the same sorts of issues we've seen in the proxy that led to us plumbing logger_thread_locals through every time we use tpool.execute(), but we've *also* seen troubles with *that*, so... I'm not sure it's a good pattern to follow.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/787649

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master)

Reviewed: https://review.opendev.org/c/openstack/swift/+/787649
Committed: https://opendev.org/openstack/swift/commit/7087fb0d7ed6e36cc2d120eb6d8125257c8c7089
Submitter: "Zuul (22348)"
Branch: master

commit 7087fb0d7ed6e36cc2d120eb6d8125257c8c7089
Author: Tim Burke <email address hidden>
Date: Thu Apr 22 13:28:36 2021 -0700

    object: Plumb logger_thread_locals through _finalize_put

    It doesn't explain how we saw the traceback we did (self.logger.txn_id
    should have been set a few lines earlier in __call__!), but at least now
    the transaction ID should come across if we're on the happy path.

    Change-Id: Ifa6f81bc02c7c84ad1f4c9ff694b645348c7c6c4
    Related-Bug: #1922810

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.