Recently compacted shard DBs can return 500s during listings

Bug #1922386 reported by Tim Burke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Committed
Undecided
Unassigned

Bug Description

Seen in the wild:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/swift/container/server.py", line 893, in __call__
    res = getattr(self, req.method)(req)
  File "/usr/local/lib/python3.9/site-packages/swift/common/utils.py", line 2045, in _timing_stats
    resp = func(ctrl, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/swift/container/server.py", line 766, in GET
    container_list = broker.get_shard_ranges(
  File "/usr/local/lib/python3.9/site-packages/swift/container/backend.py", line 1806, in get_shard_ranges
    filler_sr.lower = last_upper
  File "/usr/local/lib/python3.9/site-packages/swift/common/utils.py", line 5222, in lower
    raise ValueError(
ValueError: lower ('...') must be less than or equal to upper ('...')

The trouble comes down to the DB being "sharded" for the purposes of the proxy's "x-backend-override-shard-name-filter: sharded" header (so we clear marker/end_marker) and fill_gaps being true (since we're listing). I *think* the fix is just to reset marker/end_marker to our own_shard_range's bounds.

Revision history for this message
Tim Burke (1-tim-z) wrote :

FWIW, you can readily see this while running probe tests -- just have another window running something like

   tail -f /var/log/syslog | sed -e '/#012/!d;s/#012/\n/g'

while you run the sharding probe tests.

Revision history for this message
Alistair Coles (alistair-coles) wrote :
Changed in swift:
status: New → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.28.0

This issue was fixed in the openstack/swift 2.28.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.