OpenStack Object Storage (swift)

container-reconciler needs to scale better

Bug #1836111 reported by Tim Burke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

Currently, the reconciler tries to process every misplaced object on every pass. This is a pain: you either put it on a single node, in which case you've got a single point of failure; or multiple nodes, in which case you get a ton of duplicated work.

We should slice up the work like we do for the expirer.

Revision history for this message
Tim Burke (1-tim-z) wrote :

Clay had a start on this: https://review.opendev.org/#/c/103779/

Revision history for this message
Tim Burke (1-tim-z) wrote :

Note that the "ton of duplicate work" approach can end horribly if you've got encryption enabled. We've seen three separate reconcilers try to move an object to an 8+4 EC policy at approximately the same time, leading to

* 7 frags stored with key A,
* 3 frags stored with key B, and
* 1 frag stored with key C

all with the same timestamp -- so now data can't be reconstructed.

Revision history for this message
Matthew Oliver (matt-0) wrote :

The reconsiler should do a put with a newer timestamp, well using the source but with the timestamp offset incremented. If it's encrytion causing this issue then does that mean it's not honouring timestamp offsets?

Revision history for this message
Matthew Oliver (matt-0) wrote :

Oh and what is it reconciling here. From from 1 EC container to a slightly different EC policy?

Revision history for this message
Matthew Oliver (matt-0) wrote :

oh this is discussed here: https://bugs.launchpad.net/swift/+bug/1910804

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.