Objects are not deleted from additional handoff when primary disk fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
We had corrupted some partitions of hard drive that results I/O error on mkdir when rsync tried to replicate objects. This results more rsyncs and may result in returning objects "from grave" when disk is not replaced during reclaim age.
Description of state
-------
Probably when we had disconnected network between zones, was created data file on the 5th handoff - server "11" (I opened a ringfile to see that the partition belongs to other 4 handoffs and then was the server that repeats object replication). We have replica = 4, 2 zones and 2 region in each, and one failing disk drive (I expect that the failed drive was in the same region where server 11 - the 5th replica, but I'm not sure now).
The file was deleted, but not from server 11. Then the object was every replicator cycle rsynced to all primaries (except the failing one) and deleted, because there was a tombstone. After reclaim age was the object placed on primaries.
I write "an object", but that are few hundred objects and thousands of tombstones.
Suggestions
-----------
Make a new metrics
- age of the oldest partition that is placed on handoff
- count of "handoff objects" waiting to delete
Fix it - place objects to the best handoffs. When first handoff is used then tombstone or object updates will be placed there too. "First" means the necessary count of handoffs to satisfy replication count from the begin of handoffs list.
https:/ /review. openstack. org/#/c/ 470440/