object expirer raise exception if the target object .ts already reclaimed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Confirmed
|
Low
|
Roman Vasilets |
Bug Description
Conditions
The object be deleted by application or user before the expired due date then object-replicator's reclaim_age (default in 7 days). And the tombstone (.ts) is already deleted on the disk.
The object-updater does not updater the record in .expiring_objects yet. ( Too many async_pending jobs in queue )
In case of this, object-expirer detects the record in expiring list. So it tries to delete it from disk but actually the object's .ts already deleted by replicator. So the Exception appears in object-expirer daemon. object-expirer keep tries delete the object until object-updater to update the container DB of .expiring_
It's not a bug. The expiring object record will be removed from the container in .expiring_object account eventually by object-updater. But Swift should handle the exception even the .ts removed from disk by replicator.
If your object servers still have lot of resource available, you can tweak the object-updater's interval and concurrency to speed up the update progress. Tweaking these parameters will increase container-server and container disks loading. If you see some more container-server timeout errror in the log. You should tune it back to default.
May 14 11:52:32 localhost object-expirer: Pass completed in 6s; 0 objects expired
May 14 11:52:35 localhost object-expirer: Pass beginning; 1 possible containers; 1 possible objects
May 14 11:52:41 localhost object-expirer: Exception while deleting object 1431561594 1431628520-
Changed in swift: | |
assignee: | nobody → Roman Vasilets (rvasilets) |
Roman, are you still working on this? Do you have a patch in gerrit yet?
I'm marking this "in progress" assuming that you're working on this. Please correct me if I'm wrong