As clayg, I still see these issues in 2.12. It's not related to reconstruction or handoff. I put some debug logs in diskfile, here is what I found.

100% of these errors happen on os.rename(data_file_path, durable_data_file_path)

In every case I analysed (~10):
- it was a request coming from proxy-server, not object-reconstructor;
- the datadir was still there, but not the datafile to rename to durable;
- there were concurrent upload for the same object, but the most recent upload was completing before the first one

Let me explain last point:
1. user start to upload object, ts=1
2. user start to upload object, ts=2
3. upload with ts=2 finishes, it is marked as durable
4. upload with ts=1 finishes, it fails to mark it as durable

Based on this observation, I tried to reproduce, without success.

Here are logs from one case (Swift + custom logs)

Mar 16 11:15:10 localhost object-server: OVH ERROR: data_file_path (/srv/node/disk-01-000/objects-1/69029/2c6/43697ca96ece87902bfefd8727af72c6/1489659239.46979#13.data) does not exist (rename: /srv/node/disk-01-000/objects-1/69029/2c6/43697ca96ece87902bfefd8727af72c6/1489659239.46979#13.data -> /srv/node/disk-01-000/objects-1/69029/2c6/43697ca96ece87902bfefd8727af72c6/1489659239.46979#13#d.data): #012Traceback (most recent call last):#012  File "/opt/swift-2.12.0-ovh20170307110844/local/lib/python2.7/site-packages/swift/obj/diskfile.py", line 2711, in _finalize_durable#012    os.rename(data_file_path, durable_data_file_path)#012OSError: [Errno 2] No such file or directory
Mar 16 11:15:10 localhost object-server: Problem making data file durable /srv/node/disk-01-000/objects-1/69029/2c6/43697ca96ece87902bfefd8727af72c6/1489659239.46979#13#d.data ([Errno 2] No such file or directory): #012Traceback (most recent call last):#012  File "/opt/swift-2.12.0-ovh20170307110844/local/lib/python2.7/site-packages/swift/obj/diskfile.py", line 2711, in _finalize_durable#012    os.rename(data_file_path, durable_data_file_path)#012OSError: [Errno 2] No such file or directory
Mar 16 11:15:10 localhost object-server: ERROR __call__ error with PUT /disk-01-000/69029/AUTH_abc/container/obj : #012Traceback (most recent call last):#012  File "/opt/swift-2.12.0-ovh20170307110844/local/lib/python2.7/site-packages/swift/obj/server.py", line 1102, in __call__#012    res = getattr(self, req.method)(req)#012  File "/opt/swift-2.12.0-ovh20170307110844/local/lib/python2.7/site-packages/swift/common/utils.py", line 1609, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/opt/swift-2.12.0-ovh20170307110844/local/lib/python2.7/site-packages/swift/obj/server.py", line 824, in PUT#012    writer.commit(request.timestamp)#012  File "/opt/swift-2.12.0-ovh20170307110844/local/lib/python2.7/site-packages/swift/obj/diskfile.py", line 2770, in commit#012    self._finalize_durable, data_file_path, durable_data_file_path)#012  File "/opt/swift-2.12.0-ovh20170307110844/local/lib/python2.7/site-packages/swift/common/utils.py", line 3517, in tpool_reraise#012    raise resp#012DiskFileError: Problem making data file durable /srv/node/disk-01-000/objects-1/69029/2c6/43697ca96ece87902bfefd8727af72c6/1489659239.46979#13#d.data ([Errno 2] No such file or directory) (txn: tx145719c411234575a21eb-0058ca6567)
Mar 16 11:15:10 localhost object-server: 1.2.3.4 - - [16/Mar/2017:10:15:10 +0000] "PUT /disk-01-000/69029/AUTH_abc/container/obj" 500 1046 "PUT http://storage.example.com/v1/AUTH_abc/container/obj" "tx145719c411234575a21eb-0058ca6567" "proxy-server 30002" 71.3792 "-" 18663 1






But on disk, I found
# ls /srv/node/disk-01-000/objects-1/69029/2c6/43697ca96ece87902bfefd8727af72c6
1489659305.19142#13#d.data

which match this request
Mar 16 11:15:08 localhost object-server: 1.2.3.4 - - [16/Mar/2017:10:15:08 +0000] "PUT /disk-01-000/69029/AUTH_abc/container/obj" 201 - "PUT http://storage.example.com/v1/AUTH_abc/container/obj" "txa48ef93d878242da8135b-0058ca65a9" "proxy-server 29995" 3.3376 "-" 18663 1