| 2016-11-04 13:49:05 |
Alistair Coles |
bug |
|
|
added bug |
| 2016-11-04 13:58:51 |
Alistair Coles |
description |
admin6 reported in irc:
"thanks for the backport to mitaka for checking EC fragments metadata. I applied it, and it seems to work, but sometimes, I have errors on os.rename with no such file or directory : "
Nov 4 11:26:31 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR DiskFile /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data close failure: [Errno 2] No such file or directory : File "/usr/bin/swift-object-auditor", line 29, in <module>#012 run_daemon(ObjectAuditor, conf_file, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 110, in run_daemon#012 klass(conf).run(once=once, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 57, in run#012 self.run_forever(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 393, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 367, in audit_loop#012 pids.append(self.fork_child(**kwargs))#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 314, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 302, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 129, in audit_all_objects#012 self.failsafe_object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1600, in __iter__#012 self.close()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1800, in close#012 {'exc': e, 'stack': ''.join(traceback.format_stack()),#012
Nov 4 11:27:09 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735/1468760857.79624#5.data: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:09 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735 failed audit and was quarantined: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:58 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c/1473894901.93179#7.data: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Trying to audit /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1586, in __iter__#012 self._update_checks(chunk)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2581, in _update_checks#012 self._check_frag(self.frag_buf[cursor:cursor + self.frag_size])#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2562, in _check_frag#012 self._quarantine(msg)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1762, in _quarantine#012 self._data_file)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3295, in run_in_thread#012 result = func(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 455, in wrapper#012 return f(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 220, in quarantine_renamer#012 renamer(from_dir, to_dir, fsync=False)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer#012 os.rename(old, new)#012OSError: [Errno 2] No such file or directory
Nov 4 11:29:00 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61/1471680446.72800#1.data: Invalid EC metadata at offset 0x0
Nov 4 11:29:00 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61 failed audit and was quarantined: Invalid EC metadata at offset 0x0 |
admin6 reported in irc [1]:
"thanks for the backport to mitaka for checking EC fragments metadata. I applied it, and it seems to work, but sometimes, I have errors on os.rename with no such file or directory : "
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2016-11-04.log.html#t2016-11-04T09:59:42
logs:
Nov 4 11:26:31 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR DiskFile /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data close failure: [Errno 2] No such file or directory : File "/usr/bin/swift-object-auditor", line 29, in <module>#012 run_daemon(ObjectAuditor, conf_file, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 110, in run_daemon#012 klass(conf).run(once=once, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 57, in run#012 self.run_forever(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 393, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 367, in audit_loop#012 pids.append(self.fork_child(**kwargs))#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 314, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 302, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 129, in audit_all_objects#012 self.failsafe_object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1600, in __iter__#012 self.close()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1800, in close#012 {'exc': e, 'stack': ''.join(traceback.format_stack()),#012
Nov 4 11:27:09 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735/1468760857.79624#5.data: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:09 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735 failed audit and was quarantined: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:58 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c/1473894901.93179#7.data: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Trying to audit /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1586, in __iter__#012 self._update_checks(chunk)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2581, in _update_checks#012 self._check_frag(self.frag_buf[cursor:cursor + self.frag_size])#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2562, in _check_frag#012 self._quarantine(msg)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1762, in _quarantine#012 self._data_file)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3295, in run_in_thread#012 result = func(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 455, in wrapper#012 return f(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 220, in quarantine_renamer#012 renamer(from_dir, to_dir, fsync=False)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer#012 os.rename(old, new)#012OSError: [Errno 2] No such file or directory
Nov 4 11:29:00 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61/1471680446.72800#1.data: Invalid EC metadata at offset 0x0
Nov 4 11:29:00 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61 failed audit and was quarantined: Invalid EC metadata at offset 0x0 |
|
| 2016-11-04 13:59:21 |
Alistair Coles |
description |
admin6 reported in irc [1]:
"thanks for the backport to mitaka for checking EC fragments metadata. I applied it, and it seems to work, but sometimes, I have errors on os.rename with no such file or directory : "
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2016-11-04.log.html#t2016-11-04T09:59:42
logs:
Nov 4 11:26:31 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR DiskFile /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data close failure: [Errno 2] No such file or directory : File "/usr/bin/swift-object-auditor", line 29, in <module>#012 run_daemon(ObjectAuditor, conf_file, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 110, in run_daemon#012 klass(conf).run(once=once, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 57, in run#012 self.run_forever(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 393, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 367, in audit_loop#012 pids.append(self.fork_child(**kwargs))#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 314, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 302, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 129, in audit_all_objects#012 self.failsafe_object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1600, in __iter__#012 self.close()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1800, in close#012 {'exc': e, 'stack': ''.join(traceback.format_stack()),#012
Nov 4 11:27:09 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735/1468760857.79624#5.data: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:09 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735 failed audit and was quarantined: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:58 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c/1473894901.93179#7.data: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Trying to audit /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1586, in __iter__#012 self._update_checks(chunk)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2581, in _update_checks#012 self._check_frag(self.frag_buf[cursor:cursor + self.frag_size])#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2562, in _check_frag#012 self._quarantine(msg)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1762, in _quarantine#012 self._data_file)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3295, in run_in_thread#012 result = func(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 455, in wrapper#012 return f(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 220, in quarantine_renamer#012 renamer(from_dir, to_dir, fsync=False)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer#012 os.rename(old, new)#012OSError: [Errno 2] No such file or directory
Nov 4 11:29:00 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61/1471680446.72800#1.data: Invalid EC metadata at offset 0x0
Nov 4 11:29:00 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61 failed audit and was quarantined: Invalid EC metadata at offset 0x0 |
admin6 reported in irc [1]:
"thanks for the backport to mitaka for checking EC fragments metadata. I applied it, and it seems to work, but sometimes, I have errors on os.rename with no such file or directory : "
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2016-11-04.log.html#t2016-11-04T09:59:42
logs:
```
Nov 4 11:26:31 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR DiskFile /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data close failure: [Errno 2] No such file or directory : File "/usr/bin/swift-object-auditor", line 29, in <module>#012 run_daemon(ObjectAuditor, conf_file, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 110, in run_daemon#012 klass(conf).run(once=once, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 57, in run#012 self.run_forever(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 393, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 367, in audit_loop#012 pids.append(self.fork_child(**kwargs))#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 314, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 302, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 129, in audit_all_objects#012 self.failsafe_object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1600, in __iter__#012 self.close()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1800, in close#012 {'exc': e, 'stack': ''.join(traceback.format_stack()),#012
Nov 4 11:27:09 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735/1468760857.79624#5.data: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:09 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735 failed audit and was quarantined: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:58 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c/1473894901.93179#7.data: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Trying to audit /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1586, in __iter__#012 self._update_checks(chunk)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2581, in _update_checks#012 self._check_frag(self.frag_buf[cursor:cursor + self.frag_size])#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2562, in _check_frag#012 self._quarantine(msg)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1762, in _quarantine#012 self._data_file)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3295, in run_in_thread#012 result = func(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 455, in wrapper#012 return f(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 220, in quarantine_renamer#012 renamer(from_dir, to_dir, fsync=False)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer#012 os.rename(old, new)#012OSError: [Errno 2] No such file or directory
Nov 4 11:29:00 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61/1471680446.72800#1.data: Invalid EC metadata at offset 0x0
Nov 4 11:29:00 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61 failed audit and was quarantined: Invalid EC metadata at offset 0x0
``` |
|
| 2016-11-04 13:59:48 |
Alistair Coles |
description |
admin6 reported in irc [1]:
"thanks for the backport to mitaka for checking EC fragments metadata. I applied it, and it seems to work, but sometimes, I have errors on os.rename with no such file or directory : "
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2016-11-04.log.html#t2016-11-04T09:59:42
logs:
```
Nov 4 11:26:31 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR DiskFile /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data close failure: [Errno 2] No such file or directory : File "/usr/bin/swift-object-auditor", line 29, in <module>#012 run_daemon(ObjectAuditor, conf_file, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 110, in run_daemon#012 klass(conf).run(once=once, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 57, in run#012 self.run_forever(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 393, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 367, in audit_loop#012 pids.append(self.fork_child(**kwargs))#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 314, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 302, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 129, in audit_all_objects#012 self.failsafe_object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1600, in __iter__#012 self.close()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1800, in close#012 {'exc': e, 'stack': ''.join(traceback.format_stack()),#012
Nov 4 11:27:09 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735/1468760857.79624#5.data: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:09 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735 failed audit and was quarantined: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:58 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c/1473894901.93179#7.data: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Trying to audit /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1586, in __iter__#012 self._update_checks(chunk)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2581, in _update_checks#012 self._check_frag(self.frag_buf[cursor:cursor + self.frag_size])#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2562, in _check_frag#012 self._quarantine(msg)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1762, in _quarantine#012 self._data_file)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3295, in run_in_thread#012 result = func(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 455, in wrapper#012 return f(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 220, in quarantine_renamer#012 renamer(from_dir, to_dir, fsync=False)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer#012 os.rename(old, new)#012OSError: [Errno 2] No such file or directory
Nov 4 11:29:00 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61/1471680446.72800#1.data: Invalid EC metadata at offset 0x0
Nov 4 11:29:00 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61 failed audit and was quarantined: Invalid EC metadata at offset 0x0
``` |
admin6 reported in irc [1]:
"thanks for the backport to mitaka for checking EC fragments metadata. I applied it, and it seems to work, but sometimes, I have errors on os.rename with no such file or directory : "
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2016-11-04.log.html#t2016-11-04T09:59:42
logs copied from http://paste.openstack.org/show/587879/ :
Nov 4 11:26:31 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:26:31 STACO4 object-auditor: ERROR DiskFile /srv/node/s04z4ecd05/objects-1/175477/a2d/ab5d60b03afec2e9a0861b6817f14a2d/1470560571.46381#2.data close failure: [Errno 2] No such file or directory : File "/usr/bin/swift-object-auditor", line 29, in <module>#012 run_daemon(ObjectAuditor, conf_file, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 110, in run_daemon#012 klass(conf).run(once=once, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 57, in run#012 self.run_forever(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 393, in run_forever#012 self.audit_loop(parent, zbo_fps, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 367, in audit_loop#012 pids.append(self.fork_child(**kwargs))#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 314, in fork_child#012 self.run_audit(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 302, in run_audit#012 worker.audit_all_objects(mode=mode, device_dirs=device_dirs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 129, in audit_all_objects#012 self.failsafe_object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1600, in __iter__#012 self.close()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1800, in close#012 {'exc': e, 'stack': ''.join(traceback.format_stack()),#012
Nov 4 11:27:09 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735/1468760857.79624#5.data: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:09 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd03/objects-1/182607/735/b253de29d4eade286b440605b3243735 failed audit and was quarantined: Invalid EC metadata at offset 0x1c76e
Nov 4 11:27:58 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c/1473894901.93179#7.data: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c failed audit and was quarantined: Invalid EC metadata at offset 0x0
Nov 4 11:27:58 STACO4 object-auditor: ERROR Trying to audit /srv/node/s04z4ecd05/objects-1/175696/e8c/ab941e0c6512bd5ef51294ac96a6ee8c: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 212, in failsafe_object_audit#012 self.object_audit(location)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 244, in object_audit#012 for chunk in reader:#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1586, in __iter__#012 self._update_checks(chunk)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2581, in _update_checks#012 self._check_frag(self.frag_buf[cursor:cursor + self.frag_size])#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 2562, in _check_frag#012 self._quarantine(msg)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1762, in _quarantine#012 self._data_file)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 3295, in run_in_thread#012 result = func(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 455, in wrapper#012 return f(*args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 220, in quarantine_renamer#012 renamer(from_dir, to_dir, fsync=False)#012 File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer#012 os.rename(old, new)#012OSError: [Errno 2] No such file or directory
Nov 4 11:29:00 STACO4 object-auditor: Quarantined object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61/1471680446.72800#1.data: Invalid EC metadata at offset 0x0
Nov 4 11:29:00 STACO4 object-auditor: ERROR Object /srv/node/s04z4ecd04/objects-1/175123/d61/ab04ee3d5ed5eb9ea95dde279c595d61 failed audit and was quarantined: Invalid EC metadata at offset 0x0
``` |
|
| 2016-11-04 14:00:25 |
Alistair Coles |
swift: importance |
Undecided |
Medium |
|
| 2016-11-04 14:00:32 |
Alistair Coles |
swift: importance |
Medium |
Undecided |
|
