Daisy

Don't ack the whole queue if multiple core files can't be found

Bug #1983735 reported by Brian Murray
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Daisy
Triaged
High
Unassigned

Bug Description

During an update of the apport code in the Error Tracker there was an issue where swift was not configured correctly and consequently the retracers were not able to find the core files in swift. Due to the following code in retracer.py the whole queue ended up getting ack'ed and drained.

        if path == "Missing":
            log("Ack'ing OOPS with missing core file.")
            msg.channel.basic_ack(msg.delivery_tag)
            return

IIRC this code exists because it is possible for a crash to reported but the core file not be uploaded e.g. the transfer was interrupted or it failed to write to swift. Regardless there should probably be a counter associated with the quantity of missing core files and if we hit some reasonable number in a row we just exit the retracing process until the situation has been manually investigated.

Brian Murray (brian-murray)
Changed in daisy:
importance: Undecided → High
status: New → Triaged
summary: - Don't ack the whole queue if you can't find multiple core files
+ Don't ack the whole queue if multiple core file can't be found
summary: - Don't ack the whole queue if multiple core file can't be found
+ Don't ack the whole queue if multiple core files can't be found
Revision history for this message
Brian Murray (brian-murray) wrote :

For reference here is a log file of the event.

2022-08-03 14:08:41,248:13590:139900384646976:INFO:root:8291666c-1327-11ed-96fc-fa163e993415:swift:Could not retrieve 8291666c-1327-11ed-96fc-fa163e993415 (swift):
2022-08-03 14:08:41,248:13590:139900384646976:INFO:root:8291666c-1327-11ed-96fc-fa163e993415:swift:Traceback (most recent call last):
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 423, in write_swift_bucket_to_disk
    resp_chunk_size=65536)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1799, in get_object
    headers=headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1679, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1631, in get_auth
    timeout=self.timeout)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 676, in get_auth
    raise ClientException('No project name or project id specified.')
ClientException: No project name or project id specified.

2022-08-03 14:08:41,248:13590:139900384646976:INFO:root:8291666c-1327-11ed-96fc-fa163e993415:swift:Could not find None
2022-08-03 14:08:41,248:13590:139900384646976:INFO:root:8291666c-1327-11ed-96fc-fa163e993415:swift:Could not remove 8291666c-1327-11ed-96fc-fa163e993415 (swift):
2022-08-03 14:08:41,248:13590:139900384646976:INFO:root:8291666c-1327-11ed-96fc-fa163e993415:swift:Traceback (most recent call last):
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 468, in remove_from_swift
    _cached_swift.delete_object(bucket, key)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1862, in delete_object
    headers=headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1679, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1631, in get_auth
    timeout=self.timeout)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 676, in get_auth
    raise ClientException('No project name or project id specified.')
ClientException: No project name or project id specified.

2022-08-03 14:08:41,248:13590:139900384646976:INFO:root:8291666c-1327-11ed-96fc-fa163e993415:swift:Requeued failed to process OOPS (8291666c-1327-11ed-96fc-fa163e993415)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.