s3: Large images uploaded but then deleted
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Glance |
New
|
Undecided
|
Unassigned | ||
| glance_store |
New
|
Undecided
|
Unassigned | ||
Bug Description
We are trying to upload images to an on-premise minio-based S3 compatible storage cluster. We have been using it for ~30GB images for months. However when increasing the size to >30GB, the image is successfully uploaded and then glance seems to tell it to delete.
Successful log looks like this:
Sep 6, 2022 @ 08:40:22.812 1 1 rPR1E4MBBdpF5mC
Sep 6, 2022 @ 08:39:39.038 1 1 q_R1E4MBBdpF5mC
...
Whole bunch of PutObjectPart...
Failure log looks like this:
Sep 2, 2022 @ 18:11:46.067 1 1 PvTnAIMBBdpF5mC
Sep 2, 2022 @ 18:11:45.987 1 1 PfTnAIMBBdpF5mC
Sep 2, 2022 @ 18:11:45.927 1 1 PPTnAIMBBdpF5mC
Sep 2, 2022 @ 18:10:49.370 1 1 O_TmAIMBBdpF5mC
We were able to get the 38GB image to work by increasing the chunk size from 10MB to 100MB. However this does not help the 50GB or 100GB.
The storage system is correctly responding to all S3 request with either 200 or 204 replies. It appears that something important is happening with the HeadObject which is then causing glance to send a DeleteObject.
I've attached shortened versions of the S3 logs - I've deleted the majority of PutObjectPart entries to keep the size down.
We're not finding any information. What is causing this behavior?
| Thaddeus Fortenberry (thaddeus2715) wrote | #1 |
| Brian Rosmaita (brian-rosmaita) wrote | #2 |
| summary: |
- Large images uploaded but then deleted + s3: Large images uploaded but then deleted |
I suspect this is a glance_store issue, but keeping both components until more research is done.