OpenStack Compute (nova)

Nova-compute wouldn't retry image download when gets "Corrupt image download" error

Series victoria
Bug #1950657

Bug #1950657 reported by Stanislav Dmitriev on 2021-11-11

This bug affects 7 people

	Status	Importance	Assigned to
OpenStack Compute (nova)	Fix Released	Undecided	Stanislav Dmitriev
Victoria	In Progress	Undecided	Unassigned
Wallaby	In Progress	Undecided	Unassigned
Xena	Fix Released	Undecided	Unassigned

Bug Description

Nova-compute wouldn't retry image download when gets "Corrupt image download" error from glanceclient.

There is a configuration option num_retries (equal 3 by default) in glance section of nova-config file, so nova-compute supposed to retry image download if it failed, but it wouldn't work in case of next exception:

2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in image_chunks:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in self.iterable:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] (computed, hash_value))
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]

It wouldn't work because IOError exception is not in retry_excs list:
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179

        retry_excs = (glanceclient.exc.ServiceUnavailable,
                glanceclient.exc.InvalidEndpoint,
                glanceclient.exc.CommunicationError)

so try-except block doesn't catch and download retry never happens

Martin Chlumsky (martin-chlumsky) on 2021-11-11

Changed in nova:
status:	New → Confirmed

Benjamin FÜHRMANN (fuhrmannb) on 2021-11-12

summary:	Nova-compute wouldn't retry image download when gets "Corrupt image - download" error + download" erro
summary:	Nova-compute wouldn't retry image download when gets "Corrupt image - download" erro + download" error

Stanislav Dmitriev (sdmitriev1) on 2021-11-19

Changed in nova:
assignee:	nobody → S.A.Dmitriev (sdmitriev1)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-19: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/818503

Changed in nova:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-25: Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/818503
Committed: https://opendev.org/openstack/nova/commit/ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Submitter: "Zuul (22348)"
Branch: master

commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Author: sdmitriev1 <email address hidden>
Date: Thu Nov 18 22:05:05 2021 -0500

Retry image download if it's corrupted

    Adding IOError in list of catching exceptions in order to
    fix behavior when nova-compute wouldn't retry image download
    when got "Corrupt image download" error from glanceclient
    and had num_retries config option set.

Closes-Bug: #1950657
Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-25: Fix proposed to nova (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/819179

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-25: Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/819180

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-25: Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/819181

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-02-27: Fix merged to nova (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/nova/+/819179
Committed: https://opendev.org/openstack/nova/commit/b44ec0dc4927046d15525af78ab9d534cea9ce69
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b44ec0dc4927046d15525af78ab9d534cea9ce69
Author: sdmitriev1 <email address hidden>
Date: Thu Nov 18 22:05:05 2021 -0500

Retry image download if it's corrupted

    Closes-Bug: #1950657
    Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0
    (cherry picked from commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39)