Cinder

Glance download with allowed_direct_url_schemes sometimes fails with Python 3

Bug #1923313 reported by MarkMielke on 2021-04-10

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Cinder	Fix Released	High	Eric Harney	Cinder xena-1
Train	New	Undecided	Unassigned	Cinder 15.6.0 "train-final"
Ussuri	Fix Committed	Undecided	Unassigned	Cinder 16.4.0
Victoria	New	Undecided	Unassigned	Cinder 17.2.0
Wallaby	Fix Committed	Undecided	Unassigned	Cinder 18.1.0
Xena	Fix Released	High	Eric Harney	Cinder xena-1

Bug Description

This is similar to 1775678 and a few others. The code in GlanceImageService.download() (cinder/image/glance.py) specifies:

with open(parsed_url.path, "r") as f:
shutil.copyfileobj(f, data)

This code only takes effect if "file" is specified for allowed_direct_url_schemes.

This results in hard to debug character set errors during image download. Luckily, it seems that the image usually fails to download with an obscure error rather than becoming corrupt.

This should have been caught with the move to Python 3. Python 3 requires "rb" when opening binary files. "r" opens in text mode. It may be that allowed_direct_url_scheme is not used by very many people. However, I still use it, and I have been holding onto a local patch ("r" => "rb") since Train which resolves the problem every time. Opening this issue to start the process of getting the fix posted.

See original description

Tags:

MarkMielke (mark-mielke) on 2021-04-10

description:

updated

Revision history for this message

Eric Harney (eharney) wrote on 2021-04-12:

https://review.opendev.org/c/openstack/cinder/+/785878

Changed in cinder:
status:	New → In Progress
assignee:	nobody → Eric Harney (eharney)

Revision history for this message

Brian Rosmaita (brian-rosmaita) wrote on 2021-04-16:

@Mark: Thanks for catching this and for the detailed report!

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-19: Fix proposed to cinder (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/786922

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-28: Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/785878
Committed: https://opendev.org/openstack/cinder/commit/5d7e7547f0119300ecb8f6cd171b04ead3210ecc
Submitter: "Zuul (22348)"
Branch: master

commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

Open local image files with "rb" mode

When using "file" in allowed_direct_url_schemes,
open image files w/ binary mode.

Closes-Bug: #1923313
Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix merged to cinder (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/786922
Committed: https://opendev.org/openstack/cinder/commit/c16567bd8784761d24b73678377994903b8dc35b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c16567bd8784761d24b73678377994903b8dc35b
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

Open local image files with "rb" mode

When using "file" in allowed_direct_url_schemes,
open image files w/ binary mode.

    Closes-Bug: #1923313
    Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42
    (cherry picked from commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-26: Fix proposed to cinder (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/793169

Sofia Enriquez (lsofia-enriquez) on 2021-06-03

tags:

added: files glance python3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-25: Fix merged to cinder (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/793169
Committed: https://opendev.org/openstack/cinder/commit/f6d7b09195ce738576252736708a7f3417f761db
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f6d7b09195ce738576252736708a7f3417f761db
Author: Eric Harney <email address hidden>
Date: Mon Apr 12 09:24:47 2021 -0400

Open local image files with "rb" mode

When using "file" in allowed_direct_url_schemes,
open image files w/ binary mode.

    Closes-Bug: #1923313
    Change-Id: I52b87d03c49db74002f09cfeea86bc0ac7ce8a42
    (cherry picked from commit 5d7e7547f0119300ecb8f6cd171b04ead3210ecc)
    (cherry picked from commit c16567bd8784761d24b73678377994903b8dc35b)