Image fetch speed from Glance to Cinder volume service can be slow and capped by client CPU

Bug #2020139 reported by Nobuto Murata
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance Client
Fix Committed
Undecided
Unassigned

Bug Description

When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).

The image fetch speed is not super fast as of today:

2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s

-> ~ 549 Mbps

By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).

Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html

Revision history for this message
Nobuto Murata (nobuto) wrote :

By changing the CHUNKSIZE from 64 KiB to 1 MiB, it looks like the CPU was used more efficiently and it does the trick.

2023-05-17 11:54:23.080 376143 DEBUG cinder.image.image_utils [req-8228848b-f037-4a6a-802e-de7ffe62bae6 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details:
 dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_gnmu2b9ycinder@cinder-ceph, size 1907.35 MB, duration 14.44 sec fetch /usr/lib/python3/dist-packages/cinder/image/i
mage_utils.py:639
2023-05-17 11:54:23.080 376143 INFO cinder.image.image_utils [req-8228848b-f037-4a6a-802e-de7ffe62bae6 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.3
5 MB at 132.10 MB/s

-> 1,057 Mbps

`openstack image save` with openstacksdk has a similar issue so I reported it separately as:
https://storyboard.openstack.org/#!/story/2010759

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-glanceclient (master)
Changed in python-glanceclient:
status: New → In Progress
Nobuto Murata (nobuto)
description: updated
Revision history for this message
Nobuto Murata (nobuto) wrote :
Changed in python-glanceclient:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-glanceclient 4.4.0

This issue was fixed in the openstack/python-glanceclient 4.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.