Volume to Image upload very slow due to use of tpool.Proxy()

In our environment, we observed that the speed of a volume upload to a Glance image decreases very quickly over the course of an upload.

The upload starts at around 300mb/s but drops below 50mb/s within the first gigabyte and becomes even slower as it progresses. Using the patch in the Glance client, which disables chunked endcoding, the performance could be improved a bit. However, it was still too slow. (https://review.opendev.org/c/openstack/python-glanceclient/+/733984)

In cinder/image/image_utils.py, the FileHandler is wrapped in a tpool.Proxy(). However, if you instead run the entire image_service.update call with tpool.execute() and remove the tpool.Proxy(), you get very good performance. Like constant 700mb/s and more. But this works only in combination with the above mentioned patch for the Glance client. Otherwise you see again the behavior described above, that the speed decreases a lot during the upload.

So replace this:
image_service.update(context, image_id, {},
                     tpool.Proxy(image_file),
                     store_id=store_id,
                     base_image_ref=base_image_ref)

with this:
tpool.execute(image_service.update, context, image_id, {},
              image_file, store_id=store_id,
              base_image_ref=base_image_ref)

We use cinder and glance with the Yoga release. For Cinder we use the Netapp NFS backend and Glance with the S3 backend.