| 2021-09-29 14:49:52 |
Florian Engelmann |
bug |
|
|
added bug |
| 2021-09-29 15:16:34 |
Florian Engelmann |
description |
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/bc290840127c3179227a662584404f9c0178d588
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack@g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume but we should also remove those properties when a volume is uploaded to glance. |
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/dda179d3f901e4f23091f3095f1af58bc26e222e
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack@g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume but we should also remove those properties when a volume is uploaded to glance. |
|
| 2021-10-06 13:58:18 |
OpenStack Infra |
cinder: status |
New |
In Progress |
|
| 2021-10-06 14:03:37 |
Sofia Enriquez |
cinder: importance |
Undecided |
Medium |
|
| 2021-10-06 14:06:29 |
Sofia Enriquez |
cinder: status |
In Progress |
Incomplete |
|
| 2021-10-06 14:09:13 |
Sofia Enriquez |
summary |
filter reserved image properties |
[stable/wallaby] filter reserved image properties |
|
| 2021-10-06 18:01:22 |
OpenStack Infra |
cinder: status |
Incomplete |
In Progress |
|
| 2021-11-02 22:44:50 |
Sofia Enriquez |
summary |
[stable/wallaby] filter reserved image properties |
It's not possible to upload a volume that was build from an image back to glance, if multistore (glance) is enabled. |
|
| 2021-11-02 22:45:12 |
Sofia Enriquez |
tags |
|
glance image multistore upload-to-image |
|
| 2022-01-24 10:19:10 |
Marius L |
bug |
|
|
added subscriber Marius L |
| 2022-03-17 12:11:29 |
Thobias Trevisan |
bug |
|
|
added subscriber Thobias Trevisan |
| 2022-04-11 21:02:03 |
Dan Hill |
tags |
glance image multistore upload-to-image |
glance image multistore sts upload-to-image |
|
| 2022-04-11 21:02:14 |
Dan Hill |
bug |
|
|
added subscriber Dan Hill |
| 2023-02-01 21:44:28 |
OpenStack Infra |
cinder: status |
In Progress |
Fix Released |
|
| 2023-02-08 17:00:52 |
OpenStack Infra |
tags |
glance image multistore sts upload-to-image |
glance image in-stable-zed multistore sts upload-to-image |
|
| 2023-02-23 02:22:45 |
OpenStack Infra |
tags |
glance image in-stable-zed multistore sts upload-to-image |
glance image in-stable-yoga in-stable-zed multistore sts upload-to-image |
|
| 2023-03-13 15:51:50 |
Rodrigo Barbieri |
summary |
It's not possible to upload a volume that was build from an image back to glance, if multistore (glance) is enabled. |
[SRU] It's not possible to upload a volume that was build from an image back to glance, if multistore (glance) is enabled. |
|
| 2023-03-13 15:55:19 |
Rodrigo Barbieri |
description |
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/dda179d3f901e4f23091f3095f1af58bc26e222e
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack@g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume but we should also remove those properties when a volume is uploaded to glance. |
* SRU TEMPLATE AT THE BOTTOM *
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/dda179d3f901e4f23091f3095f1af58bc26e222e
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack@g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume but we should also remove those properties when a volume is uploaded to glance.
============
SRU TEMPLATE
============
[Impact]
The issue impacts workflows when downloading an image from glance and then uploading it back to glance, because of the extra metadata. The easy workaround is to manually delete the metadata for every image downloaded from glance prior to uploading the volumes. The fix changed code only on the upload-volume-to-image API to not submit the metadata back. Additionally, the behavior can be controlled through a config option.
[TestCase]
1. Setting up env
1a. Deploy an environment with more than 1 store for glance, such as ceph + swift
1b. Upload cirros image to glance
1c. Add the cirros image to swift store
glance image-import <image-id> --stores swift --import-method copy-image
1d. List images including store to confirm
glance image-list --include-store
1e. Create a volume using the image
openstack volume create --size 1 --image <image-id/name> testvol
1f. Confirm the "os_glance..." image metadata is in the volume
openstack volume show testvol
2. Reproducing the issue
openstack image create --volume testvol --disk-format raw image-from-vol-from-image
Result should be:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 400)
3. Cleanup not needed
4. Install package that contains the fixed code
5. Repeat command in (2), result should now succeed.
[Regression Potential]
Fix has been tested in the Upstream Cinder CI (not specific scenario) and through unit tests. Behavior is configurable through config option. In case the code for the upload-volume-to-image API breaks, then there is no other way to upload a volume to an image. |
|
| 2023-03-14 16:01:51 |
Rodrigo Barbieri |
bug task added |
|
cinder (Ubuntu) |
|
| 2023-03-14 16:02:47 |
Rodrigo Barbieri |
nominated for series |
|
Ubuntu Jammy |
|
| 2023-03-14 16:02:47 |
Rodrigo Barbieri |
bug task added |
|
cinder (Ubuntu Jammy) |
|
| 2023-03-14 16:02:47 |
Rodrigo Barbieri |
nominated for series |
|
Ubuntu Kinetic |
|
| 2023-03-14 16:02:47 |
Rodrigo Barbieri |
bug task added |
|
cinder (Ubuntu Kinetic) |
|
| 2023-03-14 16:03:25 |
Rodrigo Barbieri |
bug task added |
|
cloud-archive |
|
| 2023-03-14 16:03:38 |
Rodrigo Barbieri |
nominated for series |
|
cloud-archive/yoga |
|
| 2023-03-14 16:03:38 |
Rodrigo Barbieri |
bug task added |
|
cloud-archive/yoga |
|
| 2023-03-14 16:03:38 |
Rodrigo Barbieri |
nominated for series |
|
cloud-archive/zed |
|
| 2023-03-14 16:03:38 |
Rodrigo Barbieri |
bug task added |
|
cloud-archive/zed |
|
| 2023-03-14 16:14:26 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~rodrigo-barbieri2010/ubuntu/+source/cinder/+git/cinder/+merge/438901 |
|
| 2023-03-14 16:15:58 |
Rodrigo Barbieri |
tags |
glance image in-stable-yoga in-stable-zed multistore sts upload-to-image |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image |
|
| 2023-03-14 16:16:20 |
Rodrigo Barbieri |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2023-03-14 16:23:14 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~rodrigo-barbieri2010/ubuntu/+source/cinder/+git/cinder/+merge/438903 |
|
| 2023-03-21 15:03:47 |
Corey Bryant |
cloud-archive: status |
New |
Fix Released |
|
| 2023-03-21 15:03:50 |
Corey Bryant |
cinder (Ubuntu): status |
New |
Fix Released |
|
| 2023-03-21 15:03:55 |
Corey Bryant |
cloud-archive/zed: status |
New |
Triaged |
|
| 2023-03-21 15:04:10 |
Corey Bryant |
cinder (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2023-03-21 15:04:16 |
Corey Bryant |
cinder (Ubuntu Kinetic): importance |
Undecided |
High |
|
| 2023-03-21 15:04:21 |
Corey Bryant |
cloud-archive/yoga: status |
New |
Triaged |
|
| 2023-03-21 15:04:26 |
Corey Bryant |
cloud-archive/yoga: importance |
Undecided |
High |
|
| 2023-03-21 15:04:33 |
Corey Bryant |
cloud-archive/zed: importance |
Undecided |
High |
|
| 2023-03-21 15:04:38 |
Corey Bryant |
cinder (Ubuntu Kinetic): status |
New |
Triaged |
|
| 2023-03-21 15:04:40 |
Corey Bryant |
cinder (Ubuntu Jammy): status |
New |
Triaged |
|
| 2023-04-05 13:39:03 |
Robie Basak |
cinder (Ubuntu Kinetic): status |
Triaged |
Fix Committed |
|
| 2023-04-05 13:39:07 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2023-04-05 13:39:12 |
Robie Basak |
tags |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-kinetic |
|
| 2023-04-05 13:39:32 |
Robie Basak |
cinder (Ubuntu Jammy): status |
Triaged |
Fix Committed |
|
| 2023-04-05 13:39:39 |
Robie Basak |
tags |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-kinetic |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic |
|
| 2023-04-10 14:18:01 |
Corey Bryant |
cloud-archive/zed: status |
Triaged |
Fix Committed |
|
| 2023-04-10 14:18:04 |
Corey Bryant |
tags |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic verification-zed-needed |
|
| 2023-04-10 15:13:21 |
Corey Bryant |
cloud-archive/yoga: status |
Triaged |
Fix Committed |
|
| 2023-04-10 15:13:24 |
Corey Bryant |
tags |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic verification-zed-needed |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic verification-yoga-needed verification-zed-needed |
|
| 2023-04-13 17:05:05 |
Brian Rosmaita |
cinder: status |
Fix Released |
Triaged |
|
| 2023-04-13 17:05:35 |
Brian Rosmaita |
cloud-archive/yoga: status |
Fix Committed |
In Progress |
|
| 2023-04-13 17:05:47 |
Brian Rosmaita |
cloud-archive/zed: status |
Fix Committed |
In Progress |
|
| 2023-04-14 15:08:22 |
Rodrigo Barbieri |
tags |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic verification-yoga-needed verification-zed-needed |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-failed verification-failed-jammy verification-failed-kinetic verification-yoga-failed verification-zed-failed |
|
| 2023-04-28 15:37:22 |
Rodrigo Barbieri |
tags |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-failed verification-failed-jammy verification-failed-kinetic verification-yoga-failed verification-zed-failed |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic verification-yoga-needed verification-zed-needed |
|
| 2023-05-02 15:46:10 |
OpenStack Infra |
cloud-archive/zed: status |
In Progress |
Fix Released |
|
| 2023-05-02 15:50:33 |
OpenStack Infra |
cloud-archive/yoga: status |
In Progress |
Fix Released |
|
| 2023-05-02 16:43:06 |
Rodrigo Barbieri |
description |
* SRU TEMPLATE AT THE BOTTOM *
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/dda179d3f901e4f23091f3095f1af58bc26e222e
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack@g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume but we should also remove those properties when a volume is uploaded to glance.
============
SRU TEMPLATE
============
[Impact]
The issue impacts workflows when downloading an image from glance and then uploading it back to glance, because of the extra metadata. The easy workaround is to manually delete the metadata for every image downloaded from glance prior to uploading the volumes. The fix changed code only on the upload-volume-to-image API to not submit the metadata back. Additionally, the behavior can be controlled through a config option.
[TestCase]
1. Setting up env
1a. Deploy an environment with more than 1 store for glance, such as ceph + swift
1b. Upload cirros image to glance
1c. Add the cirros image to swift store
glance image-import <image-id> --stores swift --import-method copy-image
1d. List images including store to confirm
glance image-list --include-store
1e. Create a volume using the image
openstack volume create --size 1 --image <image-id/name> testvol
1f. Confirm the "os_glance..." image metadata is in the volume
openstack volume show testvol
2. Reproducing the issue
openstack image create --volume testvol --disk-format raw image-from-vol-from-image
Result should be:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 400)
3. Cleanup not needed
4. Install package that contains the fixed code
5. Repeat command in (2), result should now succeed.
[Regression Potential]
Fix has been tested in the Upstream Cinder CI (not specific scenario) and through unit tests. Behavior is configurable through config option. In case the code for the upload-volume-to-image API breaks, then there is no other way to upload a volume to an image. |
* SRU TEMPLATE AT THE BOTTOM *
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/dda179d3f901e4f23091f3095f1af58bc26e222e
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack@g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume but we should also remove those properties when a volume is uploaded to glance.
============
SRU TEMPLATE
============
[Impact]
The issue impacts workflows when downloading an image from glance and then uploading it back to glance, because of the extra metadata. The easy workaround is to manually delete the metadata for every image downloaded from glance prior to uploading the volumes. The fix changed code only on the upload-volume-to-image API to not submit the metadata back. Additionally, the behavior can be controlled through a config option.
[TestCase]
1. Setting up env
1a. Deploy an environment with more than 1 store for glance, such as ceph + swift
1b. Upload cirros image to glance
1c. Add the cirros image to swift store
glance image-import <image-id> --stores swift --import-method copy-image
1d. List images including store to confirm
glance image-list --include-store
1e. Create a volume using the image
openstack volume create --size 1 --image <image-id/name> testvol
1f. Confirm the "os_glance..." image metadata is in the volume
openstack volume show testvol
2. Reproducing the issue
openstack image create --volume testvol --disk-format raw image-from-vol-from-image
Result should be:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 400)
3. Cleanup not needed
4. Install package that contains the fixed code
5. Adjust cinder.conf as a workaround to issue in comment #21
glance_core_properties = checksum, container_format, disk_format, image_name, image_id, min_disk, min_ram, name, size, os_glance_failed_import,os_glance_importing_to_stores
6. Restart cinder services if needed
7. Repeat command in (2), result should now succeed.
[Regression Potential]
Fix has been tested in the Upstream Cinder CI (not specific scenario) and through unit tests. Behavior is configurable through config option. In case the code for the upload-volume-to-image API breaks, then there is no other way to upload a volume to an image. |
|
| 2023-05-02 17:07:13 |
OpenStack Infra |
cinder: status |
Triaged |
In Progress |
|
| 2023-05-08 14:01:03 |
Rodrigo Barbieri |
tags |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-needed verification-needed-jammy verification-needed-kinetic verification-yoga-needed verification-zed-needed |
glance image in-stable-yoga in-stable-zed multistore sts sts-sru-needed upload-to-image verification-failed verification-failed-jammy verification-failed-kinetic verification-yoga-failed verification-zed-failed |
|
| 2023-05-11 18:00:55 |
Launchpad Janitor |
cinder (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2023-05-11 18:00:55 |
Launchpad Janitor |
cve linked |
|
2023-2088 |
|
| 2023-05-11 18:13:54 |
Launchpad Janitor |
cinder (Ubuntu Kinetic): status |
Fix Committed |
Fix Released |
|
| 2023-06-02 19:15:42 |
OpenStack Infra |
cinder: status |
In Progress |
Fix Released |
|
