Missing [nova] section in cinder.conf "Extend an Attached Volume" doesn't work with Victoria onwards

So it looks like the [nova] section was added a while back, and the config keys that are used were removed from cinder in the train release[1]. The commit that adds the [nova] section [2].

Were going to have to add a [nova] section to the cinder.conf (as the bug reporter indicates) with the relevant details. This should be backported to the Train release.

1. https://docs.openstack.org/releasenotes/puppet-cinder/train.html#relnotes-15-4-0-stable-train
2. https://review.opendev.org/c/openstack/cinder/+/460640

Could you provide instructions to reproduce the issue?

I tried the following on focal-xena to create a volume, attach it to an instance, then resize it:

```
openstack volume create --size 1 vol1
openstack server add volume juju-7a66ea-k8s-on-openstack-0 vol1
cinder reset-state --state available vol1
openstack volume set vol1 --size 2
```

This completed successfully without errors (at least, the commands returned successfully, no errors in /var/log/cinder/cinder-*.log).

(This doesn't negate the fact that we still need to add support for the [nova] section, but I'm trying to figure out how to test it.)

I've added a snippet from the other bug filed at the same time:
https://bugs.launchpad.net/ubuntu/+source/python-os-brick/+bug/1939390
> Steps to reproduce:
>
> openstack volume create --type my-iscsi-volume-type --size 10 test-resize
> openstack server add volume test-server test-resize
> openstack volume set --size 20 test-resize

I suppose the key to reproduce it is either online resizing or using iSCSI backend. But if it's the former, it would be easier to test it with Ceph RBD backend.

Hi Samuel,

The volume needs to be "in-use". I see you are resetting it's state to "available". When in-use, cinder will attempt to notify nova to refresh the size of the disk. See:

https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L2939-L2943

On iSCSI backends the volume needs to first be refreshed on the host before the VM can be made aware of the new size. I suspect there may be some operation that needs to be done on RBD backends as well, but I cannot say for sure.

To reproduce, I recommend you use any iSCSI backend. We used a Dell Unity, but any iSCSI backend will do.

Thanks Nobuto, Gabriel!

> The volume needs to be "in-use". I see you are resetting it's state to "available".

Ah that makes more sense. When I was testing, openstack refused to change the volume size while in-use: "Volume is in in-use state, it must be available before size can be extended". Will need to try a different backend.

https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html#extend-attached-volume
`--os-volume-api-version 3.42` or later was the key to allow resize for the record.

So I tested with cinder-lvm, which using an iSCSI backend, and couldn't reproduce the issue. I then dug into the code and discovered that cinder appears to try to autodetect the keystone auth parameters if it doesn't find data in the [nova] section. (ref. code here: https://opendev.org/openstack/cinder/src/commit/85387cae8b3641d6d193c6550edbfb0de1d8384e/cinder/compute/nova.py#L98-L106 ).

So it may be difficult to reproduce the exact issue that Gabriel reported without knowing more about the openstack configuration (eg. regions, etc.).

However, I think this is missing the point, which is that the [nova] section is indeed missing and not configurable from juju for cinder.conf (and absolutely appears to be what is causing the bug). I'm not sure how we want this section to be configurable though. There are a lot of options:

[nova]
# locally declared in cinder:compute/nova.py
region_name =
interface =
token_auth_url =

# declared in keystoneauth1:loading/session.py
cafile =
certfile =
keyfile =
insecure =
timeout =
collect-timing =
split-loggers =

# declared in keystoneauth1:loading/conf.py
auth_type =
auth_section =

---

Which do we want to expose as-is to the charm config, vs exposing a higher level config option, vs other methods?

The deployment is single region, with OpenStack Victoria. The config, apart from the [nova] section I added, is the one generated by the juju charms.

I created a template for victoria, which is a copy o the Queens config, with the following section added:

[nova]
{% include "parts/keystone-authtoken" %}

Ant the contents of that parts file is:

# Authentication type to load (string value)
auth_type = password

# Authentication URL (string value)
auth_url = {{ auth_protocol }}://{{ auth_host }}:{{ auth_port }}

# Username (string value)
username = {{ admin_user }}

# User's password (string value)
password = {{ admin_password }}

# Project name to scope to (string value)
project_name = {{ admin_tenant_name }}

{% if admin_domain_name -%}
project_domain_name = {{ admin_domain_name }}
user_domain_name = {{ admin_domain_name }}
{% else -%}
project_domain_name = default
user_domain_name = default
{% endif -%}

{% if ca_cert_path -%}
# PEM encoded Certificate Authority to use when verifying
# HTTPs connections. (string value)
cafile = {{ ca_cert_path }}
{% endif -%}

region_name = {{ region }}

After a charm upgrade, the error was gone, and volumes were properly refreshed. Not sure if there is anything that is missing.

Update: have successfully reproduced this bug on Victoria and Wallaby. However I couldn't reproduce it on Xena.

Samuel has reproduced the issue on Xena finally so we will take a look when we have some cycles.

I've been trying to figure out a solution here.
We need to somehow get `project_domain_id` or `project_domain` into the context here:

2022-04-12 23:43:07.956 151261 WARNING cinder.compute.nova [req-a23f3da3-3004-4798-b248-3348fe1309ef ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -] XXX context = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7WiqLp7P1vSxN27E', 'request_id': 'req-a23f3da3-3004-4798-b248-3348fe1309ef', 'global_request_id': 'req-a718a415-0414-4a86-b4bd-87c8f5c832e4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin', 'admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:07.378961+00:00', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}]}

I could trace the context as far back as in oslo messaging rpc message received:

2022-04-12 23:43:07.506 151261 WARNING oslo_messaging.rpc.server [-] ZZZ message.ctxt = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7W...

In the above mentioned template, there is a "project_domain_name" and "user_domain_name" set if "admin_domain_name" is set. With the project_domain_name set, the keystone client used by the nova client should have the proper context.

I would suggest you do a:

charm pull cinder

add the template mentioned above, then do an upgrade of the charm using the locally modified version.

After hooks run, you should have the proper nova section with the needed settings, and the error should no longer manifest, as the nova client will have proper credentials.

Adding a bit of context.

All OpenStack projects use the same oslo-config boilerplate to provide credentials for the various clients a service may use. In this case, Cinder needs to talk to Nova, so a nova client needs to be instantiated. The sections for the various clients are identical. The only difference is the group name (eg: [nova], [neutron], etc). These individual sections are so alike, that in more recent versions of OpenStack, you can chose to either customize them (you may want a different Region for one of the clients), or use the "auth_section" option.

example:

[nova]
auth_section = "keystone_auth"

[keystone_auth]
.... The template contents from previous comment .....

In the cinder code, this is used here (for nova at least):

https://github.com/openstack/cinder/blob/master/cinder/compute/nova.py#L115-L129

A keystone session is created with the credentials from [nova] and that session is passed into the nova client.

In general, if a service needs to talk to another service, for whatever reason, a proper [<service_name>] section needs to exist in the config with the proper keystone credentials (identical most times to all other services).

Ah thanks for the context Gabriel! That is something I had no idea about, and haven't read that in the documentation anywhere.

I'm still a little hazy on exactly what modifications you made; could you share a diff?

Something that bugs me though, is that it's not missing any auth params except for the project domain id. Why is it not getting the project domain id for this case only? It expects to receive it in the context, so it can use it here: https://opendev.org/openstack/cinder/src/commit/1eb7384cb6542784a904891666cca4abbe42841d/cinder/compute/nova.py#L107-L110

My pleasure! :)

That particular bit of code, should not be hit, unless you don't have "auth_type" set in the config. The cinder user *should* be privileged I think.

Here is a quick diff of the stuff I posted in comment #9:

https://github.com/gabriel-samfira/charm-cinder/commit/82505257b129db91b1aabeac744e02c900852287

Give this a shot in your env, see if it fixes it. I wrote this a while ago. If the charm template context changed, you may need to adapt it. A relation to keystone should ensure that context exists.

Update: I managed to successfully reproduce the issue and Gabriel's fix by cherry picking https://github.com/gabriel-samfira/charm-cinder/commit/82505257b129db91b1aabeac744e02c900852287 on stable/xena. But now I have two questions:

- With the fix there is no more traceback or crash, but I'm not sure I can confirm it's actually successfully notified the instance. If I run `lsblk` on the openstack instance after live resizing the instance, it doesn't reflect the updated size. It does show the updated size if I reboot the openstack instance though.

- Is this how we want to fix the issue? It seems that if it does support the `auth_section = "SECTION_NAME"` format, this would result in less duplication. However I couldn't get it to work that way, and I can't find docs about how this is supposed to work. Also https://github.com/openstack/cinder/blob/master/cinder/compute/nova.py#L115-L129 directly accesses the nova group from config; I don't see if/where it resolves through an `auth_section` key.

Hi Samuel,

Would you mind sharing the output of lsblk and fdisk -l before and after live resizing the disk?

On the VM, after resize, could you rescan the disk by doing a:

echo 1>/sys/class/block/sdx/device/rescan

where sdx is youd block device.

In regards to the second point, I would simply use the same keystone auth "part" in that patch. Other charms have similar chunks of template that get added to the configs. It may be more verbose, but Feel free to use whichever approach seems better to you.

Opened https://review.opendev.org/c/openstack/charm-cinder/+/842902

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/842902
Committed: https://opendev.org/openstack/charm-cinder/commit/024de3783fc459fc25596ec6084dd065e945df76
Submitter: "Zuul (22348)"
Branch: master

commit 024de3783fc459fc25596ec6084dd065e945df76
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

    Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

    This issue has been observed on victoria and newer,
    hence the addition of cinder.conf for victoria.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/776

    Closes-Bug: #1939389

    Co-authored-by: Samuel Walladge <email address hidden>

    Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/848914

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-cinder/+/848915

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-cinder/+/848916

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-cinder/+/854593

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-cinder/+/854594

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/848914
Committed: https://opendev.org/openstack/charm-cinder/commit/ba21da64f6a547bd44956d71933030d529956bcd
Submitter: "Zuul (22348)"
Branch: master

commit ba21da64f6a547bd44956d71933030d529956bcd
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

    Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/870

    Depends-Onː Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    Related-Bug: #1979812
    Related-Bug: #1939389

    Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/848915
Committed: https://opendev.org/openstack/charm-cinder/commit/ed0f80b7650bd03266897c0f17aa7d84ba9711fd
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit ed0f80b7650bd03266897c0f17aa7d84ba9711fd
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

    Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

    This issue has been observed on victoria and newer,
    hence the addition of cinder.conf for victoria.

    Closes-Bug: #1939389

    Co-authored-by: Samuel Walladge <email address hidden>

    Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
    (cherry picked from commit 024de3783fc459fc25596ec6084dd065e945df76)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/848916
Committed: https://opendev.org/openstack/charm-cinder/commit/27c1c0a053545b18b1802d943b296c39c6aa4706
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 27c1c0a053545b18b1802d943b296c39c6aa4706
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

    Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/871

    Depends-Onː Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    Related-Bug: #1979812
    Related-Bug: #1939389

    Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066
    (cherry picked from commit ba21da64f6a547bd44956d71933030d529956bcd)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-cinder/+/857343

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-cinder/+/857348

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/854593
Committed: https://opendev.org/openstack/charm-cinder/commit/c77025825b21e3707d19416e9c136b01872144fc
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c77025825b21e3707d19416e9c136b01872144fc
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

    Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

    This issue has been observed on victoria and newer,
    hence the addition of cinder.conf for victoria.

    Closes-Bug: #1939389

    Co-authored-by: Samuel Walladge <email address hidden>

    Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/857343
Committed: https://opendev.org/openstack/charm-cinder/commit/850bcaa45213edfff2cc43ddb2d8dba76b9bf9f5
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 850bcaa45213edfff2cc43ddb2d8dba76b9bf9f5
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

    Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/906

    Related-Bug: #1979812
    Related-Bug: #1939389

    Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/854594
Committed: https://opendev.org/openstack/charm-cinder/commit/2b71f3b1d7bdd8a0902cd6439bc2f66e797293fc
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2b71f3b1d7bdd8a0902cd6439bc2f66e797293fc
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

    Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

    This issue has been observed on victoria and newer,
    hence the addition of cinder.conf for victoria.

    Closes-Bug: #1939389

    Co-authored-by: Samuel Walladge <email address hidden>

    Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
    (cherry picked from commit 024de3783fc459fc25596ec6084dd065e945df76)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/857348
Committed: https://opendev.org/openstack/charm-cinder/commit/aa96cb0efd86136a8f6eaaa935b7526a47e81a8f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit aa96cb0efd86136a8f6eaaa935b7526a47e81a8f
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

    Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/907

    Related-Bug: #1979812
    Related-Bug: #1939389

    Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066
    (cherry picked from commit ba21da64f6a547bd44956d71933030d529956bcd)

Reproduced the same problem on Ussuri when trying to extend already attached volume as a regular user (a user assigned 'member' role only). Steps to recreate:

1. OpenStack Ussuri, charm: cinder| ussuri/edge | rev. 568
2. create a volume and attach it to a VM
3. as a user with the 'member' role assigned to the project try to extend the volume:
`cinder extend <vol id> <new size>
4. the command completes with the rc 0 (all good), `openstack volume show` shows new size but cinder-volume.log contains following error and the volume size from the VM perspective remains unchanged:
#### actual 403 error message from nova #####
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova [req-13d1093e-9a7f-4f03-b505-c985d9d45691 754c0e6eae22488982f393b69e9800a3 d6761dacbb0649189a07a4a1a191a8c0 - 6a4439622e71431c8b96073e33c3b7e1 6a4439622e71431c8b96073e33c3b7e1] Failed to notify nova on events: [{'name': 'volume-extended', 'server_uuid': '3d51176f-281c-4ab3-8631-042ccd407d1d', 'tag': '99c14113-f1ed-4e28-a9b9-54c7eb862db8'}].: novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-6a908980-00ae-455e-b8fb-aab951032251)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova Traceback (most recent call last):
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 149, in _send_events
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova response = nova.server_external_events.create(events)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/novaclient/v2/server_external_events.py", line 38, in create
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova return self._create('/os-server-external-events', body, 'events',
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova resp, body = self.api.client.post(url, body=body)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova return self.request(url, 'POST', **kwargs)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/novaclient/client.py", line 78, in request
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova raise exceptions.from_response(resp, body, url, method)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-6a908980-00ae-455e-b8fb-aab951032251)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova

#### successful completion - that is wrong ####
2022-11-21 12:53:22.880 1243598 INFO cinder.volume.manager [req-13d1093e-9a7f-4f03-b505-c985d9d45691 754c0e6eae22488982f393b69e9800a3 d6761dacbb0649189a07a4a1a191a8c0 - 6a4439622e71431c8b96073e33c3b7e1 6a4439622e71431c8b96073e33c3b7e1] E...

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/867997

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/867998

Change abandoned by "Tiago Pasqualini da Silva <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/867998

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/868001

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/868001
Committed: https://opendev.org/openstack/charm-cinder/commit/e737668e2fa2b3c59daca5676bd27b0bf4db53fe
Submitter: "Zuul (22348)"
Branch: master

commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

    Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872571

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872572

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872573

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872574

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872575

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872576

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder/+/874334

Change abandoned by "Tiago Pasqualini da Silva <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder/+/874334
Reason: wrong cherry-pick

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872571
Committed: https://opendev.org/openstack/charm-cinder/commit/00fe7b77e85d54a6e35aaddcbf50257b99cfa2b5
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 00fe7b77e85d54a6e35aaddcbf50257b99cfa2b5
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

    Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872572
Committed: https://opendev.org/openstack/charm-cinder/commit/4496922f48c72f4636d93b9268c9dad0c63a7574
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 4496922f48c72f4636d93b9268c9dad0c63a7574
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

    Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/874334
Committed: https://opendev.org/openstack/charm-cinder/commit/2bf21135499d2fa22932c24ca06a5e520e0f2529
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 2bf21135499d2fa22932c24ca06a5e520e0f2529
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

    Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

    This issue has been observed on victoria and newer,
    hence the addition of cinder.conf for victoria.

    Closes-Bug: #1939389

    Co-authored-by: Samuel Walladge <email address hidden>

    Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
    (cherry picked from commit 024de3783fc459fc25596ec6084dd065e945df76)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872573
Committed: https://opendev.org/openstack/charm-cinder/commit/a71622b7b227b11bbc60c4ee7b391d2a98e52e1a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit a71622b7b227b11bbc60c4ee7b391d2a98e52e1a
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

    Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872574
Committed: https://opendev.org/openstack/charm-cinder/commit/90ff83272b1e3b5f732bdff95289f53bf8028069
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 90ff83272b1e3b5f732bdff95289f53bf8028069
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

    Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/867997
Committed: https://opendev.org/openstack/charm-cinder/commit/8eac344dd68f574787eb74bb56c1e62a040cb787
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 8eac344dd68f574787eb74bb56c1e62a040cb787
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

    Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

    This issue has been observed on victoria and newer,
    hence the addition of cinder.conf for victoria.

    Closes-Bug: #1939389

    Co-authored-by: Samuel Walladge <email address hidden>

    Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
    (cherry-picked from commit 2b71f3b1d7bdd8a0902cd6439bc2f66e797293fc)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872575
Committed: https://opendev.org/openstack/charm-cinder/commit/8f1ce44171eaca3bc085b921c8815c6c6a36af4c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 8f1ce44171eaca3bc085b921c8815c6c6a36af4c
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

    Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872576
Committed: https://opendev.org/openstack/charm-cinder/commit/755a8c9b7b43e99ae5bda053aff73350d4277a42
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 755a8c9b7b43e99ae5bda053aff73350d4277a42
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

    Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)