kolla-ansible

nova uses wrong RBD user for Ceph

Series wallaby
Bug #1934145

Bug #1934145 reported by Mark Goddard on 2021-06-30

This bug affects 1 person

	Status	Importance	Assigned to
kolla-ansible	Fix Released	High	Unassigned
Ussuri	Fix Released	High	Unassigned
Victoria	Fix Released	High	Unassigned
Wallaby	Fix Committed	High	Unassigned
Xena	Fix Released	High	Unassigned

Bug Description

In Ussuri, nova stopped using separate Ceph keys for the volumes and vms pools by default. Instead, we set ceph_nova_keyring to the value of ceph_cinder_keyring by default, which is ceph.client.cinder.keyring. This is in line with the Ceph OpenStack integration guide [1]. However, the user used by nova to access the vms pool (ceph_nova_user) defaults to nova, meaning that nova will still try to use a ceph.client.nova.keyring, which probably does not exist. This results in an error such as the following:

2021-06-30 08:51:27.149 8 ERROR nova.compute.manager Traceback (most recent call last):
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/storage/rbd_utils.py", line 430, in get_pool_info
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager out, _ = processutils.execute(*args)
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager cmd=sanitized_cmd)
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager Command: ceph df --format=json --id nova --conf /etc/ceph/ceph.conf
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager Exit code: 13
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager Stdout: ''
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager Stderr: '[errno 13] error connecting to the cluster\n'
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager Traceback (most recent call last):
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 9935, in _update_available_resource_for_node
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager startup=startup)
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 879, in update_available_resource
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8858, in get_available_resource
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager disk_info_dict = self._get_local_gb_info()
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 7302, in _get_local_gb_info
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager info = rbd_utils.RBDDriver().get_pool_info()
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/storage/rbd_utils.py", line 434, in get_pool_info
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager reason='Could not determine disk usage')
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager nova.exception.StorageError: Storage error: Could not determine disk usage
2021-06-30 08:51:27.149 8 ERROR nova.compute.manager
2021-06-30 08:52:26.747 7 INFO os_vif [-] Loaded VIF plugi

We did not see this issue in CI, because we set ceph_nova_user to cinder.

[1] https://docs.ceph.com/en/latest/rbd/rbd-openstack/

Mark Goddard (mgoddard) on 2021-06-30

Changed in kolla-ansible:
importance:	Undecided → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-30: Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/798851

Changed in kolla-ansible:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-06: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/798851
Committed: https://opendev.org/openstack/kolla-ansible/commit/c3f9ba835b1da740c9f3056dbda20a7072467757
Submitter: "Zuul (22348)"
Branch: master

commit c3f9ba835b1da740c9f3056dbda20a7072467757
Author: Mark Goddard <email address hidden>
Date: Wed Jun 30 09:30:54 2021 +0100

nova: Use cinder user for Ceph

    In Ussuri, nova stopped using separate Ceph keys for the volumes and vms
    pools by default. Instead, we set ceph_nova_keyring to the value of
    ceph_cinder_keyring by default, which is ceph.client.cinder.keyring.
    This is in line with the Ceph OpenStack integration guide [1]. However,
    the user used by nova to access the vms pool (ceph_nova_user) defaults
    to nova, meaning that nova will still try to use a
    ceph.client.nova.keyring, which probably does not exist. We did not see
    this issue in CI, because we set ceph_nova_user to cinder.

This change fixes the issue by setting ceph_nova_user to the value of
ceph_cinder_user by default, which is cinder.

Closes-Bug: #1934145
Related-Bug: #1928690

[1] https://docs.ceph.com/en/latest/rbd/rbd-openstack/

Change-Id: I6aa8db2214e07906f1f3e035411fc80ba911a274

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-09: Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/803838

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-09: Fix proposed to kolla-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/803839

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-09: Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/803840

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-20: Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/803838
Committed: https://opendev.org/openstack/kolla-ansible/commit/d22e3e995ca2018dfc504da45a869032ed5cd7d3
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d22e3e995ca2018dfc504da45a869032ed5cd7d3
Author: Mark Goddard <email address hidden>
Date: Wed Jun 30 09:30:54 2021 +0100

nova: Use cinder user for Ceph

    In Ussuri, nova stopped using separate Ceph keys for the volumes and vms
    pools by default. Instead, we set ceph_nova_keyring to the value of
    ceph_cinder_keyring by default, which is ceph.client.cinder.keyring.
    This is in line with the Ceph OpenStack integration guide [1]. However,
    the user used by nova to access the vms pool (ceph_nova_user) defaults
    to nova, meaning that nova will still try to use a
    ceph.client.nova.keyring, which probably does not exist. We did not see
    this issue in CI, because we set ceph_nova_user to cinder.

This change fixes the issue by setting ceph_nova_user to the value of
ceph_cinder_user by default, which is cinder.

Closes-Bug: #1934145
Related-Bug: #1928690

[1] https://docs.ceph.com/en/latest/rbd/rbd-openstack/

Change-Id: I6aa8db2214e07906f1f3e035411fc80ba911a274

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-20: Fix merged to kolla-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/803839
Committed: https://opendev.org/openstack/kolla-ansible/commit/8aa8e617d982222e4b3c8d6a26b7190baca1b199
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 8aa8e617d982222e4b3c8d6a26b7190baca1b199
Author: Mark Goddard <email address hidden>
Date: Wed Jun 30 09:30:54 2021 +0100

nova: Use cinder user for Ceph

    In Ussuri, nova stopped using separate Ceph keys for the volumes and vms
    pools by default. Instead, we set ceph_nova_keyring to the value of
    ceph_cinder_keyring by default, which is ceph.client.cinder.keyring.
    This is in line with the Ceph OpenStack integration guide [1]. However,
    the user used by nova to access the vms pool (ceph_nova_user) defaults
    to nova, meaning that nova will still try to use a
    ceph.client.nova.keyring, which probably does not exist. We did not see
    this issue in CI, because we set ceph_nova_user to cinder.

This change fixes the issue by setting ceph_nova_user to the value of
ceph_cinder_user by default, which is cinder.

Closes-Bug: #1934145
Related-Bug: #1928690

[1] https://docs.ceph.com/en/latest/rbd/rbd-openstack/

Change-Id: I6aa8db2214e07906f1f3e035411fc80ba911a274

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-20: Fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/803840
Committed: https://opendev.org/openstack/kolla-ansible/commit/fcad47657a280949a81053d65cc10a24fc2f242e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit fcad47657a280949a81053d65cc10a24fc2f242e
Author: Mark Goddard <email address hidden>
Date: Wed Jun 30 09:30:54 2021 +0100

nova: Use cinder user for Ceph

    In Ussuri, nova stopped using separate Ceph keys for the volumes and vms
    pools by default. Instead, we set ceph_nova_keyring to the value of
    ceph_cinder_keyring by default, which is ceph.client.cinder.keyring.
    This is in line with the Ceph OpenStack integration guide [1]. However,
    the user used by nova to access the vms pool (ceph_nova_user) defaults
    to nova, meaning that nova will still try to use a
    ceph.client.nova.keyring, which probably does not exist. We did not see
    this issue in CI, because we set ceph_nova_user to cinder.

This change fixes the issue by setting ceph_nova_user to the value of
ceph_cinder_user by default, which is cinder.

Closes-Bug: #1934145
Related-Bug: #1928690

[1] https://docs.ceph.com/en/latest/rbd/rbd-openstack/

Change-Id: I6aa8db2214e07906f1f3e035411fc80ba911a274

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-02: Fix included in openstack/kolla-ansible 12.2.0

This issue was fixed in the openstack/kolla-ansible 12.2.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-14: Fix included in openstack/kolla-ansible 13.0.0.0rc1

#10

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-23: Fix included in openstack/kolla-ansible 10.4.0

#11

This issue was fixed in the openstack/kolla-ansible 10.4.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-01-03: Fix included in openstack/kolla-ansible 11.2.0

#12

This issue was fixed in the openstack/kolla-ansible 11.2.0 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.