Unable to create/delete volumes after series upgrade
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Cinder Charm |
New
|
Undecided
|
Unassigned | ||
charms.ceph |
New
|
Undecided
|
Unassigned |
Bug Description
This is Openstack Mitaka that is currently going through series upgrade from trusty to xenial.
After xenial upgrade we have the following packages on cinder:
# dpkg -l | grep -E "cinder|ceph"
ii ceph-common 10.2.11-
ii cinder-api 2:8.1.1-0ubuntu3 all Cinder storage service - API server
ii cinder-backup 2:8.1.1-0ubuntu3 all Cinder storage service - Scheduler server
ii cinder-common 2:8.1.1-0ubuntu3 all Cinder storage service - common files
ii cinder-scheduler 2:8.1.1-0ubuntu3 all Cinder storage service - Scheduler server
ii cinder-volume 2:8.1.1-0ubuntu3 all Cinder storage service - Volume server
ii libcephfs1 10.2.11-
ii python-cephfs 10.2.11-
ii python-cinder 2:8.1.1-0ubuntu3 all Cinder Python libraries
And cinder charm is at revision 303
We are using ceph behind cinder and ceph-mon is at revision 48
After upgrading series from trusty to xenial,
we were not able to create or delete Volumes and all remained in pending state. However no errors were logged in cinder logs.
Checking ceph auth caps for cinder client in ceph we found that caps were the following:
# ceph auth get client.cinder
exported keyring for client.cinder
[client.cinder]
key = <REDACTED>
caps mon = "allow r; allow command "osd blacklist""
caps osd = "allow rwx"
While checking the same caps on a vanilla trusty/mistaka (w/o upgrades) caps are the following:
# ceph auth get client.cinder
exported keyring for client.cinder
[client.cinder]
key = <REDACTED>
caps mon = "allow rw"
caps osd = "allow rwx"
We updated the caps to be the same as the vanilla trusty/mitaka with:
# ceph auth caps client.cinder osd "allow rwx" mon "allow rw"
and we are able again to create/delete volumes.
tags: | added: series-upgrade |
I've added charms.ceph to this bug. It looks like this code lives in charms.ceph around https:/ /github. com/openstack/ charms. ceph/blob/ master/ charms_ ceph/utils. py#L1150