Ceph RADOS Gateway Charm

Services not running ceph-radosgw: keyring not found

Bug #1995363 reported by Aymen Frikha
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ceph RADOS Gateway Charm
New
Undecided
Unassigned

Bug Description

Deployed the openstack base bundle (yoga stable) but only ceph radosgw stays in blocked state with the following message: Services not running that should be: <email address hidden>

checking from syslog and got the following log:

Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring: (2) No such file or directory
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 AuthRegistry(0x55a269f92778) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring, disabling cephx
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring: (2) No such file or directory
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: 2022-11-01T10:40:50.566+0000 7fd5bfc1ee40 -1 AuthRegistry(0x7ffd23f00ca0) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-91fa65-23/keyring, disabling cephx
Nov 1 10:40:50 juju-91fa65-23 radosgw[19448]: failed to fetch mon config (--no-mon-config to skip)
Nov 1 10:40:50 juju-91fa65-23 systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
Nov 1 10:40:50 juju-91fa65-23 systemd[1]: <email address hidden>: Failed with result 'exit-code'.
Nov 1 10:40:50 juju-91fa65-23 systemd[1]: <email address hidden>: Scheduled restart job, restart counter is at 2.

Not sure what is causing that.

Revision history for this message
Aymen Frikha (aym-frikha) wrote :

subscribed ~field-high

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

There's not enough information here to reproduce the bug, can you please add more information as documented in the charm guide: https://docs.openstack.org/charm-guide/latest/community/software-bug.html

Changed in charm-ceph-radosgw:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for Ceph RADOS Gateway Charm because there has been no activity for 60 days.]

Changed in charm-ceph-radosgw:
status: Incomplete → Expired
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

I'm seeing this too. Also openstack yoga (in my case, with focal).

Only one of the 3 units is in "blocked" state, and I can see this in syslog:

Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring: (2) No such file or directory
Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 AuthRegistry(0x564718e807f8) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring, disabling cephx
Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring: (2) No such file or directory
Jan 4 12:52:01 juju-b4a340-3-lxd-4 radosgw[1286223]: 2023-01-04T12:52:01.936+0000 7fa88e77ae40 -1 AuthRegistry(0x7ffc3da03590) no keyring found at /var/lib/ceph/radosgw/ceph-rgw.juju-b4a340-3-lxd-4/keyring, disabling cephx

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

This is the complete log for the unit: https://pastebin.canonical.com/p/w7swdbsBSN/

Changed in charm-ceph-radosgw:
status: Expired → New
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

What I find interesting is that in that log it says:

2023-01-03 20:32:52 WARNING unit.ceph-radosgw/0.juju-log server.go:316 Inconsistent or absent auth returned by mon units. Setting auth_supported to 'none'

and on the mon units you can see:

2023-01-03 20:53:42 WARNING unit.ceph-mon/0.osd-relation-changed logger.go:60 Error ENOENT: failed to find client.rgw.juju-b4a340-3-lxd-4 in keyring

Here are the logs for the three mon units just in case:
https://pastebin.canonical.com/p/b9ntnzyD67/
https://pastebin.canonical.com/p/XXcdVcSYxp/
https://pastebin.canonical.com/p/5WWwSSPjHz/

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

This is a show-unit of the ceph-radosgw/0 unit: https://pastebin.canonical.com/p/g9Z7qJfFJ5/

and this is a show-unit of a ceph-mon unit: https://pastebin.canonical.com/p/MDcPntnCYc/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.