Bionic w/ CMR exporter fails to start
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Prometheus Ceph Exporter Charm |
Confirmed
|
Medium
|
Unassigned |
Bug Description
Installing Rev 5 on a bionic LXD and relating to Ceph-Mon via CMR is failing to start and register the service.
The error from systemd shows:
Aug 16 22:38:29 juju-5815ea-12 systemd[1]: Started Service for snap application prometheus-
-- Subject: Unit snap.prometheus
-- Defined-By: systemd
-- Support: http://
--
-- Unit snap.prometheus
--
-- The start-up result is RESULT.
Aug 16 22:38:29 juju-5815ea-12 prometheus-
Aug 16 22:38:29 juju-5815ea-12 prometheus-
Aug 16 22:38:29 juju-5815ea-12 systemd[1]: snap.prometheus
Aug 16 22:38:29 juju-5815ea-12 systemd[1]: snap.prometheus
Aug 16 22:38:30 juju-5815ea-12 systemd[1]: snap.prometheus
Aug 16 22:38:30 juju-5815ea-12 systemd[1]: snap.prometheus
It appears the service is trying to authenticate as "ceph-exporter". Checking `ceph auth ls` shows that user doesn't exist in ceph.
Ceph-Mon is being used via a CMR, and is the recently released Rev 42.
Both controllers are Juju 2.6.6, which is the current latest.
Confirmed.
The relation does create a user and the key does match between nodes, but it's actually created with a different username:
ceph-mon: user with the following pattern - "client. remote- <hash>" ceph-exporter: "client. prometheus- ceph-exporter"
prometheus-
So it cannot connect, and the journal is full of entries like:
Mar 20 01:02:47 juju-e56e6c-0 prometheus- ceph-exporter. ceph-exporter[ 16251]: * Running /snap/prometheu s-ceph- exporter/ x1/bin/ ceph_exporter with args: -ceph.user prometheus- ceph-exporter ceph-exporter. ceph-exporter[ 16251]: 2020/03/20 01:02:47 cannot connect to ceph cluster: rados: Operation not permitted
Mar 20 01:02:47 juju-e56e6c-0 prometheus-