Ceph exporter takes ~100 seconds to return
root@juju-733f2f-5-lxd-13:~# /usr/lib/nagios/plugins/check_http -I 127.0.0.1 -p 9128 -u /metrics -t 100
HTTP OK: HTTP/1.0 200 OK - 416838 bytes in 87.556 second response time |time=87.555859s;;;0.000000;100.000000 size=416838B;;;0
Ceph cluster size:
root@juju-733f2f-3-lxd-1:~# ceph -s
cluster:
id: 3d196f64-760e-11eb-b486-e7aafffac620
health: HEALTH_OK
services:
mon: 3 daemons, quorum juju-733f2f-3-lxd-1,juju-733f2f-5-lxd-1,juju-733f2f-4-lxd-1 (age 26h)
mgr: juju-733f2f-3-lxd-1(active, since 2w), standbys: juju-733f2f-4-lxd-1, juju-733f2f-5-lxd-1
osd: 189 osds: 189 up (since 24h), 189 in (since 2w)
rgw: 3 daemons active (juju-733f2f-3-lxd-12, juju-733f2f-4-lxd-12, juju-733f2f-5-lxd-12)
task status:
data:
pools: 20 pools, 6177 pgs
objects: 15.92M objects, 54 TiB
usage: 162 TiB used, 1.7 PiB / 1.9 PiB avail
pgs: 6170 active+clean
7 active+clean+scrubbing+deep
io:
client: 57 KiB/s rd, 3.3 MiB/s wr, 19 op/s rd, 470 op/s wr
Also syslog is flooded with:
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6c2","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6c4","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6c3","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6c3","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6c4","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6c2","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6c5","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6c1","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6c6","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6c0","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6c7","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6bf","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6b8","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6be","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6b9","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6bd","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6ba","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6bc","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6bb","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6bb","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6bc","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"3.6ba","prefix":"query"}: rados: Permission denied
Apr 15 14:32:10 juju-733f2f-5-lxd-13 prometheus-ceph-exporter.ceph-exporter[832270]: 2021/04/15 14:32:10 failed sending PG command {"format":"json","pgid":"4.6bd","prefix":"query"}: rados: Permission denied
We should understand if there is a better configuration to run ceph-exporter
i.e. setting CEPH_RADOS_OP_TIMEOUT when runnign ceph-exporter
https://github.com/digitalocean/ceph_exporter
