Config change doesn't restart ceph-mon service

Bug #1979330 reported by Facundo Ciccioli
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph Monitor Charm
Triaged
Medium
Unassigned

Bug Description

A production ceph cluster was in HEALTH_WARN state showing a mon low on available space. Both warn and crit thresholds were at their defaults, 30 and 5 resp.

We appled a config change via juju:

        juju config ceph-mon monitor-data-available-warning=25
        juju config ceph-mon monitor-data-available-critical=20

The ceph.conf file reflected the changes but we noticed that ceph was still reporting HEALTH_WARN.

Looking at ceph-mon@*.service we noticed that it hadn't been restarted. After roll-restarting all three mons, the cluster went back to HEALTH_OK.

Shouldn't the ceph-mon services be restarted after ceph.conf is modified?

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

Give that we render the ceph.conf in one place [1], it's easy to confirm that we don't restart the mons. That said, we'd have to be _very_ careful in restarting the mons due to a config=changed as we have to ensure that we maintain quorum during a rolling restart. There is some code in the charm, today, to handle a rolling upgrade & restart of the charm that could be leveraged to do this.

1: https://github.com/openstack/charm-ceph-mon/blob/d3b2494ee8a23fe58cff4eb3308306a24a8f1434/hooks/ceph_hooks.py#L225

Revision history for this message
Facundo Ciccioli (fandanbango) wrote :

If restarts are a delicate matter, maybe for certain configuration keys (ie, known ones like the ones involved in this bug and not arbitrary ones like whatever you may specify on config-flags) they could be injected live without restarting. I believe ceph allows such a thing to be done, though we could be introducing further complexity.

Revision history for this message
Nobuto Murata (nobuto) wrote :

I'm speaking only for myself, but
`ceph config assimilate-conf -i /etc/ceph/ceph.conf`
could be executed whenever ceph.conf is changed. So any appropriate config can be reflected on the fly.

https://docs.ceph.com/en/latest/cephadm/adoption/?highlight=assimilate-conf#adoption-process
https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/?highlight=assimilate-conf#commands

Revision history for this message
Nobuto Murata (nobuto) wrote :

Hmm, never mind. It looks like assimilate-conf is not idempotent but only effective once for the initial migration to mon config database. After that, explicit `ceph config set mon <key> <value>` seems necessary.

Changed in charm-ceph-mon:
importance: Undecided → Medium
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.