Ceph Monitor Charm

"crash" module is always on but not properly configured

Bug #2000630 reported by Nobuto Murata
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph Monitor Charm
Fix Committed
Undecided
Samuel Walladge
Quincy.2
Fix Committed
Undecided
Unassigned
Ceph OSD Charm
Fix Committed
Undecided
Samuel Walladge
Quincy.2
Fix Committed
Undecided
Unassigned

Bug Description

cloud:focal-yoga (quincy)

$ juju ssh ceph-mon/leader -- sudo ceph version
ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)

How to reproduce:

1. make sure "crash" module is on (it's a part of "always on" modules)

    https://docs.ceph.com/en/quincy/mgr/crash/

    $ juju ssh ceph-mon/leader -- sudo ceph mgr module ls | grep crash
    crash on (always on)

2. intentionally crash ceph-osd process (in this example I used SIGSEGV)

    $ juju ssh ceph-osd/leader -- sudo pkill --signal SIGSEGV ceph-osd

3. make sure a normal crash file is generated for apport *and* a set of files for ceph crash module.

    # ll -h /var/crash/
    total 121M
    drwxrwxrwt 2 root root 4.0K Dec 28 10:42 ./
    drwxr-xr-x 13 root root 4.0K Dec 12 21:41 ../
    -rw-r----- 1 ceph ceph 121M Dec 28 10:42 _usr_bin_ceph-osd.64045.crash

    # ll -h /var/lib/ceph/crash/*
    '/var/lib/ceph/crash/2022-12-28T10:42:04.661282Z_51be6c87-4a42-4fbb-afe5-264d94cd6c79':
    total 1.6M
    drwx------ 2 ceph ceph 4.0K Dec 28 10:42 ./
    drwxr-xr-x 4 ceph ceph 4.0K Dec 28 10:42 ../
    -r--r--r-- 1 ceph ceph 0 Dec 28 10:42 done
    -rw-r--r-- 1 ceph ceph 1.6M Dec 28 10:42 log
    -rw------- 1 ceph ceph 926 Dec 28 10:42 meta

    /var/lib/ceph/crash/posted:
    total 8.0K
    drwxr-xr-x 2 root root 4.0K Sep 13 17:47 ./
    drwxr-xr-x 4 ceph ceph 4.0K Dec 28 10:42 ../

4. check syslog for post failures to MON units.

Dec 28 10:51:18 famous-skunk ceph-crash[10667]: WARNING:ceph-crash:post /var/lib/ceph/crash/2022-12-28T10:42:04.661282Z_51be6c87-4a42-4fbb-afe5-264d94cd6c79 as client.crash.famous-skunk failed: (None, b'2022-12-28T10:51:18.368+0000 7f427dc2f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.famous-skunk.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory\n2022-12-28T10:51:18.368+0000 7f427dc2f700 -1 AuthRegistry(0x7f427805f4f0) no keyring found at /etc/ceph/ceph.client.crash.famous-skunk.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx\n2022-12-28T10:51:18.376+0000 7f427c9cd700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.famous-skunk.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory\n2022-12-28T10:51:18.376+0000 7f427c9cd700 -1 AuthRegistry(0x7f4278065748) no keyring found at /etc/ceph/ceph.client.crash.famous-skunk.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx\n2022-12-28T10:51:18.376+0000 7f427c9cd700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.crash.famous-skunk.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory\n2022-12-28T10:51:18.376+0000 7f427c9cd700 -1 AuthRegistry(0x7f427c9cc000) no keyring found at /etc/ceph/ceph.client.crash.famous-skunk.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx\n[errno 2] RADOS object not found (error connecting to the cluster)\n')

Revision history for this message
Nobuto Murata (nobuto) wrote :

By configuring authentication for the crash module, OSD nodes posted a recent crash to MON. Real outages can happen after a few crashes of MONs or OSDs so this should be helpful to give a heads-up to operators to diagnose a recent crash.

https://docs.ceph.com/en/quincy/mgr/crash/

$ juju ssh ceph-mon/leader -- sudo ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash'
[client.crash]
     key = AQCRI6xje9HrHxAAU20bKTeL3k2pIlPNazeVfQ==

$ juju run -a ceph-osd '
cat <<EOF | sudo tee /etc/ceph/ceph.client.crash.keyring
[client.crash]
     key = AQCRI6xje9HrHxAAU20bKTeL3k2pIlPNazeVfQ==
EOF
'

$ sudo ceph health detail
HEALTH_WARN 1 daemons have recently crashed
[WRN] RECENT_CRASH: 1 daemons have recently crashed
    osd.2 crashed on host famous-skunk at 2022-12-28T10:42:04.661282Z

Samuel Walladge (swalladge)
Changed in charm-ceph-mon:
assignee: nobody → Samuel Walladge (swalladge)
Samuel Walladge (swalladge)
Changed in charm-ceph-mon:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-mon (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/869138

Revision history for this message
Samuel Walladge (swalladge) wrote :

A fix in two parts is ready for review:

- https://review.opendev.org/c/openstack/charm-ceph-mon/+/869138
- https://review.opendev.org/c/openstack/charm-ceph-osd/+/869139

Samuel Walladge (swalladge)
Changed in charm-ceph-osd:
status: New → In Progress
assignee: nobody → Samuel Walladge (swalladge)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-mon (master)

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/869138
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/b2408e9dd7130e3edf977c62533e6ea408c589eb
Submitter: "Zuul (22348)"
Branch: master

commit b2408e9dd7130e3edf977c62533e6ea408c589eb
Author: Samuel Walladge <email address hidden>
Date: Wed Jan 4 15:59:34 2023 +1030

    Create a key for ceph-osd for crash module auth

    This will be set on the osd relation,
    so the ceph-osd charm can use this key for auth
    by the crash reporting module.

    ref. https://docs.ceph.com/en/latest/mgr/crash/

    See https://review.opendev.org/c/openstack/charm-ceph-osd/+/869139
    for how this key is used by ceph-osd.

    Closes-Bug: #2000630
    Change-Id: Ic95aae6b5981a6df1e0b3c310bcef8018c494a24

Changed in charm-ceph-mon:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-osd (master)

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-osd/+/869139
Committed: https://opendev.org/openstack/charm-ceph-osd/commit/97be046f9b541067193c50323e00657110184f97
Submitter: "Zuul (22348)"
Branch: master

commit 97be046f9b541067193c50323e00657110184f97
Author: Samuel Walladge <email address hidden>
Date: Wed Jan 4 15:59:04 2023 +1030

    Save the crash module auth key

    Read the key set on the mon relation,
    and use ceph-authtool to save it to a keyring,
    for use by the crash module for crash reporting.

    When this auth key is set, the crash module (enabled by default)
    will update ceph-mon with a report.
    It also results in a neat summary of recent crashes
    that can be viewed by `ceph health detail`.
    For example:

    ```
    $ juju ssh ceph-mon/leader -- sudo ceph health detail

    HEALTH_WARN 1 daemons have recently crashed
    [WRN] RECENT_CRASH: 1 daemons have recently crashed
        osd.1 crashed on host node-3 at 2023-01-04T05:25:18.218628Z
    ```

    ref. https://docs.ceph.com/en/latest/mgr/crash/

    See also https://review.opendev.org/c/openstack/charm-ceph-mon/+/869138
    for where the client_crash_key relation data set is implemented.

    Depends-On: https://review.opendev.org/c/openstack/charm-ceph-mon/+/869138

    Closes-Bug: #2000630
    Change-Id: I77c84c368e6665e4988ebe9a735f000f99d0b78e

Changed in charm-ceph-osd:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-osd (stable/quincy.2)

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/876456

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-mon (stable/quincy.2)

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/876457

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-mon (stable/quincy.2)

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/876457
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/9df0390d3f17251e3fb664cb90341a86435f764c
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 9df0390d3f17251e3fb664cb90341a86435f764c
Author: Samuel Walladge <email address hidden>
Date: Wed Jan 4 15:59:34 2023 +1030

    Create a key for ceph-osd for crash module auth

    This will be set on the osd relation,
    so the ceph-osd charm can use this key for auth
    by the crash reporting module.

    ref. https://docs.ceph.com/en/latest/mgr/crash/

    See https://review.opendev.org/c/openstack/charm-ceph-osd/+/869139
    for how this key is used by ceph-osd.

    Closes-Bug: #2000630
    Change-Id: Ic95aae6b5981a6df1e0b3c310bcef8018c494a24
    (cherry picked from commit b2408e9dd7130e3edf977c62533e6ea408c589eb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-osd (stable/quincy.2)

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-osd/+/876456
Committed: https://opendev.org/openstack/charm-ceph-osd/commit/ad59f1bda762cbb75ed2cd2be758135eee2ffc36
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit ad59f1bda762cbb75ed2cd2be758135eee2ffc36
Author: Samuel Walladge <email address hidden>
Date: Wed Jan 4 15:59:04 2023 +1030

    Save the crash module auth key

    Read the key set on the mon relation,
    and use ceph-authtool to save it to a keyring,
    for use by the crash module for crash reporting.

    When this auth key is set, the crash module (enabled by default)
    will update ceph-mon with a report.
    It also results in a neat summary of recent crashes
    that can be viewed by `ceph health detail`.
    For example:

    ```
    $ juju ssh ceph-mon/leader -- sudo ceph health detail

    HEALTH_WARN 1 daemons have recently crashed
    [WRN] RECENT_CRASH: 1 daemons have recently crashed
        osd.1 crashed on host node-3 at 2023-01-04T05:25:18.218628Z
    ```

    ref. https://docs.ceph.com/en/latest/mgr/crash/

    See also https://review.opendev.org/c/openstack/charm-ceph-mon/+/869138
    for where the client_crash_key relation data set is implemented.

    Depends-On: https://review.opendev.org/c/openstack/charm-ceph-mon/+/869138

    Closes-Bug: #2000630
    Change-Id: I77c84c368e6665e4988ebe9a735f000f99d0b78e
    (cherry picked from commit 97be046f9b541067193c50323e00657110184f97)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.