Bug #2033886 “[rev 38] ceph-dashboard fails hook “certificates-r...” : Bugs : Ceph Dashboard Charm

Bas de Bruijne (basdbruijne) on 2023-09-01

tags:

added: cdo-qa foundations-engine

Revision history for this message

utkarsh bhatt (utkarshbhatthere) wrote on 2023-09-04:

#1

The root cause for this error is a race condition between mgr enablement and certificates relation handle:

Log Print:
2023-08-29T23:17:47.373+0000 7f168d179640 -1 mgr.server reply reply (95) Operation not supported Module 'dashboard' is not enabled (required by command 'dashboard set-ssl-certificate'): use ceph mgr module enable dashboard to enable it

The newly split event handlers require to defer the handling until dashboard mgr module is enabled.

OpenStack Infra (hudson-openstack) on 2023-09-04

Changed in charm-ceph-dashboard:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-13: Fix merged to charm-ceph-dashboard (master)

#2

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/893479
Committed: https://opendev.org/openstack/charm-ceph-dashboard/commit/33f60aea0971cdf49a78f475d8469431b57b0b30
Submitter: "Zuul (22348)"
Branch: master

commit 33f60aea0971cdf49a78f475d8469431b57b0b30
Author: Utkarsh Bhatt <email address hidden>
Date: Fri Sep 1 13:11:41 2023 +0530

Adds dashboard enablement/deferral logic improvements

    Closes-Bug: 2033886
    Change-Id: I2f4d4dfa890fc79f7f5d5b1273a11907b60e8255
    Signed-off-by: Utkarsh Bhatt <email address hidden>

Changed in charm-ceph-dashboard:
status:	In Progress → Fix Committed

Revision history for this message

Karthik K (hikkart) wrote on 2023-09-14:

#4

Download full text (6.5 KiB)

Team

Faced the same issue with charm-ceph-dashboard rev 40 as well on manually provisioned juju.

Please let me know, on which version the fix would be available to check again.
Thank you.

---

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 769, in <module>
    main(CephDashboardCharm)
.....
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 745, in _enable_ssl_from_config
    self._configure_tls_from_charm_config()
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 715, in _configure_tls_from_charm_config
    self._configure_tls(key, cert, ca_cert, self.TLS_CHARM_CA_CERT_PATH)
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 583, in _configure_tls
    ceph_utils.dashboard_set_ssl_certificate(
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/venv/charms_ceph/utils.py", line 3609, in _dashboard_set_ssl_artifact
    subprocess.check_call(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'set-ssl-certificate', 'ceph-2', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')]' returned non-zero exit status 22.
unit-ceph-dashboard-4: 22:33:37 ERROR juju.worker.uniter.operation hook "dashboard-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-ceph-dashboard-4: 22:33:37 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-ceph-dashboard-7: 22:34:12 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-ceph-dashboard-7: 22:34:13 WARNING unit.ceph-dashboard/7.dashboard-relation-changed Error EINVAL: unrecognized config option 'mgr/dashboard/ceph-3/server_addr'
unit-ceph-dashboard-7: 22:34:13 ERROR unit.ceph-dashboard/7.juju-log dashboard:3: Uncaught exception while in charm code:

----
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/./src/charm.py", line 769, in <module>
    main(CephDashboardCharm)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 784, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/src/interface_dashboard.py", line 50, in on_changed
    self.on.mon_ready.emit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 784, i...

Team

Faced the same issue with charm-ceph-dashboard rev 40 as well on manually provisioned juju.

Please let me know, on which version the fix would be available to check again.
Thank you.

---

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 769, in <module>
    main(CephDashboardCharm)
.....
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 745, in _enable_ssl_from_config
    self._configure_tls_from_charm_config()
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 715, in _configure_tls_from_charm_config
    self._configure_tls(key, cert, ca_cert, self.TLS_CHARM_CA_CERT_PATH)
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/./src/charm.py", line 583, in _configure_tls
    ceph_utils.dashboard_set_ssl_certificate(
  File "/var/lib/juju/agents/unit-ceph-dashboard-4/charm/venv/charms_ceph/utils.py", line 3609, in _dashboard_set_ssl_artifact
    subprocess.check_call(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'set-ssl-certificate', 'ceph-2', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')]' returned non-zero exit status 22.
unit-ceph-dashboard-4: 22:33:37 ERROR juju.worker.uniter.operation hook "dashboard-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-ceph-dashboard-4: 22:33:37 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-ceph-dashboard-7: 22:34:12 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-ceph-dashboard-7: 22:34:13 WARNING unit.ceph-dashboard/7.dashboard-relation-changed Error EINVAL: unrecognized config option 'mgr/dashboard/ceph-3/server_addr'
unit-ceph-dashboard-7: 22:34:13 ERROR unit.ceph-dashboard/7.juju-log dashboard:3: Uncaught exception while in charm code:

----
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/./src/charm.py", line 769, in <module>
    main(CephDashboardCharm)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 784, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/src/interface_dashboard.py", line 50, in on_changed
    self.on.mon_ready.emit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 784, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/./src/charm.py", line 405, in _configure_dashboard
    ceph_utils.mgr_config_set(
  File "/var/lib/juju/agents/unit-ceph-dashboard-7/charm/venv/charms_ceph/utils.py", line 3568, in ceph_config_set
    subprocess.check_call(['ceph', 'config', 'set', who, name, value])
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceph', 'config', 'set', 'mgr', 'mgr/dashboard/ceph-3/server_addr', '10.xx.xx.127']' returned non-zero exit status 22.
unit-ceph-dashboard-7: 22:34:14 ERROR juju.worker.uniter.operation hook "dashboard-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-ceph-dashboard-7: 22:34:14 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-ceph-osd-1: 22:33:48 INFO unit.ceph-osd/1.juju-log Updating status.
unit-ceph-osd-1: 22:33:49 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-ceph-osd-0: 22:34:33 INFO unit.ceph-osd/0.juju-log Updating status.
unit-ceph-osd-0: 22:34:33 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)

---

user@juju-manual:~$ juju status
Model  Controller            Cloud/Region          Version  SLA          Timestamp
ceph   manual-cloud-default  manual-cloud/default  3.2.3    unsupported  22:34:40-07:00

App             Version  Status  Scale  Charm           Channel        Rev  Exposed  Message
ceph-dashboard           error       3  ceph-dashboard  latest/edge     40  no       hook failed: "dashboard-relation-changed"
ceph-mon        17.2.6   active      3  ceph-mon        quincy/stable  183  no       Unit is ready and clustered
ceph-osd        17.2.6   active      3  ceph-osd        quincy/stable  564  no       Unit is ready (1 OSD)

Unit                 Workload  Agent  Machine  Public address  Ports  Message
ceph-mon/0*          active    idle   0        10.xx.xx.94           Unit is ready and clustered
  ceph-dashboard/5   error     idle            10.xx.xx.94           hook failed: "dashboard-relation-changed"
ceph-mon/1           active    idle   1        10.xx.xx.109          Unit is ready and clustered
  ceph-dashboard/4*  error     idle            10.xx.xx.109          hook failed: "dashboard-relation-changed"
ceph-mon/2           active    idle   2        10.xx.xx.127          Unit is ready and clustered
  ceph-dashboard/7   error     idle            10.xx.xx.127          hook failed: "dashboard-relation-changed"
ceph-osd/0           active    idle   0        10.xx.xx.94           Unit is ready (1 OSD)
ceph-osd/1*          active    idle   1        10.xx.xx.109          Unit is ready (1 OSD)
ceph-osd/2           active    idle   2        10.xx.xx.127          Unit is ready (1 OSD)

Machine  State    Address        Inst id               Base          AZ  Message
0        started  10.xx.xx.94   manual:10.xx.xx.94   ubuntu@22.04      Manually provisioned machine
1        started  10.xx.xx.109  manual:10.xx.xx.109  ubuntu@22.04      Manually provisioned machine
2        started  10.xx.xx.127  manual:10.xx.xx.127  ubuntu@22.04      Manually provisioned machine

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-27: Fix proposed to charm-ceph-dashboard (stable/quincy.2)

#5

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/896527

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-27: Change abandoned on charm-ceph-dashboard (stable/quincy.2)

#6

Change abandoned by "Peter Sabaini <email address hidden>" on branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/896527
Reason: Wront cherry pick

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-28: Fix proposed to charm-ceph-dashboard (stable/quincy.2)

#7

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/896742

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-28: Change abandoned on charm-ceph-dashboard (stable/quincy.2)

#8

Change abandoned by "Peter Sabaini <email address hidden>" on branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/896527
Reason: Redo due to conflict

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-28: Fix merged to charm-ceph-dashboard (stable/quincy.2)

#9

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-dashboard/+/896742
Committed: https://opendev.org/openstack/charm-ceph-dashboard/commit/2f7c246e38e8148940502ac0bd0745da6b4f244c
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 2f7c246e38e8148940502ac0bd0745da6b4f244c
Author: Utkarsh Bhatt <email address hidden>
Date: Fri Sep 1 13:11:41 2023 +0530

Adds dashboard enablement/deferral logic improvements

Closes-Bug: 2033886
cherry-picked from: I2f4d4dfa890fc79f7f5d5b1273a11907b60e8255

Change-Id: I63fbe0776c92419d8b047bab151e01c4f1f3f45e

Ceph Dashboard Charm

[rev 38] ceph-dashboard fails hook "certificates-relation-changed"

Bug Description

Other bug subscribers

Remote bug watches