charm fails hook "update-status" due to failed "set-ssl-certificate" command

Bug #2038943 reported by Bas de Bruijne
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph Dashboard Charm
New
Undecided
Unassigned

Bug Description

In test run https://solutions.qa.canonical.com/testruns/5fbb800e-9335-42d6-a3fe-b782ce4a5661/, which tests the following versions,

maas 3.3.4-13189-g.f88272d1e
juju 3.1.6
fce-container-image ubuntu:jammy
openstack yoga
charms yoga/candidate
ceph quincy/candidate
charmed-kubernetes 1.28
ceph-dashboard quincy/stable

the testrun fails because ceph-dashboard is in error status:
=======
ceph-mon/0 active idle 0/lxd/2 10.244.41.5 Unit is ready and clustered
  ceph-dashboard/2 error idle 10.244.41.5 hook failed: "update-status"
  filebeat/47 active idle 10.244.41.5 Filebeat ready.
  logrotated/46 active idle 10.244.41.5 Unit is ready.
  nrpe/53 active idle 10.244.41.5 5666/tcp icmp Ready
  prometheus-grok-exporter/47 active idle 10.244.41.5 9144/tcp Unit is ready
  telegraf/47 active idle 10.244.41.5 9103/tcp Monitoring ceph-mon/0 (source version/commit 23.07)
  ubuntu-advantage/47 active idle 10.244.41.5 Attached (esm-apps,esm-infra)
=======

In the ceph-dashboard logs we see that the set-ssl-certificate command failed:
=======
2023-10-08 13:17:07 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 Operator Framework 1.5.2 up and running.
2023-10-08 13:17:07 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 Legacy hooks/update-status does not exist.
2023-10-08 13:17:07 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 yaml does not have libyaml extensions, using slower pure Python yaml loader
2023-10-08 13:17:07 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 Using local storage: /var/lib/juju/agents/unit-ceph-dashboard-2/charm/.unit-state.db already exists
2023-10-08 13:17:07 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 Re-emitting <TLSConfigReady via CephDashboardCharm/CAClient[certificates]/on/tls_server_config_ready[147]>.
2023-10-08 13:17:13 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 Empty key/cert pair :
Key True,
Certs: True
2023-10-08 13:17:13 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 Attempting to collect TLS config from relation
2023-10-08 13:17:14 DEBUG unit.ceph-dashboard/2.update-status logger.go:60 Updating certificates in /etc/ssl/certs...
2023-10-08 13:17:15 DEBUG unit.ceph-dashboard/2.update-status logger.go:60 0 added, 0 removed; done.
2023-10-08 13:17:15 DEBUG unit.ceph-dashboard/2.update-status logger.go:60 Running hooks in /etc/ca-certificates/update.d...
2023-10-08 13:17:15 DEBUG unit.ceph-dashboard/2.update-status logger.go:60 done.
2023-10-08 13:17:15 DEBUG unit.ceph-dashboard/2.juju-log server.go:325 ['ceph', 'dashboard', 'set-ssl-certificate', 'juju-09be5d-0-lxd-2', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')]
2023-10-08 13:17:16 WARNING unit.ceph-dashboard/2.update-status logger.go:60 Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: [('x509 certificate routines', '', 'key values mismatch')]
2023-10-08 13:17:16 ERROR unit.ceph-dashboard/2.juju-log server.go:325 Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/./src/charm.py", line 769, in <module>
    main(CephDashboardCharm)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/main.py", line 429, in main
    framework.reemit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 794, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/./src/charm.py", line 765, in _enable_ssl_from_relation
    self._configure_tls_from_relation()
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/./src/charm.py", line 726, in _configure_tls_from_relation
    self._configure_tls(key, cert, ca_cert, self.TLS_VAULT_CA_CERT_PATH)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/./src/charm.py", line 583, in _configure_tls
    ceph_utils.dashboard_set_ssl_certificate(
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/charms_ceph/utils.py", line 3610, in _dashboard_set_ssl_artifact
    subprocess.check_call(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'set-ssl-certificate', 'juju-09be5d-0-lxd-2', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')]' returned non-zero exit status 5.
=======

The logs don't show extra context on why this command failed, but exit status 5 indicates some kind of permission denied.

All the configs and logs can be found here: https://oil-jenkins.canonical.com/artifacts/5fbb800e-9335-42d6-a3fe-b782ce4a5661/index.html

tags: added: cdo-qa foundations-engine
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.