Monitor expiration of OVN certs

Hello, Giuseppe, and thank you for your bug report.

Providing a way to easily monitor certificate lifetime in a system that make use of PKI for authentication and authorization is indeed important.

The OVN charms make use of Juju to integrate with an application providing the Certificate Authority and automation for Certificate Issuance, for example Vault.

Would it perhaps make sense to implement monitoring of lifetime of issued certificates in the charm/application responsible for managing the certificates instead?

Hi Frode, thanks for your reply.

If I read correctly you are suggesting to implement the check on the application providing the certs, which in our cases for OVN is 100% of times Vault.

I see some pros and cons of this choice:

Pros:
* we get out of the box monitoring for all certs provided by Vault, i.e. OpenStack API certs, Octavia certs, OVN certs etc.

Cons:
* We are only monitoring the certs that are in Vault. If the distribution of these certs fails, see LP#1940549, our monitoring will miss that as it will think the certs are all renewed but in reality the certs in the ovn-chassis or ovn-central units are still the old ones as Vault failed to provide the updated ones.

To be on the safe side, I think we should always monitor what is actually being used by the applications so I would like to see these checks in the OVN charms.

https://review.opendev.org/c/x/charm-ovn-central/+/887411

Having looked into this a bit further I think it might actually not be possible. Nrpe checks run as the nagios user and the ovn certs have the following permissions:

# ll /etc/ovn/
total 24
dr-xr-xr-x 2 root root 4096 Jun 23 13:07 ./
drwxr-xr-x 101 root root 4096 Jul 3 12:44 ../
-rw-r----- 1 root root 1532 Jun 23 13:07 cert_host
-rw-r----- 1 root root 1674 Jun 23 13:07 key_host
-rw-r--r-- 1 root root 1244 Jul 3 10:50 ovn-central.crt
-rw-r----- 1 root root 211 Jun 23 13:07 ovn-northd-db-params.conf

So an nrpe check cannot read the cert file unless its permissions are opened up which I assume is not what anyone wants to do.

It has been brought to my attention that we do have some charms with nrpe checks for root owned resources and they are implemented using a cron job that runs as root and performs the check itself while the nrpe check checks the output of the cron check e.g. rabbitmq-server:

https://github.com/openstack/charm-rabbitmq-server/blob/3c155e2bdaeaec5090111749a2cf366b55875575/hooks/rabbit_utils.py#L1630

Another option is to use the update-status hook, which runs by default every 5 minutes, to perform the check then store the output in a location that the nrpe check can read.

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/887411
Committed: https://opendev.org/x/charm-ovn-central/commit/e3502b2c9c5f65ee8c0f3ed1aff8ef66c94de2f1
Submitter: "Zuul (22348)"
Branch: master

commit e3502b2c9c5f65ee8c0f3ed1aff8ef66c94de2f1
Author: Edward Hope-Morley <email address hidden>
Date: Fri Jun 30 17:27:22 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: I7c7cd238ddf3fd9f92bfa5879d19d78c091cf2ac

Fix proposed to branch: stable/22.09
Review: https://review.opendev.org/c/x/charm-ovn-central/+/895021

Fix proposed to branch: stable/23.03
Review: https://review.opendev.org/c/x/charm-ovn-central/+/895022

Fix proposed to branch: master
Review: https://review.opendev.org/c/x/charm-ovn-chassis/+/896409

decide not to implement it in charm-openstack-service-checks , so mark it as invalid