prometheus restart needed when new target relations appear

Bug #1940565 reported by Adam Dyess
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Prometheus2 charm
New
Undecided
Unassigned

Bug Description

Many times I do not see prometheus scraping the endpoints which have been provided via the target relation and which have been written to the config file /var/snap/prometheus/*/prometheus.yml.

I will check the relation data is correct

$ juju run -u telegraf/<unit> "relation-get -r prometheus-client:$rid - <unit>"

then check the config file

$ juju ssh prometheus/0 "cat /var/snap/prometheus/*/prometheus.yml" | less

to find the IP and PORT of the target prometheus should scrape and it's in the config

But it requires the prometheus service to be restarted

$ juju run -u prometheus/0 "service snap.prometheus.prometheus restart"

Shouldn't this be done automatically when a new target relation is changed -- or at least anytime the config file is updated

Revision history for this message
Stephan Pampel (stephanpampel) wrote :

In general Prometheus does not need to restart to reload its' config: "Prometheus can reload its configuration at runtime." https://prometheus.io/docs/prometheus/latest/configuration/configuration/

A reload can be triggered via SIGHUP process signal or http request to '<ip>:<port>/-/reload' if prometheus runs with --web.enable-lifecycle.

Revision history for this message
Stephan Pampel (stephanpampel) wrote :

The charm uses SIGHUP by default whenever the flag set_state("prometheus.do-reload") is present (e.g. after config change https://git.launchpad.net/charm-prometheus2/tree/src/reactive/prometheus.py#n425)

It looks to me that the code is correctly sending the signal to prometheus. https://git.launchpad.net/charm-prometheus2/tree/src/reactive/prometheus.py#n580

There was a previous bug about the reloading: https://bugs.launchpad.net/charm-prometheus2/+bug/1891942
and it go fixed:
https://git.launchpad.net/charm-prometheus2/commit/src/reactive/prometheus.py?id=798b492c33bc2a3c83ac0c096aaf8d812ef1d997

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.