update_prometheus_rules hook restarts prometheus

Bug #1847472 reported by Ed Stewart
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Prometheus2 charm
Fix Released
High
Stuart Bishop

Bug Description

We are using the prometheus2 charm with relations to some custom charms we have developed.

We have prometheus-client and prometheus-rules relations, however, we have noticed that the prometheus-rules relation hook is firing every 5 minutes in our custom charm.

Our custom charm runs this:

@when('prometheus-rules.available')
def render_prometheus_rules(prometheus_rules):
    # Send a list of rules for alerting to Prometheus
    formatted_rules = []
    template_files = glob.glob(os.path.join(get_templates_dir(), "rule_*.tmpl"))
    for template_file in template_files:
        with open(template_file, 'r') as fd:
            #formatted_rules.append(render_template(fd.read(), context))
            formatted_rules.append(fd.read())

    prometheus_rules.configure("\n".join(formatted_rules))

Then update_prometheus_rules fires within the prometheus2 charm:

@when('prometheus.started')
@when('prometheus-rules.available')
def update_prometheus_rules(prometheus_rules):
    rules = prometheus_rules.prometheus_rules()
    # insert into a template
    template = """
groups:
  - name: JujuRelation
    rules:
{{ rules }}
    """
    with open(PATHS['juju_rules_path'], 'w') as fh:
        fh.write(render_template(template, {'rules': "\n".join(rules)}))
    set_state('prometheus.do-check-reconfig')
    set_state('prometheus.do-restart')

The last line of this code forces a restart of Prometheus.

Therefore our Prometheus is restarting every 5 minutes even though the prometheus-rules aren't changing.

1. Is it correct that the prometheus-rules.available hook fires every 5 minutes? Shouldn't this just fire once?
2. Even if it does fire, why does it need to do a full Prometheus restart, isn't do-check-reconfig sufficient?

thanks,

Ed

Tags: atos

Related branches

Revision history for this message
Ed Stewart (emcs2) wrote :

Prometheus2 charm version as below:

App Version Status Scale Charm Store Rev OS Notes
prometheus active 1 prometheus2 jujucharms 11 ubuntu

Stuart Bishop (stub)
Changed in prometheus2-charm:
status: New → In Progress
assignee: nobody → Stuart Bishop (stub)
Revision history for this message
Stuart Bishop (stub) wrote :
Revision history for this message
Tom Haddon (mthaddon) wrote :

This has been published as cs:prometheus2-12. Leaving the bug status as is for now, but please update to fix released if confirmed.

Joe Guo (guoqiao)
Changed in charm-prometheus2:
importance: Undecided → High
status: In Progress → Fix Committed
Andrea Ieri (aieri)
Changed in charm-prometheus2:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.