stop polling per pipeline

Bug #1480442 reported by gordon chung
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
gordon chung

Bug Description

with a pipeline

sources:
    - name: meter_source
      interval: 5
      meters:
          - "*"
      sinks:
          - meter_sink
sinks:
    - name: meter_sink
      transformers:
      publishers:
          - udp://

everything will be polled once. makes sense.

with a pipeline:

***updated (31/07/15)***
sources:
    - name: meter_source
      interval: 600
      meters:
          - "*"
      sinks:
          - meter_sink
    - name: cpu_source
      interval: 600
      meters:
          - "cpu"
      sinks:
          - cpu_sink
sinks:
    - name: meter_sink
      transformers:
      publishers:
          - notifier://
    - name: cpu_sink
      transformers:
          - name: "rate_of_change"
            parameters:
                target:
                    name: "cpu_util"
                    unit: "%"
                    type: "gauge"
                    scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"
      publishers:
          - notifier://

cpu is polled twice. this isn't needed.

Revision history for this message
Chris Dent (cdent) wrote :

From my standpoint this is not a bug. Given the pipeline you have create you are saying:

* create 2 sources
* each source will poll all meters
* each source will send its meters to meter_sink

It's not the job of a sink to remove duplicates. It _is_ the job of a source to declare what is polled. If you write that everything should be polled twice, then so be it, yeah?

Revision history for this message
gordon chung (chungg) wrote :

i think this is a side effect of removing pipeline from polling agent -- the pipeline functionality doesn't actually apply anymore. if i put this in context of current pipeline:

sources:
    - name: meter_source
      interval: 600
      meters:
          - "*"
      sinks:
          - meter_sink
    - name: cpu_source
      interval: 600
      meters:
          - "cpu"
      sinks:
          - cpu_sink
sinks:
    - name: meter_sink
      transformers:
      publishers:
          - notifier://
    - name: cpu_sink
      transformers:
          - name: "rate_of_change"
            parameters:
                target:
                    name: "cpu_util"
                    unit: "%"
                    type: "gauge"
                    scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"
      publishers:
          - notifier://

this will poll cpu twice. i don't need to poll twice. i just want the cpu meter and i want the cpu_util value i can derive from it.

description: updated
Revision history for this message
gordon chung (chungg) wrote :

i think we should be creating unique polling tasks. you can poll the same thing, they just need to be at different intervals. also, they should not be multiple of another task. ie. you should have a cpu polling task at 3 seconds and another at 6 seconds.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ceilometer (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/208745

Revision history for this message
gordon chung (chungg) wrote :

this is a two part bug:

1) we poll twice -- this is resolved by https://review.openstack.org/#/c/208745/
2) we build sample twice -- tbh resolved

Changed in ceilometer:
assignee: nobody → gordon chung (chungg)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/209160

Changed in ceilometer:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/208745
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=3ec58ba18e907b277903927e0c98deaa9c42031d
Submitter: Jenkins
Branch: master

commit 3ec58ba18e907b277903927e0c98deaa9c42031d
Author: gordon chung <email address hidden>
Date: Mon Aug 3 20:54:09 2015 -0400

    group pollsters by interval

    there was a regression in code which led to discrete polling tasks
    being created per pipeline. while logical, this meant that the cache
    between tasks with same interval never got shared and therefore
    duplicate calls were made.

    Change-Id: I72876f61a73f4ca2f4ca48aebb1489ce7ebb702e
    Related-Bug: #1480442

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/209160
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=ca1f65bcb114819abd06fbc25006e39b8875ac7a
Submitter: Jenkins
Branch: master

commit ca1f65bcb114819abd06fbc25006e39b8875ac7a
Author: gordon chung <email address hidden>
Date: Tue Aug 4 12:16:02 2015 -0400

    add poll history to avoid duplicate samples

    this is a patch until we properly separate polling logic. this
    patch adds history so if a resource is defined in separate pipelines
    it will only be processed in one. by doing this, we send only a
    single copy of each sample and it can be processed appropriately
    by the notification agent

    this removes duplication check as it is covered by poll history and
    removes the pipeline references in polling.

    Change-Id: I9c1d3e25740fff5281d796c7bfea2f0e105bb5c5
    Closes-Bug: #1480442

Changed in ceilometer:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: liberty-3 → 5.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.