Gnocchi

gnocchi-metricd fails to start

Bug #1470787 reported by Tomer Shtilman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi
Fix Released
Critical
Julien Danjou
Gnocchi 1.1.0

Bug Description

ceilometer compute agent is posting metric to gnocchi api (every 10 secs) , all rest requests requests ending with success, i have added some printing to the log (in _carbonara.py) I can the values coming through to gnocchi

2015-07-02 09:02:03.387083 for metric 646b678f-2f71-448b-97f9-217383eaeaec meausre is [(datetime.datetime(2015, 7, 2, 9, 2, 3), 98.5)]
2015-07-02 09:02:13.411970 for metric 646b678f-2f71-448b-97f9-217383eaeaec meausre is [(datetime.datetime(2015, 7, 2, 9, 2, 13), 98.1)]
2015-07-02 09:02:23.614727 for metric 646b678f-2f71-448b-97f9-217383eaeaec meausre is [(datetime.datetime(2015, 7, 2, 9, 2, 23), 101.6)]
.....

when i am fetching the measures I get an empty list

GET /v1/resource/instance/470efe10-2916-413f-ac6c-3a15565118cb/metric/cpu_util/measures HTTP/1.1
Host: 135.248.18.191:8041
X-Auth-Token: d41c666abf9c48b89b22d504af06bb2f
Cache-Control: no-cache
[]

or by metric

GET /v1/metric/646b678f-2f71-448b-97f9-217383eaeaec/measures HTTP/1.1
Host: 135.248.18.191:8041
X-Auth-Token: d41c666abf9c48b89b22d504af06bb2f
Cache-Control: no-cache
[]

i have also used the carbonara dump to dump the metric calculated

root@tshtilma-gnocchi-devstack:/var/log/apache2# carbonara-dump /opt/stack/data/gnocchi/646b678f-2f71-448b-97f9-217383eaeaec/count

Aggregation method: count
Number of aggregated timeseries: 3
Back window: 0 × 86400s = 0s

Number of full resolution measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

Aggregated timeserie #1: 300s × 12 = 1:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

Aggregated timeserie #2: 3600s × 24 = 1 day, 0:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

Aggregated timeserie #3: 86400s × 30 = 30 days, 0:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

root@tshtilma-gnocchi-devstack:/var/log/apache2# carbonara-dump /opt/stack/data/gnocchi/646b678f-2f71-448b-97f9-217383eaeaec/max
Aggregation method: max
Number of aggregated timeseries: 3
Back window: 0 × 86400s = 0s

Number of full resolution measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

Aggregated timeserie #1: 300s × 12 = 1:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

Aggregated timeserie #2: 3600s × 24 = 1 day, 0:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

Aggregated timeserie #3: 86400s × 30 = 30 days, 0:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
+-----------+-------+
+-----------+-------+

I have tried both in swift and a file, got the same result

tried to fix the bug ,its look a bit too complicated for me

my local.conf
[[local|localrc]]
ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken
enable_plugin gnocchi https://github.com/openstack/gnocchi master

enable_service q-lbaas
enable_service ceilometer-api,ceilometer-collector
enable_service ceilometer-acompute
enable_service ceilometer-alarm-notifier,ceilometer-alarm-evaluator
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
#enable_service heat,h-api,h-eng,h-api-cfn
enable_service gnocchi-api
IMAGE_URLS="http://10.5.5.121/Installs/Templates/cloudify/CENTOS/KVM/tinyCentos.qcow2"
#GNOCCHI_COORDINATOR_URL=redis://localhost:6379?timeout=5
#Enable an eager processing of the ceilometer pipeline (every 10sec):
CEILOMETER_PIPELINE_INTERVAL=10

[[post-config|$GNOCCHI_CONF]]
[archive_policy]
default_aggregation_methods = mean,min,max,sum,std,median,count,last,95pct

Revision history for this message
Julien Danjou (jdanjou) wrote :

Is the problem happening without Ceilometer involved?

Changed in gnocchi:
status: New → Incomplete
Revision history for this message
Tomer Shtilman (tomer-shtilman) wrote :

yes also happening without ceilometer

1.creating a custom metric

POST /v1/metric HTTP/1.1
Host: 135.248.18.191:8041
Content-Type: application/json
X-Auth-Token: de092cd158a140dcbf1e2b59614d1463
Cache-Control: no-cache

{
  "name": "my_metric",
  "archive_policy_name": "low"
}

response is:

{
  "created_by_user_id": "7870737d-e615-40f8-9cf3-d4cc3e0b78ce",
  "name": "my_metric",
  "resource_id": null,
  "created_by_project_id": "fb14bbed-0704-4065-828f-9e237424cb8a",
  "archive_policy": {
    "definition": [
      {
        "points": 12,
        "timespan": "1:00:00",
        "granularity": "0:05:00"
      },
      {
        "points": 24,
        "timespan": "1 day, 0:00:00",
        "granularity": "1:00:00"
      },
      {
        "points": 30,
        "timespan": "30 days, 0:00:00",
        "granularity": "1 day, 0:00:00"
      }
    ],
    "back_window": 0,
    "name": "low",
    "aggregation_methods": [
      "std",
      "count",
      "last",
      "95pct",
      "min",
      "max",
      "sum",
      "median",
      "mean"
    ]
  },
  "id": "6c421661-fdb6-4d11-9baf-fb6fd42928f8"
}

2.posting measures to metric 6c421661-fdb6-4d11-9baf-fb6fd42928f8

POST /v1/metric/6c421661-fdb6-4d11-9baf-fb6fd42928f8/measures HTTP/1.1
Host: 135.248.18.191:8041
Content-Type: application/json
X-Auth-Token: de092cd158a140dcbf1e2b59614d1463
Cache-Control: no-cache

[
  {
    "timestamp": "2015-01-07T16:17:57",
    "value": 43.1
  },
  {
    "timestamp": "2015-01-07T16:17:47",
    "value": 40.1
  }
]

3.getting measures for metric

GET /v1/metric/6c421661-fdb6-4d11-9baf-fb6fd42928f8/measures HTTP/1.1
Host: 135.248.18.191:8041
Content-Type: application/json
X-Auth-Token: de092cd158a140dcbf1e2b59614d1463
Cache-Control: no-cache
empty response
[]

4.checking also with carbonara-dump , count is 0

root@tshtilma-gnocchi-devstack:/var/log/apache2# carbonara-dump /opt/stack/data/gnocchi/6c421661-fdb6-4d11-9baf-fb6fd42928f8/count
Aggregation method: count
Number of aggregated timeseries: 3
Back window: 0 × 86400s = 0s

Number of full resolution measures: 0
+-----------+-------+
| Timestamp | Value |
 +-----------+-------+
+-----------+-------+

Aggregated timeserie #1: 300s × 12 = 1:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
 +-----------+-------+
+-----------+-------+

Aggregated timeserie #2: 3600s × 24 = 1 day, 0:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
 +-----------+-------+
+-----------+-------+

Aggregated timeserie #3: 86400s × 30 = 30 days, 0:00:00
Number of measures: 0
+-----------+-------+
| Timestamp | Value |
 +-----------+-------+
+-----------+-------+

Revision history for this message
Julien Danjou (jdanjou) wrote :

is gnocchi-metricd running?

Revision history for this message
Tomer Shtilman (tomer-shtilman) wrote :

tried to start metricd manually
stack@tshtilma-gnocchi-devstack:~$ gnocchi-metricd
2015-07-02 13:50:04.372 7421 WARNING oslo_config.cfg [-] Option "username" from group "keystone_authtoken" is deprecated. Use option "user-name" from group "keystone_authtoken".
2015-07-02 13:50:04.385 7421 CRITICAL gnocchi [-] PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
2015-07-02 13:50:04.385 7421 ERROR gnocchi Traceback (most recent call last):
2015-07-02 13:50:04.385 7421 ERROR gnocchi File "/usr/local/bin/gnocchi-metricd", line 10, in <module>
2015-07-02 13:50:04.385 7421 ERROR gnocchi sys.exit(metricd())
2015-07-02 13:50:04.385 7421 ERROR gnocchi File "/opt/stack/gnocchi/gnocchi/cli.py", line 79, in metricd
2015-07-02 13:50:04.385 7421 ERROR gnocchi p.map(_wrap_metricd, range(conf.metricd.workers))
2015-07-02 13:50:04.385 7421 ERROR gnocchi File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
2015-07-02 13:50:04.385 7421 ERROR gnocchi return self.map_async(func, iterable, chunksize).get()
2015-07-02 13:50:04.385 7421 ERROR gnocchi File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
2015-07-02 13:50:04.385 7421 ERROR gnocchi raise self._value
2015-07-02 13:50:04.385 7421 ERROR gnocchi PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
2015-07-02 13:50:04.385 7421 ERROR gnocchi

Revision history for this message
Julien Danjou (jdanjou) wrote :

Thanks, that's a bug indeed. I'm not sure why we don't see it in devstack-gate. Maybe Gabbi runs it differently in a thread?

Changed in gnocchi:
status: Incomplete → Triaged
importance: Undecided → Critical
assignee: nobody → Julien Danjou (jdanjou)
Julien Danjou (jdanjou)
summary: - measures in gnocchi are empty
+ gnocchi-metricd fails to start
OpenStack Infra (hudson-openstack)
Changed in gnocchi:
status: Triaged → In Progress
OpenStack Infra (hudson-openstack)
Changed in gnocchi:
status: In Progress → Fix Committed
Julien Danjou (jdanjou)
Changed in gnocchi:
milestone: none → 1.1.0rc1
Julien Danjou (jdanjou)
Changed in gnocchi:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.