Ubuntu
gnocchi package

IndexError: index is out of bounds for axis with size

Bug #1857175 reported by Larry Lile
30
This bug affects 6 people
Affects Status Importance Assigned to Milestone
gnocchi (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

I am receiving the following trace from gnocchi

2019-12-20 16:35:06,334 [31] ERROR gnocchi.chef: Error processing new measures
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gnocchi/chef.py", line 165, in process_new_measures_for_sack
    for metric in metrics
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 714, in add_measures_to_metrics
    before_truncate_callback=_map_compute_splits_operations,
  File "/usr/lib/python2.7/site-packages/gnocchi/carbonara.py", line 364, in set_values
    return_value = before_truncate_callback(self)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 673, in _map_compute_splits_operations
    tstamp = max(bound_timeserie.first, measures['timestamps'][0])
IndexError: index 0 is out of bounds for axis 0 with size 0

Unfortunately there is not enough information here to determine what measurement is causing the issue.

Could gnocchi/chef.py be modified to try/catch the single failed self.storage.add_measures_to_metrics individually instead of dropping the remainder of the measurements?

        try:
            with self.incoming.process_measures_for_sack(sack) as measures:
                # process only active metrics. deleted metrics with unprocessed
                # measures will be skipped until cleaned by janitor.
                if not measures:
                    return 0

                metrics = self.index.list_metrics(
                    attribute_filter={
                        "in": {"id": measures.keys()}
                    })
-------- try:
                self.storage.add_measures_to_metrics({
                    metric: measures[metric.id]
                    for metric in metrics
                })
-------- except:
                Report failed metric and any details?

                LOG.debug("Measures for %d metrics processed",
                          len(metrics))
                return len(measures)
        except Exception:
            if sync:
                raise
            LOG.error("Error processing new measures", exc_info=True)
            return 0
        finally:
            lock.release()

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in gnocchi (Ubuntu):
status: New → Confirmed
Revision history for this message
Rizky (ridz) wrote :

I have the same problem. This error log begins showing once I set the archive-policy granularity to 3s as well as setting the polling interval in ceilometer to 3 seconds. This error never showing days before when the granularity and the polling interval is both at 3600s

I think the cause is because some of the metrics has zero value, or perhaps because of the error it became zero?

gnocchi aggregates '(/ (aggregate rate:mean (metric cpu mean)) 3000000000)' id=65592309-1e03-4c9e-8696-af7ebe30583c | tail

| aggregated | 2021-03-31T21:16:48+00:00 | 3.0 | 0.0 |
| aggregated | 2021-03-31T21:16:54+00:00 | 3.0 | 0.01 |
| aggregated | 2021-03-31T21:16:57+00:00 | 3.0 | 0.006666666666666667 |
| aggregated | 2021-03-31T21:17:00+00:00 | 3.0 | 0.006666666666666667 |
| aggregated | 2021-03-31T21:17:03+00:00 | 3.0 | 0.0033333333333333335 |
| aggregated | 2021-03-31T21:17:09+00:00 | 3.0 | 0.008333333333333333 |
| aggregated | 2021-03-31T21:17:12+00:00 | 3.0 | 0.008333333333333333 |
| aggregated | 2021-03-31T21:17:15+00:00 | 3.0 | 0.0 |
| aggregated | 2021-03-31T21:17:18+00:00 | 3.0 | 0.01 |
+------------+---------------------------+-------------+-----------------------+

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gnocchi/chef.py", line 163, in process_new_measures_for_sack
    self.storage.add_measures_to_metrics({
  File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 675, in add_measures_to_metrics
    keys_and_splits_to_store) = ts.set_values(
  File "/usr/lib/python3/dist-packages/gnocchi/carbonara.py", line 367, in set_values
    return_value = before_truncate_callback(self)
  File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 638, in _map_compute_splits_operations
    tstamp = max(bound_timeserie.first, measures['timestamps'][0])
IndexError: index 0 is out of bounds for axis 0 with size 0

My installed versions :

gnocchi-api/groovy,now 4.4.0-0ubuntu1 all [installed]
gnocchi-common/groovy,now 4.4.0-0ubuntu1 all [installed,automatic]
gnocchi-metricd/groovy,now 4.4.0-0ubuntu1 all [installed]
gnocchi-statsd/groovy 4.4.0-0ubuntu1 all

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.