StackLight

The FS alert threshold for Elasticsearch volume is not coherent with the default disk-based shard allocation policy

Bug #1595117 reported by Swann Croiset
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StackLight
Fix Released
Medium
Swann Croiset
StackLight 0.10.0

Bug Description

Elasticsearch shard allocation depends on available disk space:
low-watermark: 85%
high-watermak: 90%

see https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

Currently, StackLight thresholds are:
WARN: 10% free space
CRITICAL: 5% free space

https://github.com/openstack/fuel-plugin-lma-collector/blob/master/deployment_scripts/puppet/modules/fuel_lma_collector/templates/alarming.yaml.erb#L685-L699

The thresholds must be coherent with the ES settings and the operator must be alerted before ES reaches his watermarks:

WARN: 16% free space
CRITICAL: 11% free space

Swann Croiset (swann-w)
Changed in lma-toolchain:
assignee: nobody → Swann Croiset (swann-w)
milestone: none → 0.10.0
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Simon Pasquier (simon-pasquier) wrote :

This is somehow related to https://bugs.launchpad.net/lma-toolchain/+bug/1587002

tags: added: customer-found
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (master)

Fix proposed to branch: master
Review: https://review.openstack.org/333161

Changed in lma-toolchain:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (master)

Reviewed: https://review.openstack.org/333161
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=86849c87f6fae9ed60badee2dc66c43476757fec
Submitter: Jenkins
Branch: master

commit 86849c87f6fae9ed60badee2dc66c43476757fec
Author: Swann Croiset <email address hidden>
Date: Thu Jun 23 10:02:55 2016 +0200

    Increase free space thresholds for Elasticsearch volume

    The new thresholds detect free space exhaustion before
    Elasticsearch reaches its watermarks used by the disk-based shard
    allocation policy.

    Change-Id: I44893e6467c8b4766400d543c2c4d62c5d452dbd
    Closes-bug: #1595117

Changed in lma-toolchain:
status: In Progress → Fix Committed
Simon Pasquier (simon-pasquier)
Changed in lma-toolchain:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.