[mos] Ceilometer test to check the alarm can change status failed according to metrics are not created

Bug #1288637 reported by Andrey Sledzinskiy
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
MOS Ceilometer
5.0.x
Invalid
Medium
Vadim Rovachev

Bug Description

Bug is reproduced on ISO#235

Steps:
1. Create next env - Centos, HA, KVM, Nova-Network with Vlan, Ceph for volumes and Ceph for images, Ceilometer service
2. Add 3 controllers, 1 compute, 3 Ceph nodes
3. Deploy cluster - deployment is successful
4. Open Health Check tab and run 'Ceilometer test to check the alarm can change status'

Actual result - test failed on step 3. Create metrics for instance. - with timeout error

5. After that allocate more memory for compute and all controller nodes and restart these nodes

Actual result - test failed on step 2. Instance become active. with timeout error

Diagnostic snapshot is attached

Get next error in ceilometer-all.log
libvirtError: Requested operation is not valid: domain is not running
<0>Mar 6 09:03:48 node-4 �<44>ceilometer-ceilometer.compute.pollsters.disk WARNING: Ignoring instance instance-00000021: Requested operation is not valid: domain is not running
<43>Mar 6 09:03:48 node-4 ceilometer-ceilometer.compute.pollsters.disk ERROR: Requested operation is not valid: domain is not running
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ceilometer/compute/pollsters/disk.py", line 87, in get_samples
    instance_name,
  File "/usr/lib/python2.6/site-packages/ceilometer/compute/pollsters/disk.py", line 59, in _populate_cache
    for disk, info in inspector.inspect_disks(instance_name):
  File "/usr/lib/python2.6/site-packages/ceilometer/compute/virt/libvirt/inspector.py", line 150, in inspect_disks
    block_stats = domain.blockStats(device)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1820, in blockStats
    if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
tags: added: ceilometer
tags: added: ostf
Changed in fuel:
milestone: none → 4.1.1
Changed in fuel:
assignee: nobody → Fuel QA Team (fuel-qa)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

In ceilometer meters (metrics) should be created automatically (default metrics for instance). Sometimes by some reason metrics do not created with next error in the logs
http://paste.openstack.org/show/73657/
On this case ceilometer tests fail on step 'Create metrics for instance', according it polls for metric and receive empty list.
There is not 100% reproducible steps, try 7 times - first runs fails with trace in ceilometer, other times - metrics were created:
http://paste.openstack.org/show/73658/

So delete ostf tag from this issue and re-assign from fuel-qa team according the problem is placed in ceilometer api

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
tags: removed: ostf
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → nobody
status: New → Confirmed
summary: - [OSTF] Ceilometer test to check the alarm can change status failed
+ Ceilometer test to check the alarm can change status failed according to
+ metrics are not created
Changed in fuel:
assignee: nobody → Roman Sokolkov (rsokolkov)
milestone: 4.1.1 → 5.0
tags: added: backports-4.1.1
Revision history for this message
Roman Sokolkov (rsokolkov) wrote : Re: Ceilometer test to check the alarm can change status failed according to metrics are not created

Here are two main points to keep in mind.

* Ceilometer+MySQL was transition state to more stable MongoDB backend
  - Unfortunately Mongo moved to 5.0
* This bug most probably not related to deployment itself

My guess this should be checked by pythonic developer familiar with Ceilometer code, because it requires deep diving into sql driver.

Changed in fuel:
assignee: Roman Sokolkov (rsokolkov) → nobody
Changed in fuel:
milestone: 5.0 → 5.1
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Fuel Hardening Team (fuel-hardening)
Ilya Shakhat (shakhat)
Changed in fuel:
assignee: Fuel Hardening Team (fuel-hardening) → MOS Ceilometer (mos-ceilometer)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Need to commit in 5.0.1, because Ceilometer OSTF tests will fail without this fix.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ostf (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/104894

Revision history for this message
Tatyanka (tatyana-leontovich) wrote : Re: Ceilometer test to check the alarm can change status failed according to metrics are not created

Dmitry could you explain why this issue has higt pripory? Does it reproduced on tkhe latest sys tests? As I can see it not reproduced start from June(It really was found on 4.1 and then move from release to realse)

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Move to invalid. Issue affected only 4-x releases, where ceilometer works with mysql, and do not reproduce on 5.0-x(see tests results for the last month)

Dmitry Ilyin (idv1985)
summary: - Ceilometer test to check the alarm can change status failed according to
- metrics are not created
+ [mos] Ceilometer test to check the alarm can change status failed
+ according to metrics are not created
Revision history for this message
Dina Belova (dbelova) wrote :

Invalid for 5.1 due to the Tatyana's comments.

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-ostf (stable/5.0)

Change abandoned by Timur Nurlygayanov (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/104894

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.