Fuel for OpenStack

Ceilometer Mongo DB backend cannot failover

Bug #1371799 reported by Tatyanka on 2014-09-19

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Released	High	Igor Degtiarov	Fuel for OpenStack 6.1
5.0.x	Fix Committed	High	Unassigned	Fuel for OpenStack 5.0-updates
5.1.x	Fix Committed	High	Unassigned	Fuel for OpenStack 5.1.1-updates
6.0.x	Fix Committed	High	Unassigned	Fuel for OpenStack 6.0-updates

Bug Description

{"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Steps to reproduce:
1. Ubuntu with nova
3 controllers with mongo + ceiloemeter
1 compute
2. Deploy
3. Run ostf
4. ssh on primary controller and drop traffic to the mongo port
iptables -I OUTPUT 1 -p tcp --dport 27017 -j DROP
iptables -I INPUT 1 -p tcp --dport 27017 -j DROP

5. Run ostf
6. ssh on other controller and request alarm-list

Actual result:
OSTF tests are failed. alarm list failed. Seems we do not monitor mongo at all

Expected:
mongo runs under crm - so in case of failover - user data will be available

INFO: the same result if I kill mongo pid , but it works fine in case if node was shut down

Tags:

Revision history for this message

Tatyanka (tatyana-leontovich) wrote on 2014-09-19:

fuel-snapshot-2014-09-19_21-55-44.tgz Edit (68.2 MiB, application/x-tar)

summary:

- Ceilometer randomply fails in ha if it is mongo was blocked on primary
+ Ceilometer randomply fails in ha if mongo was blocked on primary
controller

Bogdan Dobrelya (bogdando) on 2014-09-19

Changed in fuel:
status:	New → Confirmed

Vladimir Kuklin (vkuklin) on 2014-10-09

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Sergii Golovatiuk (sgolovatiuk)

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2014-11-14: Re: Ceilometer randomply fails in ha if mongo was blocked on primary controller

This needs to be converted to a blueprint for next release.

Changed in fuel:
milestone:	6.0 → 6.1

Tomasz 'Zen' Napierala (tzn) on 2014-11-24

summary:

- Ceilometer randomply fails in ha if mongo was blocked on primary
+ Ceilometer randomly fails in HA if mongo was blocked on primary
controller

Sergii Golovatiuk (sgolovatiuk) on 2014-12-08

Changed in fuel:
assignee:	Sergii Golovatiuk (sgolovatiuk) → nobody
assignee:	nobody → Fuel Library Team (fuel-library)

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-12-11: Re: Ceilometer randomly fails in HA if mongo was blocked on primary controller

could be related https://bugs.launchpad.net/mos/+bug/1380800

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-01-14:

Matthew, which excatly blueprint do you mean? The issue looks like not a feature request or an improvement

summary:

- Ceilometer randomly fails in HA if mongo was blocked on primary
- controller
+ Ceilometer Mongo DB backend cannot failover

Revision history for this message

Ivan Berezovskiy (iberezovskiy) wrote on 2015-02-06:

This patch https://review.fuel-infra.org/#/c/2168/ should fix this problem.

I have verified it on ISO:

{"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}

Changed in fuel:
status:	Confirmed → Fix Committed

Bogdan Dobrelya (bogdando) on 2015-02-12

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Igor Degtiarov (idegtiarov)

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-03-18:

Verified on ISO #202

"build_id": "2015-03-16_22-54-44", "ostf_sha": "e86c961ceacfa5a8398b6cbda7b70a5f06afb476", "build_number": "202", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-16_22-54-44", "ostf_sha": "e86c961ceacfa5a8398b6cbda7b70a5f06afb476", "build_number": "202", "api": "1.0", "nailgun_sha": "874df0d06e32f14db77746cfeb2dd74d4a6e528c", "production": "docker", "python-fuelclient_sha": "2509c9b72cdcdbe46c141685a99b03cd934803be", "astute_sha": "93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "608b72a6f79a719cf01c35a19d0091fe20c8288a", "fuellib_sha": "924d73ae4766646e1c3a44d7b59c4120985e45f0"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "874df0d06e32f14db77746cfeb2dd74d4a6e528c", "production": "docker", "python-fuelclient_sha": "2509c9b72cdcdbe46c141685a99b03cd934803be", "astute_sha": "93e427ac49109fa3fd8b0e1d0bb3d14092be2e8c", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "608b72a6f79a719cf01c35a19d0091fe20c8288a", "fuellib_sha": "924d73ae4766646e1c3a44d7b59c4120985e45f0"