Fuel for OpenStack

[SWARM][8.0] RabbitMQ availability test failure after removing rabbit node

Bug #1598801 reported by Vladimir Jigulin on 2016-07-04

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Confirmed	Medium	MOS Maintenance QA team	Fuel for OpenStack 8.0-updates
	Mitaka	New	Undecided	Fuel QA Team

Bug Description

Reproduced on CI:
https://patching-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/23/testReport/(root)/separate_rabbit_service_add_delete_node/separate_rabbit_service_add_delete_node/

Steps to reproduce:
1. Revert snapshot separate_rabbit_service
2. Add one rabbit node and re-deploy cluster
3. Run network verification
4. Run OSTF
5. Check hiera hosts are the same for different group of roles
6. Delete one rabbit node
7. Run network verification
8. Run OSTF

better use separate_rabbit_service_add_delete_node test from fuel-qa
test fail at line:
https://github.com/openstack/fuel-qa/blob/stable/8.0/fuelweb_test/tests/tests_separate_services/test_separate_rabbitmq.py#L266

Expected results: ostf tests passed successfully

Actual result: 2 ostf tests failed:
- RabbitMQ availability (failure) Number of RabbitMQ nodes is not equal to number of cluster nodes.
- RabbitMQ replication (failure) Failed to establish AMQP connection to 5673/tcp port on 10.109.1.11 from controller node!

Reproducibility: rarely

Snapshot: https://drive.google.com/file/d/0Bw7ZahkM7_sJZVBkd0VxRVRiVmM/view?usp=sharing

Tags:

Vadim Rovachev (vrovachev) on 2016-07-04

Changed in fuel:
milestone:	none → 8.0-updates
assignee:	nobody → MOS Maintenance (mos-maintenance)
importance:	Undecided → Medium

Vitaly Sedelnik (vsedelnik) on 2016-07-22

Changed in fuel:
status:	New → Confirmed

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-08-03:

The similar issue is present on 9.x SWARM test(s)

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.repetitive_restart/15/testReport/(root)/ceph_partitions_repetitive_cold_restart/ceph_partitions_repetitive_cold_restart/

Test scenario:
1. Revert snapshot 'prepare_load_ceph_ha'
2. Wait until MySQL Galera is UP on some controller
3. Check Ceph status
4. Run ostf
5. Fill ceph partitions on all nodes up to 30%
6. Check Ceph status
7. Disable UMM
8. Run RALLY
9. 100 times repetitive reboot:
10. Cold restart of all nodes
11. Wait for HA services ready
12. Wait until MySQL Galera is UP on some controller
13. Run ostf

Note: there is one more similar bug
https://bugs.launchpad.net/fuel/+bug/1495885

Denis Meltsaykin (dmeltsaykin) on 2016-09-02