During the Cloud operation stage, some of controllers may go down and return back - as a planned downtime or as a sudden failure (and that could happen even more often in a case of a network flapping). This is OK from the HA cluster health perspective. But
the number of Rabbitmq queues grows every time Openstack services reconnect to the AMQP server. In a long time that could end up running out of any connection limits or a major performance issues.
For example, in the files attached you can see how initial amount of ~100 queues grew up to the 1500 after a 100 failovers.
Steps to reproduce:
1. Deploy any HA environment with at least 3 controllers and collect info about queues from any controller:
"rabbitmqctl list_queues pid name arguments | column -t"
2. Shutdown any controller non gracefully (or kill with -9 its beam process)
3. Wait until OSTF HA health check will pass w/o failures
4. Bring the 'failed' controller up again (or skip steps 4-5, if you have killed a beam process instead)
5. Wait until OSTF HA health check will pass w/o failures
6. Collect info about queues from any controller again and compare it with the previous results.
7. Wait for not less than 60 minutes and repeat steps 2-6.
As a result you will see that some of the queues created more than 1h ago with 'x-expires' set to 1h will persist.
That happens likely due to Openstack services do not release an unused or obsolete or temporary (uuid based) queues after a reconnect. Looks like this issue cannot be addressed by Fuel but should be addressed by Oslo.messaging instead.
{"build_id": "2014-11-20_18-11-27", "ostf_sha": "8be702d5cafcb78ac1d93e5fc27d9272cce4773e", "build_number": "121", "auth_required": true, "api": "1.0", "nailgun_sha": "e4c59607c852f1b3761ef06c9c348994c98a7269", "production": "docker", "fuelmain_sha": "3a6833b3e5223c8eacadb1a0a22600813419e9a9", "astute_sha": "46d23177f9c0e53fb7a4729f394d9841b1bae338", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-20_18-11-27", "ostf_sha": "8be702d5cafcb78ac1d93e5fc27d9272cce4773e", "build_number": "121", "api": "1.0", "nailgun_sha": "e4c59607c852f1b3761ef06c9c348994c98a7269", "production": "docker", "fuelmain_sha": "3a6833b3e5223c8eacadb1a0a22600813419e9a9", "astute_sha": "46d23177f9c0e53fb7a4729f394d9841b1bae338", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "666c40d0f5c0bb3dbeaa02b7ba62d038453bae0f"}}}, "fuellib_sha": "666c40d0f5c0bb3dbeaa02b7ba62d038453bae0f"}
Logs snapshot https:/