Comment 25 for bug 1599104

Revision history for this message
Hua Zhang (zhhuabj) wrote :

Has this problem been solved? But I still reproduced the problem on Ussuri release in the following way as described in the bug description. Is the workaround mentioned in comment #9 the final best-recommended solution?

1, create a heat ussuri test env

2, queue number before the test is 100 (here's more detailed data - https://paste.ubuntu.com/p/sN5pnSP8Mc/)

# rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |wc -l
100

3, restart heat-engine to trigger the problem

juju ssh heat/0 -- sudo systemctl restart heat-engine

4, queue number has increased from 100 to 108 (here's more detailed data - https://paste.ubuntu.com/p/GWqWhSGyXm/)

# rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |wc -l
108

and two new services (42cee820-4f0c-4aef-b8b6-705e7db3253a 8c12d70c-b00c-4e9b-b33e-7cbf0cb8c510) were created, so we saw:

# rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |grep -E '42cee820-4f0c-4aef-b8b6-705e7db3253a|8c12d70c-b00c-4e9b-b33e-7cbf0cb8c510'
engine_worker.42cee820-4f0c-4aef-b8b6-705e7db3253a 0
engine_worker.8c12d70c-b00c-4e9b-b33e-7cbf0cb8c510 0
heat-engine-listener.42cee820-4f0c-4aef-b8b6-705e7db3253a 0
heat-engine-listener.8c12d70c-b00c-4e9b-b33e-7cbf0cb8c510 0

5, 4550fe68-ecad-457c-b080-29d6b5fb2e7f is a old engine, and it was also soft deleted by L2360[1], but the old queue record was still there so the queues will keep growing.

# rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |grep -E '4550fe68-ecad-457c-b080-29d6b5fb2e7f'
heat-engine-listener.4550fe68-ecad-457c-b080-29d6b5fb2e7f 0
engine_worker.4550fe68-ecad-457c-b080-29d6b5fb2e7f 0

[1] https://github.com/openstack/heat/blob/stable/ussuri/heat/engine/service.py#L2360