Juniper Openstack

[Build 3.0.2.1-10] Rabbitmq queue messages not cleaned up after active schema/svc service crashes

Bug #1669698 reported by Umamaheshwar
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
New
Undecided
Unassigned
R3.2
Fix Committed
Undecided
Édouard Thuleau
Juniper Openstack r3.2.3.0
Trunk
Fix Committed
Undecided
Édouard Thuleau
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

On customer environment running R3.0.2.1-10 build.
When the active schema-transformer crash for some reason, the rabbitmq queue messages keeps increasing. Since customer monitors Rabbitmq services, this could generate an alarm.

root@b05-38-controller:~# rabbitmqctl list_queues name consumers messages | grep -v 0$
Listing queues ...
metering.sample 358
notifications.info363
schema_transformer.node-292.domain.tld075616

The messages are increasing.

Node-292 is one of the controller, on this node, contrail-schema moved to backup mode.

schema logs of all the 3 nodes @10.84.5.8:/users/urao/sscc/schema-issue

Nagabhushana R (bhushana)
Changed in juniperopenstack:
assignee: nobody → Sachin Bansal (sbansal)
Sachin Bansal (sbansal)
Changed in juniperopenstack:
assignee: Sachin Bansal (sbansal) → Édouard Thuleau (ethuleau)
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/29401
Submitter: ?douard Thuleau (<email address hidden>)

Sachin Bansal (sbansal)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/29401
Committed: http://github.org/Juniper/contrail-controller/commit/86e6821e8676cee8a5eb6fd14a92e5c724374736
Submitter: Zuul (<email address hidden>)
Branch: master

commit 86e6821e8676cee8a5eb6fd14a92e5c724374736
Author: Édouard Thuleau <email address hidden>
Date: Wed Mar 22 18:08:05 2017 +0100

[VNC config] Purge remaining AMQP queue

Config services create AMQP queue to consume messages sent by API
server. When they abort, that queue is not deleted and continue to be
filled. Habitually, these services are directly restarted and put in
standby mode (another node should have take the lead). So the queue will
be purge when the service passes active.
That patch proposes to delete the queue if it exists before the service
goes in standby.

Change-Id: Ie1eb1f2a55d879c712d2914964ee1ff0edba9864
Closes-Bug: #1669698

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/29861
Submitter: ?douard Thuleau (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/29861
Committed: http://github.org/Juniper/contrail-controller/commit/54348b0b5e17af1d1658af70e79bbce1972882be
Submitter: Zuul (<email address hidden>)
Branch: master

commit 54348b0b5e17af1d1658af70e79bbce1972882be
Author: Édouard Thuleau <email address hidden>
Date: Fri Mar 24 11:37:35 2017 +0100

[VNC config] Use new class method of ST and DM

New class method was introduced in [1] to destroy properly a
device-manager or schema-transformer instance. This patch use it on
keyboard interruption.

[1] https://review.opencontrail.org/#q,I5283077989769b5818369a73fe3b70e2e353ca01,n,z

Change-Id: Iebf74775d2f0bffc7039ddf5616bd0440c741f60
Closes-bug: #1669698

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/30193
Submitter: ?douard Thuleau (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/30193
Committed: http://github.org/Juniper/contrail-controller/commit/4fc64704b39361e80b0d958a881e43a61cef2ddc
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 4fc64704b39361e80b0d958a881e43a61cef2ddc
Author: Édouard Thuleau <email address hidden>
Date: Wed Mar 22 18:08:05 2017 +0100

[VNC config] Purge remaining AMQP queue

Config services create AMQP queue to consume messages sent by API
server. When they abort, that queue is not deleted and continue to be
filled. Habitually, these services are directly restarted and put in
standby mode (another node should have take the lead). So the queue will
be purge when the service passes active.
That patch proposes to delete the queue if it exists before the service
goes in standby.

Change-Id: Ie1eb1f2a55d879c712d2914964ee1ff0edba9864
Closes-Bug: #1669698
(cherry picked from commit 86e6821e8676cee8a5eb6fd14a92e5c724374736)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.