Murano

Apparently dead connections can pile up in engine/agent rabbitmq

Bug #1460037 reported by Kirill Zaitsev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Murano
Won't Fix
High
Unassigned
Murano newton-3 "n3"
Mitaka
Won't Fix
High
Unassigned
Murano mitaka-rc1 "RC1"
Newton
Confirmed
Undecided
Unassigned
Murano newton-3 "n3"

Bug Description

Seems that engine can leave stale connections in rabbitmq, that are not dropped even if engine process/get's killed.

This can potentially lead to exhaustion of connections in rabbitmq, that would prevent new connections from being established and therefore deploys can hang during execution of agent.call/agent.prepare from muranopl.

The reason for this could be lack of timeout(kombu.Connection's, heartbeat parameter) during connection creation. Therefore connections in rabbitmq do not have a timeout and seem to stay there forever.
The issue can be mitigated by adding heartbeat parameter to murano.conf, that would control this option.

Kirill Zaitsev (kzaitsev)
Changed in murano:
assignee: nobody → Kirill Zaitsev (kzaitsev)
Ekaterina Chernova (efedorova)
Changed in murano:
importance: Undecided → High
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano (master)

Fix proposed to branch: master
Review: https://review.openstack.org/187213

Kirill Zaitsev (kzaitsev)
Changed in murano:
milestone: none → liberty-1
Serg Melikyan (smelikyan)
Changed in murano:
status: In Progress → Fix Released
Serg Melikyan (smelikyan)
Changed in murano:
status: Fix Released → In Progress
Serg Melikyan (smelikyan)
Changed in murano:
milestone: liberty-1 → liberty-2
Serg Melikyan (smelikyan)
Changed in murano:
milestone: liberty-2 → liberty-3
Serg Melikyan (smelikyan)
Changed in murano:
milestone: liberty-3 → liberty-rc1
Revision history for this message
Kirill Zaitsev (kzaitsev) wrote :

Preferred way to prevent this would be to use

[
 {rabbit, [{tcp_listen_options, [binary,
                                {packet, raw},
                                {reuseaddr, true},
                                {backlog, 128},
                                {nodelay, true},
                                {exit_on_close, false},
                                {keepalive, true}]}
          ]}
].

settings for rabbitmq.

Kirill Zaitsev (kzaitsev)
no longer affects: murano/future
Changed in murano:
assignee: Kirill Zaitsev (kzaitsev) → nobody
status: In Progress → Won't Fix
Serg Melikyan (smelikyan)
no longer affects: murano/mitaka
no longer affects: murano/liberty
Changed in murano:
assignee: nobody → Kirill Zaitsev (kzaitsev)
milestone: liberty-rc1 → none
milestone: none → liberty-rc1
Serg Melikyan (smelikyan)
Changed in murano:
assignee: Kirill Zaitsev (kzaitsev) → nobody
Kirill Zaitsev (kzaitsev)
Changed in murano:
milestone: mitaka-3 → mitaka-rc1
Kirill Zaitsev (kzaitsev)
no longer affects: murano/liberty
no longer affects: murano/mitaka
no longer affects: murano/newton
Changed in murano:
status: Confirmed → Won't Fix
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on murano (master)

Change abandoned by Kirill Zaitsev (<email address hidden>) on branch: master
Review: https://review.openstack.org/187213

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.