Hangs of Neutron l3 agent

I will update this bug with new info when I will reproduce it again.

FYI: don't use 'crm' utility.
CRM utility will be removed at 6.0

please use 'pcs'

There are some AMQP connection traces at the bottom of the log and A LOT near the top.

It would be helpful to have some time stamps of the incident to correlate with the messages in the log.

The log also shows that the service restarted about a dozen times with in a few min at the end of the log. Was this expected?

Please include output of 'strace -p <pid> -s 2048 2>&1 >log' for ~ 1 minute when it locks up. prior to restarting it. Also include ouput of lsof | grep <pid> and the output from top

Please include the neutron server log around the time of the next event ( a few min before and after should work )

Hello Andrew.
See the time stamp: Friday 22 aug 2014. Time start: 00:00

For restarting of agent - it's normal. We troubleshooted a lot.

For now - cloud works fine.

Moved to MOS space, please triage.

I see a number of issues with vpn functionality which can lead to the described result.
So far I am unable to identify the root cause.
If it's possible to get the environment with the issue, I would look at the issue myself.

Apparently the issue is caused by agent-cleanup.py script that removes router interface from qrouter namespace.

We're continuing to work on the issue.

Meg, this information is for you:

Sometimes, the external network connection and floating IP may stop working, but L3 agent has no errors in the logs.
Currently, it's supposed that this is somehow caused by clean-up script that launches or works out in a wrong way, so that it deletes router interfaces that must support a connection to an external network.

There is no stable workaround yet.

A temporary way of getting rid of this bug is the following: to recreate router or reschedule L3 agent.

Here's cleanup script with more logging

Target milestone moved to 6.0 since the bug priority is below High and we're in code freeze for 5.1.

Hey Guys. This Bug affect at one more our customer.
Please feel free to ask operation team about it. We need to fix it asap.

But steps to reproduce still not clear.
One of assumption - if we restart 3 agent only it unable to run till old metadata agents restart.

Raised to critical due to increased # of affected deployments and due to its operations-blocking behavior (actually, it prevents neutron from operating normally)

This
https://review.openstack.org/#/c/125110/2
and this
https://review.openstack.org/#/c/125116/
should fix this bug for 5.1 and 6.0

Looks like forgotten quotation marks on passing arguments to namespace - so only first argument was used and it led to only single namespace creation.

Dmitry Borodanko, it is not clear if changes referenced by Miroslav bug is fixed. I would prefer Neutron team to confirm that, hence moving back to triaged for 5.1 and 6.0.

The fix is verified on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1.1"
  api: "1.0"
  build_number: "27"
  build_id: "2014-10-13_00-01-06"
  astute_sha: "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13"
  fuellib_sha: "46ad455514614ec2600314ac80191e0539ddfc04"
  ostf_sha: "64cb59c681658a7a55cc2c09d079072a41beb346"
  nailgun_sha: "88a94a11426d356540722593af1603e5089d442c"
  fuelmain_sha: "431350ba204146f815f0e51dd47bf44569ae1f6d"

Scenario:
 1. Create Router
 2. Set gateway to external network
 3. Create network with subnets 11.0.0.0/24 and 111.0.0.0/24
 4. Create another network with subnet 12.0.0.0/24
 5. Plug networks into the router
 6. Start VM on any of the networks, check that VM acquired IP and ping to 8.8.8.8 works
 7. Check where L3 agent is running by command 'neutron agent-list'
 8. Verify network configuration: (L3 agent is running on node-1)

(.venv)developer@fuel:stack$ ssh node-1 ip netns
Warning: Permanently added 'node-1' (RSA) to the list of known hosts.
qrouter-8a74bfea-0d27-4b29-92a4-942ab6a552dd
haproxy
qrouter-af4f6833-d01c-4e1c-b9b2-29e5c3f2b6e0

(.venv)developer@fuel:stack$ ssh node-1 ip netns exec qrouter-8a74bfea-0d27-4b29-92a4-942ab6a552dd ip ro
Warning: Permanently added 'node-1' (RSA) to the list of known hosts.
172.18.161.0/24 dev qg-aed693c2-b9 proto kernel scope link src 172.18.161.205
11.0.0.0/24 dev qr-7af18f99-f1 proto kernel scope link src 11.0.0.1
12.0.0.0/24 dev qr-f3512f79-90 proto kernel scope link src 12.0.0.1
111.0.0.0/24 dev qr-1e578c6b-6d proto kernel scope link src 111.0.0.1
default via 172.18.161.1 dev qg-aed693c2-b9

 9. Ask pacemaker to disable-enable L3 agent:
pcs resource disable p_neutron-l3-agent
wait until it is stopped and then:
pcs resource enable p_neutron-l3-agent
 10. L3-agent is moved to other controller (say to node-2)
 11. Verify network configuration:

(.venv)developer@fuel:stack$ ssh node-2 ip netns
Warning: Permanently added 'node-2' (RSA) to the list of known hosts.
qrouter-8a74bfea-0d27-4b29-92a4-942ab6a552dd
qrouter-af4f6833-d01c-4e1c-b9b2-29e5c3f2b6e0
haproxy

(.venv)developer@fuel:stack$ ssh node-2 ip netns exec qrouter-8a74bfea-0d27-4b29-92a4-942ab6a552dd ip ro
Warning: Permanently added 'node-2' (RSA) to the list of known hosts.
172.18.161.0/24 dev qg-aed693c2-b9 proto kernel scope link src 172.18.161.205
11.0.0.0/24 dev qr-7af18f99-f1 proto kernel scope link src 11.0.0.1
12.0.0.0/24 dev qr-f3512f79-90 proto kernel scope link src 12.0.0.1
111.0.0.0/24 dev qr-1e578c6b-6d proto kernel scope link src 111.0.0.1
default via 172.18.161.1 dev qg-aed693c2-b9
 12. Verify that ping from VM still works

Dmitry M: does Ilya's comment #19 confirm that the fixes listed by Miroslav in comment #17 do fix this problem? If so, can we get these ported to 5.0 and 6.0?

Dmitry B: absolutely yes. Actually Miroslav ported fix to 5.0.x as well, we just wait for Neutron team to confirm it works there as well. And sure the team is expected to port fix to 6.0 too.

Merged to master by the following fix https://review.openstack.org/#/c/125110/2

Can you propose the 5.0.x version for stable/5.0?

Had a conversation with Sergey Vasilenko, he will send CR to stable/5.0 for this issue soon.

It was a month since previous comment, any update on a backport for stable/5.0?

Dmitry, fixing 5.0.3 was postponed since 5.0.3 was postponed itself.

We may have postponed the 5.0.3 release, but customers running 5.0.x who have this problem still need a patched package. Now that 6.0 is out, please backport this fix.

Fix verified on 4.1.1

'astute_sha': '55df06b2e84fa5d71a1cc0e78dbccab5db29d968',
 'build_id': '2015-01-09_18-40-26',
 'build_number': '5',
 'fuellib_sha': '469ed82eae57bd85939678054dfa0260e8dbf895',
 'fuelmain_sha': 'fa218a36d2686de6bb36ef8c6b33526c9d802e34',
 'mirantis': 'yes',
 'nailgun_sha': '8cc1dff8ab43f50bd28a9c081d7d06fbd3831b98',
 'ostf_sha': 'f4f15b4d98459650c1945b0efc30290a619be824',
 'release': '4.1.1'

Backport here: https://review.openstack.org/#/c/146994/

The backport to stable/5.0 is here https://review.openstack.org/#/c/147095/