neutron L3-agent fail to start interfaces

Andrew, this snapshot contains no logs at all, unfortunately. We cannot debug anything without it.

If it is snapshot-generation tool that is broken - please, pull the log files from master and slave nodes and attach them to the bug.

Also, it may be a duplicate of https://bugs.launchpad.net/fuel/+bug/1309087 as it depends on newer upstart script for ovs-agent in Ubuntu.

There is no useful information in https://bugs.launchpad.net/fuel/+bug/1309087 to make it appear to be same.

From /var/log/neutron/l3-agent.log

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py", line 438, in process_router
    p['ip_cidr'], p['mac_address'])
  File "/usr/lib/python2.7/dist-packages/neutron/agent/l3_agent.py", line 707, in internal_network_added
    prefix=INTERNAL_DEV_PREFIX)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 195, in plug
    namespace_obj = ip.ensure_namespace(namespace)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 136, in ensure_namespace
    ip = self.netns.add(name)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 446, in add
    self._as_root('add', name, use_root_namespace=True)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 217, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 70, in _as_root
    namespace)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 81, in _execute
    root_helper=root_helper)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 76, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-27429c22-0ec8-46dc-83e2-13baf13e6241']
Exit code: 1
Stdout: ''
Stderr: 'Could not create /var/run/netns/qrouter-27429c22-0ec8-46dc-83e2-13baf13e6241: File exists\n'

Andrew, the issue I referenced was related to OVS agent not creating ports due to agent not functioning on Ubuntu nodes.

Neutron call 'ip netns list' without using rootwrap:

2014-04-23 16:15:43.806 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None] Running command: ['ip', '-o', 'netns', 'list'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
2014-04-23 16:15:43.828 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None]
Command: ['ip', '-o', 'netns', 'list']
Exit code: 0
Stdout: ''
Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:74
2014-04-23 16:15:43.829 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-b582586e-70e3-4a38-8b19-039f30ce87a9'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
2014-04-23 16:15:43.909 28240 DEBUG neutron.agent.linux.utils [req-d0f812f6-d987-45f5-9cff-11f1fa52fed6 None]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-b582586e-70e3-4a38-8b19-039f30ce87a9']
Exit code: 0
Stdout: ''

HOTFIX (CENTOS):
# vim /usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py
line #469:
    def exists(self, name):
        output = self._parent._execute('o', 'netns', ['list'])

Should be changed to:
    def exists(self, name):
        output = self._parent._execute('o', 'netns', ['list'], root_helper=self._parent.root_helper)

Bug in Neutron: https://bugs.launchpad.net/neutron/+bug/1311804

Reviewed: https://review.openstack.org/89872
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c0e3d76a8a96d17d2f20695a36d170824cd581ca
Submitter: Jenkins
Branch: master

commit c0e3d76a8a96d17d2f20695a36d170824cd581ca
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 23 19:02:41 2014 +0400

    kill all proceses inside all l3-agent's net.namespaces,

    that using ip protocol. When l3 agent start/stop.
    Also remove unnided namespaces.

    Change-Id: I7e5203f6cec646a8b70609f2bf3b785066068aeb
    Closes-Bug: #1310926

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/90221

Reviewed: https://review.openstack.org/90221
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1e290c398712683566a11c78ceea532719e1f72e
Submitter: Jenkins
Branch: stable/4.1

commit 1e290c398712683566a11c78ceea532719e1f72e
Author: Sergey Vasilenko <email address hidden>
Date: Wed Apr 23 19:02:41 2014 +0400

    kill all proceses inside all l3-agent's net.namespaces,

    that using ip protocol. When l3 agent start/stop.
    Also remove unneeded namespaces.

    Change-Id: I7e5203f6cec646a8b70609f2bf3b785066068aeb
    Closes-Bug: #1310926

This commit was applied to the wrong bug.

all the commits for this bug were successfully merged. moving this bug to 4.1.1

needs fix in https://bugs.launchpad.net/fuel/+bug/1310926/comments/10

this fix is already merged into FUEL packages

{
build_id: "2014-06-05_15-25-50",
mirantis: "yes",
build_number: "344",
nailgun_sha: "a828d6b7610f872980d5a2113774f1cda6f6810b",
ostf_sha: "2b7b39e4b6ea89751b65171f24a8e80b5cac56aa",
fuelmain_sha: "9964da7dec34d3100419c1c77c8f5235d8e30f14",
astute_sha: "55df06b2e84fa5d71a1cc0e78dbccab5db29d968",
release: "4.1B",
fuellib_sha: "3511461a2b529619a787a6306441d9039699e71d"
}

Change is merged to 6.0 in the following patch: https://gerrit.mirantis.com/#/c/29630/

Package neutron has been built from changeset: https://review.fuel-infra.org/92
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-92/centos

Package neutron has been built from changeset: https://review.fuel-infra.org/92
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-92/ubuntu

Package neutron has been built from changeset: https://review.fuel-infra.org/92
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

Package neutron has been built from changeset: https://review.fuel-infra.org/92
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

Related https://bugs.launchpad.net/fuel/+bug/1397284

Verify 6.0
iso
 {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

ubuntu neutron vlan, 3 controller + 2 compute / ceph

Steps to reproduce:
1) Create new net, subnet, router
2) lounch instance on this new net
2) associate floating IP to the VM
3) in CLi ping floating IP
4) go with ssh to the VM

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Sergey Vasilenko <email address hidden>
Review: https://review.fuel-infra.org/7996

Change abandoned by Oleg Bondarev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/7996
Reason: Not needed now as from Kilo there is an option to use root rights when listing namespaces, True by default use_helper_for_ns_read