Instance can not lease ip on Centos Neutron GRE

astute.yaml contains
role: controller.

I gues, that in the manifests we have part of code, bordered by
if $primary_controller {
......
}

I tried to investigate the same issue with neutron+gre

http://jenkins-product.srt.mirantis.net:8080/job/fuel_master.centos.bvt_1/201/testReport/junit/(root)/deploy_neutron_gre/deploy_neutron_gre/

and found that GRE tunnels from compute nodes to controller were down:

node-1 (controller): 10.108.7.2
node-2 (compute): 10.108.7.3
node-3 (compute): 10.108.7.4

http://paste.openstack.org/show/88190/

So I just restarted 'neutron-openvswitch-agent' on computes and tunnels to controller became UP:

http://paste.openstack.org/show/88191/

Then I restarted instance and it became accessible via private and floating IPs, OSTF test 'Check network connectivity from instance via floating IP' also passed.

Fix proposed to branch: master
Review: https://review.openstack.org/110140

look into logs before for neutron openvswitch agent on node-2 (10.108.5.2/var/log/docker-logs/remote/node-2...).

GRE interface is being created:

<167>Jul 28 16:37:16 node-2 neutron-openvswitch-agent 2014-07-28 16:37:16.374 17387 DEBUG neutron.agent.linux.utils [-]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--may-exist', 'add-port', 'br-tun', 'gre-0a6c0702', '--', 'set', 'Interface', 'gre-0a6c0702', 'type=gre', 'options:remote_ip=10.108.7.2', 'options:local_ip=10.108.7.3', 'options:in_key=flow', 'options:out_key=flow']
Exit code: 0

and then tunneling bridge is destroyed.

<167>Jul 28 16:37:17 node-2 neutron-openvswitch-agent 2014-07-28 16:37:17.634 17387 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-br', 'br-tun'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48
<167>Jul 28 16:37:17 node-2 neutron-openvswitch-agent 2014-07-28 16:37:17.669 17387 DEBUG neutron.agent.linux.ovsdb_monitor [-] Output received from ovsdb monitor: {"data":[["cf16004c-4ff5-471b-b897-ee8bb90cefe4","delete","br-tun",65534],["1eb65dfc-b0a9-411e-ac9b-0c8d6c9019d2","delete","patch-int",1],["0533c85c-e099-435e-9382-69d255f8b2c2","delete","gre-0a6c0704",3],["b391cf59-a61e-43b2-a268-969f136f4583","delete","gre-0a6c0702",2]],"headings":["row","action","name","ofport"]}
 _read_stdout /usr/lib/python2.6/site-packages/neutron/agent/linux/ovsdb_monitor.py:53
<167>Jul 28 16:37:17 node-2 neutron-openvswitch-agent 2014-07-28 16:37:17.678 17387 DEBUG neutron.agent.linux.utils [-]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-br', 'br-tun']
Exit code: 0

this looks really ridiculous

Package neutron has been built from changeset: http://gerrit.mirantis.com/19602
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-19602/centos

Package neutron has been built from changeset: http://gerrit.mirantis.com/19602
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-19602/ubuntu

The issue is reproducible on CentOS nodes in compute roles only.

Logs show that ovs agent initializes all ovs objects, but 2 seconds after it re-creates them from scratch. GRE port however is created only initially and not recreated. The process of recreation is triggered by code that checks if there was ovs restart (this is done by monitoring existence of ovs flow inside artificial table - see https://review.openstack.org/95060). Tunnels are synched only during start of agent. The proposed fix is to sync tunnels every time ovs restart is detected.

The reason why ovs restarts is not known, but most likely the issue is in ovs itself. The issue is reproducible on specific configuration only most probably because ovs-agent starts too soon and is hit by ovs restart.

Proposed fix: https://gerrit.mirantis.com/#/c/19602/

Fixed in upstream https://review.openstack.org/#/c/104517/2

Ported change: https://gerrit.mirantis.com/#/c/19605/

Package neutron has been built from changeset: http://gerrit.mirantis.com/19605
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-19605/centos

Package neutron has been built from changeset: http://gerrit.mirantis.com/19605
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-19605/ubuntu

Package neutron has been built from changeset: http://gerrit.mirantis.com/19605
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos

Package neutron has been built from changeset: http://gerrit.mirantis.com/19605
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/110140

verified

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "377"
  build_id: "2014-07-31_02-07-54"
  astute_sha: "b16efcec6b4af1fb8669055c053fbabe188afa67"
  fuellib_sha: "9a64237c70fc464ef0b11ac7c0bad34e8c202135"
  ostf_sha: "b4c5efa51909404fd9ec1d0bbc38a31b200e1d6d"
  nailgun_sha: "2df0d1ebe4fc7e8870768176b3dd6fcffcfbe261"
  fuelmain_sha: "63d0775708b0f5fa4d6d1e09a316d9c26f7e5444"

Issue is reproduced again on next cluster configuration:
Simple, Ubuntu, Neutron GRE, 1 controller, 2 compute

Instance can't get ip - http://paste.openstack.org/show/88165/

Logs are attached

The new observed issue is not related to the original:
 * GRE endpoints are configured successfully
 * there is no evidence of tunnel reconfiguration

As Ilya said in previous comments the reopening of this issue is not quite correct because originally this bug was fixed and not related to new one. I found a duplicate of new issue: https://bugs.launchpad.net/mos/+bug/1352203 and consider to track all activities there.