Fuel for OpenStack

Neutron services are not restarted after config changes in common config sections

  1. Series newton
  2. Bug #1613724
Bug #1613724 reported by Ilya Bumarskov
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Igor Gajsin
Fuel for OpenStack 11.0
Mitaka
Fix Released
High
Igor Gajsin
Fuel for OpenStack 9.2
Newton
Fix Committed
High
Alexey Shtokolov
Fuel for OpenStack 10.1
Ocata
Fix Committed
High
Igor Gajsin
Fuel for OpenStack 11.0

Bug Description

Fuel 9.1 (snapshot-id#138) with DVS plugin 3.0.0

Detailed bug description:
OSTF failed on step "Create instance".

Steps to reproduce:
   - Create env with vCenter and DVS plugin
   - Add following nodes:
         * Controller
         * Compute + Cinder
         * Cinder-vmware
         * Compute-vmware
   - Configure VMware vCenter Settings. Add 2 vSphere clusters and configure Nova Compute instances on controller and compute-vmware.
   - Deploy env
   - Add two nodes with controller role.
   - Deploy changes
   - Shutdown primary controller
   - Run OSTF

Observed behavior:
Instance in nova az can't be created.
Nova-compute logs:

nova.compute.manager [req-a6012749-c2bd-4caa-b7e5-dda2dcb40d34 629f4e545c0e499c8450c890439e6e1f 58c4468a004e44888bf6723925513638 - - -] Instance failed network setup after 1 attempt(s) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager Traceback (most recent call last): 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager bind_host_id=bind_host_id) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 668, in allocate_for_instance 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager self.force_reraise() 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 657, in allocate_for_instance 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 259, in _create_port 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager raise exception.PortBindingFailed(port_id=port_id) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager PortBindingFailed: Binding failed for port bce50f48-71f5-4044-ad1a-89f2301aeb16,

Neutron-server logs:

neutron.plugins.ml2.managers [req-0442810d-f7a2-4260-9183-0a7788cee79d 39c82b217fc5421ca668297e97b0aed8 d5f913b567b84343bcd98abe18ea3603 - - -] Failed to bind port bce50f48-71f5-4044-ad1a-89f2301aeb16 on host node-1.test.domain.local for vnic_type normal using segments [{'segmentation_id': 1000L, 'physical_network': u'physnet2', 'id': u'5bbb93c3-bc9e-4094-ac1c-114ea342046f', 'network_type': u'vlan'}]

neutron agent-list: http://paste.openstack.org/show/558392/

See original description
Ilya Bumarskov (ibumarskov)
description: updated
Changed in fuel:
importance: Undecided → High
milestone: none → 9.1
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Please provide diagnostic snapshot, marking as Incomplete.

Changed in fuel:
status: New → Incomplete
Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :
Changed in fuel:
status: Incomplete → New
Revision history for this message
Vyacheslav Tabolin (slavchick) wrote :

root@node-1:~# cat /var/log/neutron/neutron-openvswitch-agent.log | grep 'Logging enabled!'
2016-08-16 10:02:28.833 24060 INFO neutron.common.config [-] Logging enabled!

It means that neutron-openvswitch-agent never restarted. It can't reach message queue because agent takes conf params at start only.

Oleksiy Molchanov (omolchanov)
tags: added: area-library
Maksim Malchuk (mmalchuk)
no longer affects: fuel/newton
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 10.0 → 10.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/414930

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Igor Gajsin (igajsin)
status: Confirmed → In Progress
Vladimir Kuklin (vkuklin)
summary: - Can't create instance in nova az after shutdown primary controller after
- scale environment.
+ Neutron services are not restarted after config changes in common config
+ sections
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Igor Gajsin (igajsin) → Vladimir Kuklin (vkuklin)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Igor Gajsin (igajsin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/420047

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/420054

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/414930
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b4f804ea4cb96c3dd8c3d7c48d77464b87213bba
Submitter: Jenkins
Branch: master

commit b4f804ea4cb96c3dd8c3d7c48d77464b87213bba
Author: Igor Gajsin <email address hidden>
Date: Mon Dec 26 13:32:38 2016 +0300

    Restart neutron services on neutron config change

    There are many neutron tasks, e.g.: common-config and network-plugins-l2.
    Normally the common-config configures neutron.conf and then plugns-l2 starts
    the ovs-agent.

    But there is the wrong behaviour when new controllers are added to an
    environment:
      1. configure and deploy an env.
      2. common-config configures neutron.conf
      3. plugins-l2 starts ovs-agent.
      4. add new controllers to the env and redeploy it.
      5. common-config configures neutron.conf.
      6. plugins-l2 knows nothing about changes in the config.

    So ovs-agent isn't restarted and works with old configuration data.

    This commit makes neutron tasks check md5 hash of neutron.conf and
    restart them when config is changed.

    Change-Id: Ia26f4a5105407ce322a5185455093d69a938edfc
    Closes-bug: #1613724

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/420054
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0ecf4ef3282f4785c39c6d6f64dd88484264c89c
Submitter: Jenkins
Branch: stable/mitaka

commit 0ecf4ef3282f4785c39c6d6f64dd88484264c89c
Author: Igor Gajsin <email address hidden>
Date: Mon Dec 26 13:32:38 2016 +0300

    Restart neutron services on neutron config change

    There are many neutron tasks, e.g.: common-config and network-plugins-l2.
    Normally the common-config configures neutron.conf and then plugns-l2 starts
    the ovs-agent.

    But there is the wrong behaviour when new controllers are added to an
    environment:
      1. configure and deploy an env.
      2. common-config configures neutron.conf
      3. plugins-l2 starts ovs-agent.
      4. add new controllers to the env and redeploy it.
      5. common-config configures neutron.conf.
      6. plugins-l2 knows nothing about changes in the config.

    So ovs-agent isn't restarted and works with old configuration data.

    This commit makes neutron tasks check md5 hash of neutron.conf and
    restart them when config is changed.

    (cherry-picked from 5912ee7ea5715f17f2188c3eb9c814867c9af571)
    Change-Id: Ia26f4a5105407ce322a5185455093d69a938edfc
    Closes-bug: #1613724

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/420047
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=73599f5f44f6849c0d47639124a091824a21a543
Submitter: Jenkins
Branch: stable/newton

commit 73599f5f44f6849c0d47639124a091824a21a543
Author: Igor Gajsin <email address hidden>
Date: Mon Dec 26 13:32:38 2016 +0300

    Restart neutron services on neutron config change

    There are many neutron tasks, e.g.: common-config and network-plugins-l2.
    Normally the common-config configures neutron.conf and then plugns-l2 starts
    the ovs-agent.

    But there is the wrong behaviour when new controllers are added to an
    environment:
      1. configure and deploy an env.
      2. common-config configures neutron.conf
      3. plugins-l2 starts ovs-agent.
      4. add new controllers to the env and redeploy it.
      5. common-config configures neutron.conf.
      6. plugins-l2 knows nothing about changes in the config.

    So ovs-agent isn't restarted and works with old configuration data.

    This commit makes neutron tasks check md5 hash of neutron.conf and
    restart them when config is changed.

    (cherry-picked from 5912ee7ea5715f17f2188c3eb9c814867c9af571)
    Change-Id: Ia26f4a5105407ce322a5185455093d69a938edfc
    Closes-bug: #1613724

Ilya Bumarskov (ibumarskov)
tags: added: on-verification
Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

I can't verify it until developers fix the https://bugs.launchpad.net/fuel/+bug/1658952

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/425055

Revision history for this message
Igor Gajsin (igajsin) wrote :

there is another bug that reveals in the same deployment scenario: https://bugs.launchpad.net/fuel/+bug/1658952

When it will be fixed the current bug could be tested for moving to the 'fix-released' state.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/425648

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/425649

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/425055
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=418b09fc1f9b43f78ff7b40ae31e70784a656524
Submitter: Jenkins
Branch: master

commit 418b09fc1f9b43f78ff7b40ae31e70784a656524
Author: Michael Polenchuk <email address hidden>
Date: Wed Jan 25 12:27:00 2017 +0400

    Adjust scope of neutron agent notify

    Shift notify under if condition to meet service resource relationship.
    Also function has been modified to exclude creation
    files with :: symbols.

    Change-Id: I2cbf9c4b2c7a9796b7b7ab53733cc489b03e533b
    Related-Bug: #1613724
    Closes-bug: #1658952

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc1

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/425649
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=15be3f5f130e81b31b6132ab2835ce7d4906cc18
Submitter: Jenkins
Branch: stable/newton

commit 15be3f5f130e81b31b6132ab2835ce7d4906cc18
Author: Michael Polenchuk <email address hidden>
Date: Wed Jan 25 12:27:00 2017 +0400

    Adjust scope of neutron agent notify

    Shift notify under if condition to meet service resource relationship.
    Also function has been modified to exclude creation
    files with :: symbols.

    Change-Id: I2cbf9c4b2c7a9796b7b7ab53733cc489b03e533b
    Related-Bug: #1613724
    Closes-bug: #1658952

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/425648
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=94a6a10423067051ff3d509774a66aa0ce7303f8
Submitter: Jenkins
Branch: stable/mitaka

commit 94a6a10423067051ff3d509774a66aa0ce7303f8
Author: Michael Polenchuk <email address hidden>
Date: Wed Jan 25 12:27:00 2017 +0400

    Adjust scope of neutron agent notify

    Shift notify under if condition to meet service resource relationship.
    Also function has been modified to exclude creation
    files with :: symbols.

    Change-Id: I2cbf9c4b2c7a9796b7b7ab53733cc489b03e533b
    Related-Bug: #1613724
    Closes-bug: #1658952

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.