networking-cisco

EMC-POC: Nexus plugin: replay fails intermittently

Bug #1466501 reported by Danny Choi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-cisco
New
Undecided
Unassigned

Bug Description

Replay code parameters:
   - switch_heartbeat_time = 30
   - switch_replay_count = 3

Reboot Nexus switch to activate the config replay.

During the replay, it would fail. Then 30 seconds later, it would replay the config successfully.

This is an intermittent problem. 3 out of 10 reboots.

2015-06-17 15:57:27.907 7442 INFO neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Re-established connection to switch ip 10.86.1.128
2015-06-17 15:57:27.907 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Replaying config for switch ip 10.86.1.128 replay_config /usr/lib/python2.7/site-package
s/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py:70
2015-06-17 15:57:27.907 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [-] get_nve_switch_bindings() called get_nve_switch_bindings /usr/lib/python2.7/site-packages/neu
tron/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:216
2015-06-17 15:57:27.909 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [-] get_nexusport_switch_bindings() called get_nexusport_switch_bindings /usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:46
2015-06-17 15:57:27.912 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Nexus: create & trunk vlan neutron-502 _configure_port_binding /usr/lib/python2.7/site-p
ackages/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py:414
2015-06-17 15:57:27.912 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver: create_vlan /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drive
rs/cisco/nexus/nexus_network_driver.py:247
2015-06-17 15:57:27.912 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver edit config:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
            <vlan>
              <vlan-id-create-delete>
                <__XML__PARAM_value>502</__XML__PARAM_value>
                <__XML__MODE_vlan>
                  <name>
                    <vlan-name>neutron-502</vlan-name>
                  </name>
                </__XML__MODE_vlan>
              </vlan-id-create-delete>
            </vlan>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
 _edit_config /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:102
2015-06-17 15:57:28.026 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver edit config:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
            <vlan>
              <vlan-id-create-delete>
                <__XML__PARAM_value>502</__XML__PARAM_value>
                <__XML__MODE_vlan>
                  <state>
                    <vstate>active</vstate>
                  </state>
                </__XML__MODE_vlan>
              </vlan-id-create-delete>
            </vlan>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
 _edit_config /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:102
2015-06-17 15:57:28.140 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver edit config:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
            <vlan>
              <vlan-id-create-delete>
                <__XML__PARAM_value>502</__XML__PARAM_value>
                <__XML__MODE_vlan>
                  <no>
                    <shutdown/>
                  </no>
                </__XML__MODE_vlan>
              </vlan-id-create-delete>
            </vlan>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
 _edit_config /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:102
2015-06-17 15:57:28.254 7442 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver created VLAN: 502 create_and_trunk_vlan /usr/lib/python2.7/site-packages
/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:331
2015-06-17 15:57:28.481 7442 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Nexus Driver cisco_nexus failed in replay _configure_port_binding
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus Traceback (most recent call last):
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 509, in configure_switch_entries
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus port.vni)
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 417, in _configure_port_binding
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus vni)
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 334, in create_and_trunk_vlan
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus nexus_port)
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 297, in enable_vlan_on_trunk_int
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus intf_type, interface)
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 188, in get_interface_switch_trunk_allowed
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus response = self._get_config(nexus_host, confstr)
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 78, in _get_config
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus exc=e)
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus NexusConfigFailed: Failed to configure Nexus switch: 10.86.1.128 XML:
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <cmd>show running-config interface ethernet 1/22</cmd>
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus . Reason: Syntax error while parsing 'show running-config interface ethernet 1/22'
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus .
2015-06-17 15:57:28.481 7442 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus
2015-06-17 15:57:28.482 7442 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Failed to configure port binding for switch 10.86.1.128, vlan 502 vni 0, port ethernet:1/22, reason Failed to configure Nexus switch: 10.86.1.128 XML:
    <cmd>show running-config interface ethernet 1/22</cmd>
. Reason: Syntax error while parsing 'show running-config interface ethernet 1/22'
.

Tags: nexus cisco e-rel
Revision history for this message
Danny Choi (dannchoi) wrote :

This issue is seen with the following configuration:

- Tenant admin has network 192.168.1.0/24 defined.
     - One VM using this network launched at Compute-1, with port-channel.
     - One VM using the same network launched at Compute-2, with trunk interface.
- Tenant tenant1 has same network 192.168.1.0/24 defined.
     - One VM using this network launched at Compute-1, with port-channel.
     - One VM using the same network launched at Compute-2, with trunk interface.

Revision history for this message
Danny Choi (dannchoi) wrote :
Download full text (15.4 KiB)

When it fails, it always happen at the first VLAN configuration at the trunk interface. Then it waits 30 seconds (the heartbeat timeout) and replay the config again from the beginning.

2015-06-25 16:43:53.178 13954 INFO neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Re-established connection to switch ip 10.86.1.128
2015-06-25 16:43:53.178 13954 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Replaying config for switch ip 10.86.1.128 replay_config /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py:77
2015-06-25 16:43:53.178 13954 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [-] get_nve_switch_bindings() called get_nve_switch_bindings /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:216
2015-06-25 16:43:53.180 13954 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [-] get_nexusport_switch_bindings() called get_nexusport_switch_bindings /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:46
2015-06-25 16:43:53.187 13954 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Nexus: create & trunk vlan neutron-502 _configure_port_binding /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py:431
2015-06-25 16:43:53.187 13954 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver: create_vlan /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:247
2015-06-25 16:43:53.187 13954 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver edit config:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
            <vlan>
              <vlan-id-create-delete>
                <__XML__PARAM_value>502</__XML__PARAM_value>
                <__XML__MODE_vlan>
                  <name>
                    <vlan-name>neutron-502</vlan-name>
                  </name>
                </__XML__MODE_vlan>
              </vlan-id-create-delete>
            </vlan>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
 _edit_config /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:102
2015-06-25 16:43:53.301 13954 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [-] NexusDriver edit config:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
            <vlan>
              <vlan-id-create-delete>
                <__XML__PARAM_value>502</__XML__PARAM_value>
                <__XML__MODE_vlan>
                  <state>
                    <vstate>active</vstate>
                  </state>
                </__XML__MODE_vlan>
              </vlan-id-create-delete>
            </vlan>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
 _edit_config /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:102
2015-06-25 16:43:53.414 13954 DEBUG neutron.p...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.