DVR issues with supporting multiple subnets per network on DVR routers - PortbindingContext does not have the status.

This is exposed when the IPv6 tempest test "test_multi_prefix_dhcpv6_stateless" and "test_multi_prefix_slaac" is run with multiple ipv6 subnets.

On further investigation it seems that even when a "single" IPv4 router-interface port is updated with a new IP address then the L2pop - update-port-postcommit fails.

See the failure logs below.

http://pastebin.com/660JuPAm

For some reason when a port update happens the PortContext is provided for DVR Routed ports instead of the DVRPortbinding context.

This seems critical since it's blocking the DVR job from running. Swami, are you going to fix this in Neutron itself? I think Yamamoto can possibly fix this if you don't have time, I've added him to the bug.

Hi Kyle, yes I am currently investigating the problem and also working with Rob Kukura to understand the port-binding issues.

I can work with Yamamoto if he has some bandwith to help on this issue.

First patch.
https://review.openstack.org/#/c/196908/

With current review comments, I tried to refactor the patch and I am seeing another issue, where the "DVR Port Binding" is not returning "binding-list" when the first time the update_port is called.
Later the binding-list is visible, so there is no way to make this code generic.

See the logs in the paste bin.

Search for "Binding List" and this will show in the beginning the list is empty and later gets populated.

http://paste.openstack.org/show/353242/

Swaminathan, i couldn't find "Binding List". maybe paste truncated the log?

Yamamoto , yes I think the paste is truncated.

The log is too big for some reason I could not upload all the logs.

But what is happening here is, sync_routers is were the "dvr binding is introduced".
In this case, there is a timing issue where before the port is binding, the update_port is called while add_router_interface.

In this case, the first time when the "dvr port binding" is fetched for the port, it returns an empty list.

So when I use the same list for processing all other function calls, the list returns "out-of-bound" errors.

You can look at the "logstash" for the patch with the current changes in the code.

sure, i found it.
http://logs.openstack.org/08/196908/8/check/check-tempest-dsvm-neutron-dvr/a4b28ce/logs/screen-q-svc.txt.gz?#_2015-07-07_01_47_41_677

The patch is up for review. It would be great if anyone can review it.

https://review.openstack.org/#/c/196908/

Reviewed: https://review.openstack.org/196908
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9eed4167b63f0c0ad6c7ceeedc644e511c8dac75
Submitter: Jenkins
Branch: master

commit 9eed4167b63f0c0ad6c7ceeedc644e511c8dac75
Author: Swaminathan Vasudevan <email address hidden>
Date: Mon Jun 29 16:19:49 2015 -0700

    Fix update_port_postcommit and port not found with DVR

    Updating DVR Router interface ports was throwing
    errors in the l2pop mechanism drivers function
    update_port_postcommit.

    PortContext's portbinding information does not show
    the status of the ports. For DVR Router interface
    ports the DVRPortbinding table contains the status
    information for the ports.

    In the case of the update_port method, there was
    no code related to DVR that retreives the port
    binding information from the DVRPortBinding table.

    This was working before, since in the driver_context,
    the PortContext was just returning the port status for
    all router interfaces.

    With the recent refactor to the driver_context, this
    behavior changed and the PortContext was returning the
    _binding.status for the DVR router interface ports and
    the _port.status for the non DVR ports.

    When the update_port function calls update_port_postcommit
    with PortContext for DVR router interface ports, l2pop
    was throwing an error saying that Portbinding does not
    have the attribute 'status'.

    This was causing addition of any second subnet to the
    same network with respect to IPv6 to fail.
    Because in the case of IPv6, when you add additional
    subnets to the existing network, it just updates the port
    with the IPv6 prefix instead of creating additional port.

    In the case of IPv4 still we could see that there are
    two different ports created for each subnet we try to
    add.

    This patch fixes the above issue in l2pop and allows the
    DVR router interface ports to be successfull.

    Also the _find_ipv6_router_port_by_network was returning
    all the ports for DVR including the DVR CSNAT internal
    ports which are not part of the router interface ports.

    This patch also fixes this problem by returning false,
    when it finds a DVR SNAT port.

    Closes-Bug: #1465434

    Change-Id: Id243a4b3f30071226411ace6d12550fc099901cc

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/207903

Reviewed: https://review.openstack.org/207903
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9badcd249dab2d3330f0cd608496b59c9f44499a
Submitter: Jenkins
Branch: feature/pecan

commit 991ea00e6c115343eabecc62e86072175823f81f
Author: Moshe Levi <email address hidden>
Date: Thu Jul 30 12:04:15 2015 +0300

    SR-IOV: Fix SR-IOV agent to run ip link commands as root

    Commit https://review.openstack.org/#/c/155523/
    remove the remaining root_helper args, but didn't
    update the SR-IOV agent to execute them as root.

    This patch updates the agent to execute ip link commands
    as root and pass options argument as a list in the
    self._as_root method.

    Closes-Bug: #1479694
    Change-Id: I53cafd61845a69fae3a759fb7526950d655ffa20

commit 5b3bacedf6c014815bef03c2e821b5eb8ef92dcb
Author: Saksham Varma <email address hidden>
Date: Thu Jul 23 22:46:23 2015 -0700

    Moved out cisco n1kv mech driver and db models

    Moving out Cisco N1Kv database models and the mech driver
    to the openstack/networking-cisco repo as part of the second
    phase vendor-core decomposition

    Partial-Bug: #1479123
    Partial-implements: blueprint core-vendor-decomposition

    Change-Id: I65a704b238d8cbe2951a6912fa4f1e8518c6f412

commit 65ac8cd0a80548e115b8ec1d4cfd47a94422fddf
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Jul 29 20:44:01 2015 +0000

    Updated from global requirements

    Change-Id: I3a884a73d51df0f93b5cc88b3e3250d81bb1f455

commit f0e8356d04d04600c072a1c0f0bdf274ff19ab8c
Author: sridhargaddam <email address hidden>
Date: Mon Jul 27 03:46:48 2015 +0000

    Update dhcp agent cache for network:dhcp ports

    When a network with a dhcp_enabled subnet is scheduled on a dhcp
    agent, dhcp driver creates the network:dhcp port for the subnet.
    However, the port info is not updated in dhcp agents internal cache.
    Subsequently if the user deletes the network:dhcp port, the port is
    properly deleted on the server side (i.e., in the database) and when
    the port_delete_end notification is sent to the dhcp agent, it simply
    ignores it as the port entry would be missing in the cache. This patch
    fixes this issue by updating the dhcp agents cache when dhcp driver
    creates the network:dhcp port for the subnets.

    Closes-Bug: #1478426
    Change-Id: I69f5834dd964a4320c606c4e0aa2cdba70416943

commit cb60d0bb4e0cc0cba68f59fdf5f4e89d6ec52950
Author: changzhi <email address hidden>
Date: Thu Jul 16 10:14:16 2015 +0800

    Keep dns nameserver order consistency

    Currently, there is no dns servers prioritization for subnets
    for Neutron.

    Generally speaking, it is useful to keep the order of dns
    nameservers consistent. Add a new column named 'order' in table
    'dnsnameservers' and add nameserver into DB one by one.

    Closes-Bug: #1218629
    Implements: blueprint keep-dns-nameserver-orderconsistency
    Change-Id: Id937aea411397d39370368a4eb45be26c4eefa9e

commit b39e1469e824bc8bc79e1ecafa98825a94811c0b
Author: Salvatore Orlando <email address hidden>
Date: Tue Jun 23 04:54:2...

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/241616

Change abandoned by Erik Colnick (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/241616
Reason: Not convinced this was actually an issue in kilo. Abandoning.

Change abandoned by Erik Colnick (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/241616
Reason: This particular change is not backwards compatible owing to modification in the abstract interface for PortContext, which will break other existing implementations of the interface.