DVR: Inter Tenant Traffic between two networks and connected through a shared network not reachable with DVR routers

Fix proposed to branch: master
Review: https://review.openstack.org/547696

Reviewed: https://review.openstack.org/547696
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee
Submitter: Zuul
Branch: master

commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/554644

Reviewed: https://review.openstack.org/554644
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=02d31ffb8abb157dff02fc977241ebdbf8ead89c
Submitter: Zuul
Branch: stable/queens

commit 02d31ffb8abb157dff02fc977241ebdbf8ead89c
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/558585

This bug was fixed in the package neutron - 2:12.0.1-0ubuntu1

---------------
neutron (2:12.0.1-0ubuntu1) bionic; urgency=medium

  * d/p/dvr-inter-tenant-traffic.patch: Cherry-picked from upstream
    stable/queens branch (LP: #1751396).
  * New stable point release for OpenStack Queens (LP: #1765138).

 -- Corey Bryant <email address hidden> Wed, 18 Apr 2018 12:07:48 -0400

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

Hello Swaminathan, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This issue was fixed in the openstack/neutron 12.0.2 release.

12.0.2 is available in the Queens cloud archive.

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/578112

Reviewed: https://review.openstack.org/558585
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6956821764e81c61f4c3620a2a6753394b5bc69a
Submitter: Zuul
Branch: stable/pike

commit 6956821764e81c61f4c3620a2a6753394b5bc69a
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

Reviewed: https://review.openstack.org/578112
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f65e7ba05d0502bdc7163a6221ed8138ac183055
Submitter: Zuul
Branch: stable/ocata

commit f65e7ba05d0502bdc7163a6221ed8138ac183055
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

Hi, this appears to have broken DVR functionality for a simple shared tenant network. The DVR flows translating the DVR macs are missing on the physical bridges now for a VLAN based network. This tenant network is shared, and only attached to 1 router.

As you can see port_shared_only is set to True here because its shared and not an external network.

2018-07-25 15:53:02.386 673 DEBUG neutron.api.rpc.handlers.dvr_rpc [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] neutron.api.rpc.handlers.dvr_rpc.DVRServerRpcApi method get_network_info_for_id called with arguments (<neutron_lib.context.ContextBase object at 0x7f67cb87fd50>, u'3f6ec232-7649-4639-b828-c3af9960481b') {} wrapper /opt/pf9/pf9-neutron/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-07-25 15:53:02.779 673 INFO neutron.common.rpc [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] PF9_EVENT: RPC_get_network_info_for_id elapsed: 0.39 seconds
2018-07-25 15:53:02.781 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: port_net_info = [{u'provider:physical_network': u'dogfood-internal', u'ipv6_address_scope': None, u'dns_domain': u'', u'revision_number': 7, u'port_security_enabled': True, u'mtu': 1500, u'id': u'3f6ec232-7649-4639-b828-c3af9960481b', u'router:external': False, u'availability_zone_hints': [], u'availability_zones': [u'nova'], u'ipv4_address_scope': None, u'shared': True, u'project_id': u'f175f441ebbb4c2b8fedf6469d6415fc', u'status': u'ACTIVE', u'subnets': [u'3707b250-b6f5-4701-9b17-01a8f288c17a'], u'description': None, u'tags': [], u'provider:segmentation_id': 795, u'name': u'df-tenant-795-dont-delete', u'admin_state_up': True, u'tenant_id': u'f175f441ebbb4c2b8fedf6469d6415fc', u'provider:network_type': u'vlan', u'vlan_transparent': None}]
2018-07-25 15:53:02.782 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: net_shared_only = True
2018-07-25 15:53:02.784 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: Not applying DVR rules to tunnel bridge because 3f6ec232-7649-4639-b828-c3af9960481b is a shared network
2018-07-25 15:53:02.785 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: vlan = 1, port.vif_mac = fa:16:3e:42:a2:ec, dvr_mac = fa:16:3f:1a:bf:de

I think we need more checks here - seems like it was intended only for a shared network that acts as a link between 2 routers, but doesnt attach any compute ports?

Logged https://bugs.launchpad.net/neutron/+bug/1783654 to address the missing DVR flows on a shared VLAN tenant network

Related fix proposed to branch: master
Review: https://review.openstack.org/595496

Reviewed: https://review.openstack.org/595496
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f
Submitter: Zuul
Branch: master

commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/596402

Reviewed: https://review.openstack.org/596402
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7
Submitter: Zuul
Branch: stable/queens

commit 2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)

Moving status back to New for Ubuntu due to reverted patch.

s/New/Triaged

Marking Artful as invalid since it is EOL.

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/607345

This does not affect Ubuntu Pike because neutron 11.0.5 does not include the original patches that are being reverted.

This bug was fixed in the package neutron - 2:13.0.1-0ubuntu1

---------------
neutron (2:13.0.1-0ubuntu1) cosmic; urgency=medium

  * New stable point release for OpenStack Rocky.
  * d/p/revert-dvr-add-error-handling.patch: Cherry-picked from upstream to
    revert DVR regressions (LP: #1751396)
  * d/p/revert-dvr-inter-tenant.patch: Cherry-picked from upstream to revert
    DVR regression (LP: #1783654).

 -- Corey Bryant <email address hidden> Tue, 02 Oct 2018 17:18:19 -0400

Hello Swaminathan, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.0.4-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Swaminathan, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package neutron - 2:13.0.1-0ubuntu1~cloud0
---------------

 neutron (2:13.0.1-0ubuntu1~cloud0) bionic-rocky; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 neutron (2:13.0.1-0ubuntu1) cosmic; urgency=medium
 .
   * New stable point release for OpenStack Rocky.
   * d/p/revert-dvr-add-error-handling.patch: Cherry-picked from upstream to
     revert DVR regressions (LP: #1751396)
   * d/p/revert-dvr-inter-tenant.patch: Cherry-picked from upstream to revert
     DVR regression (LP: #1783654).

Reviewed: https://review.openstack.org/607345
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d1a0d630d22180cbe602d56cfb308732add453de
Submitter: Zuul
Branch: stable/rocky

commit d1a0d630d22180cbe602d56cfb308732add453de
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)

Regression testing successful for bionic-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1318.6413 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 494.8999 sec.

Regression testing successful for queens-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1000.6584 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 465.0920 sec.

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/611396

I don't think the regression testing is sufficient for this SRU verification, please follow the test case in the description to verify that this is fixed.

Reviewed: https://review.openstack.org/611396
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=eb7bb07184bfcc2f64cbf53f9d69891cd7261eea
Submitter: Zuul
Branch: stable/pike

commit eb7bb07184bfcc2f64cbf53f9d69891cd7261eea
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)
    (cherry picked from commit 2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7)

This issue was fixed in the openstack/neutron 11.0.6 release.

This SRU is no longer needed as we're uploading neutron 12.0.5 to bionic (queens) which includes this fix.

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

This bug has been fixed in the neutron 12.0.5 stable point release via https://bugs.launchpad.net/cloud-archive/+bug/1795424. neutron 12.0.5 was tested with a tempest smoke test run against a juju-deployed openstack as described in https://wiki.ubuntu.com/OpenStackUpdates.
Please see test results at: https://bugs.launchpad.net/cloud-archive/+bug/1795424/comments/17

Thanks,
Corey

This issue was fixed in the openstack/neutron ocata-eol release.