Networking is not working with OVS bridges

Bug #1682835 reported by Oleksiy Molchanov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Sergey Vasilenko
Nominated for Ocata by Oleksiy Molchanov
Mitaka
Fix Committed
Critical
Sergey Vasilenko
Newton
In Progress
Critical
Unassigned

Bug Description

Network scheme

{"endpoints"=>
  {"br-ex"=>
    {"IP"=>["10.109.18.6/24"],
     "gateway"=>"10.109.18.1",
     "vendor_specific"=>{"provider_gateway"=>"10.109.18.1"}},
   "br-floating"=>{"IP"=>"none"},
   "br-fw-admin"=>
    {"IP"=>["10.109.15.3/24"],
     "vendor_specific"=>{"provider_gateway"=>"10.109.15.1"}},
   "br-mgmt"=>{"IP"=>["192.168.0.5/24"]},
   "br-prv"=>{"IP"=>"none"},
   "br-storage"=>{"IP"=>["192.168.1.3/24"]}},
 "interfaces"=>
  {"enp0s3"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:03.0", "driver"=>"e1000"}},
   "enp0s4"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:04.0", "driver"=>"e1000"}},
   "enp0s5"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:05.0", "driver"=>"e1000"}},
   "enp0s6"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:06.0", "driver"=>"e1000"}},
   "enp0s7"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:07.0", "driver"=>"e1000"}}},
 "provider"=>"lnx",
 "roles"=>
  {"admin/pxe"=>"br-fw-admin",
   "aodh/api"=>"br-mgmt",
   "ceilometer/api"=>"br-mgmt",
   "ceph/public"=>"br-storage",
   "ceph/radosgw"=>"br-ex",
   "ceph/replication"=>"br-storage",
   "cinder/api"=>"br-mgmt",
   "cinder/iscsi"=>"br-storage",
   "ex"=>"br-ex",
   "fw-admin"=>"br-fw-admin",
   "glance/api"=>"br-mgmt",
   "glance/glare"=>"br-mgmt",
   "heat/api"=>"br-mgmt",
   "horizon"=>"br-mgmt",
   "ironic/api"=>"br-mgmt",
   "keystone/api"=>"br-mgmt",
   "management"=>"br-mgmt",
   "mgmt/corosync"=>"br-mgmt",
   "mgmt/database"=>"br-mgmt",
   "mgmt/memcache"=>"br-mgmt",
   "mgmt/messaging"=>"br-mgmt",
   "mgmt/vip"=>"br-mgmt",
   "mongo/db"=>"br-mgmt",
   "murano/api"=>"br-mgmt",
   "murano/cfapi"=>"br-mgmt",
   "neutron/api"=>"br-mgmt",
   "neutron/floating"=>"br-floating",
   "neutron/private"=>"br-prv",
   "nova/api"=>"br-mgmt",
   "nova/migration"=>"br-mgmt",
   "public/vip"=>"br-ex",
   "sahara/api"=>"br-mgmt",
   "storage"=>"br-storage",
   "swift/api"=>"br-mgmt",
   "swift/replication"=>"br-storage"},
 "transformations"=>
  [{"action"=>"add-br", "name"=>"br-fw-admin", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-mgmt", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-storage", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-ex", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-floating", "provider"=>"ovs"},
   {"action"=>"add-patch",
    "bridges"=>["br-floating", "br-ex"],
    "mtu"=>65000,
    "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-prv", "provider"=>"ovs"},
   {"action"=>"add-patch",
    "bridges"=>["br-prv", "br-fw-admin"],
    "mtu"=>65000,
    "provider"=>"ovs"},
   {"action"=>"add-port", "bridge"=>"br-fw-admin", "name"=>"enp0s3"},
   {"action"=>"add-port", "bridge"=>"br-mgmt", "name"=>"enp0s3.101"},
   {"action"=>"add-port", "bridge"=>"br-storage", "name"=>"enp0s3.102"},
   {"action"=>"add-port", "bridge"=>"br-ex", "name"=>"enp0s4"}],
 "version"=>"1.1"}

ovs-vsctl show

root@node-6:~# ovs-vsctl show
bf8b8f19-0176-4fc0-8211-a8f2d9557c24
    Bridge br-prv
        Port br-prv
            Interface br-prv
                type: internal
        Port "p_eeee51a2-1"
            Interface "p_eeee51a2-1"
                type: patch
                options: {peer="p_eeee51a2-0"}
    Bridge br-mgmt
        Port "enp0s3.101"
            tag: 101
            Interface "enp0s3.101"
                type: internal
        Port v_management
            Interface v_management
        Port br-mgmt
            Interface br-mgmt
                type: internal
        Port v_vrouter
            Interface v_vrouter
    Bridge br-fw-admin
        Port br-fw-admin
            Interface br-fw-admin
                type: internal
        Port "enp0s3"
            Interface "enp0s3"
        Port "p_eeee51a2-0"
            Interface "p_eeee51a2-0"
                type: patch
                options: {peer="p_eeee51a2-1"}
    Bridge br-floating
        Port "p_ff798dba-1"
            Interface "p_ff798dba-1"
                type: patch
                options: {peer="p_ff798dba-0"}
        Port br-floating
            Interface br-floating
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port v_vrouter_pub
            Interface v_vrouter_pub
        Port "enp0s4"
            Interface "enp0s4"
        Port v_public
            Interface v_public
        Port "p_ff798dba-0"
            Interface "p_ff798dba-0"
                type: patch
                options: {peer="p_ff798dba-1"}
    Bridge br-storage
        Port br-storage
            Interface br-storage
                type: internal
        Port "enp0s3.102"
            tag: 102
            Interface "enp0s3.102"
                type: internal
    ovs_version: "2.6.1"

https://drive.google.com/open?id=0ByBwUBq5EbLEUjcxVlZ0R2JnQW8

description: updated
Changed in fuel:
milestone: none → 10.x-updates
milestone: 10.x-updates → 11.x-updates
tags: added: sla2
Anton Matveev (amatveev)
tags: added: sla1
removed: sla2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/458851

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/458879

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/458880

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/458881

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/newton)

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/458879

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/458881

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/ocata)

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/458880

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/458881
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d096bfe121cba47ae27f5f803d3d65b239910c1e
Submitter: Jenkins
Branch: master

commit d096bfe121cba47ae27f5f803d3d65b239910c1e
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/ocata)

Reviewed: https://review.openstack.org/458880
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f7466d00c406e0fac03d0e012b21107933574bdd
Submitter: Jenkins
Branch: stable/ocata

commit f7466d00c406e0fac03d0e012b21107933574bdd
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/458851
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e6907520ae67ad35105ecbf609161c4bf0901f10
Submitter: Jenkins
Branch: stable/mitaka

commit e6907520ae67ad35105ecbf609161c4bf0901f10
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/465591

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/465601

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/465602

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/465591
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=22e7dcb5396b503636c0a2f4ba14f100fd2850d1
Submitter: Jenkins
Branch: stable/mitaka

commit 22e7dcb5396b503636c0a2f4ba14f100fd2850d1
Author: Sergey Vasilenko <email address hidden>
Date: Wed May 17 17:02:03 2017 +0300

    Add more strict checking for veth pair into OVS bridge.

    Some time VETH pair may be self-disassempled or broken by 3d party action,
    but OVS doesn't know about it and does not remove port with broken interface.
    Such ports we can detect by 'ifindex' property. For really existing interfaces
    'ifindex' property should be equal to real ifindex. For broken interfaces
    'ifindex' mabe have zero or empty value.

    This patchset add checking for it.

    Also:
     * improove logging
     * monitor() function was refactored, because previous logic had no full
       coverage of failure cases.
     * set sysctl -w net.ipv4.ip_nonlocal_bind=1 into network namespace while creating

    Change-Id: I05062f84895e020b10a94ac09f8573598434b0e6
    Closes-Bug: #1682835

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/465601
Reason: This repo is retired now, no further work will get merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/ocata)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/465602
Reason: This repo is retired now, no further work will get merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/newton)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/458879
Reason: This repo is retired now, no further work will get merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library ocata-eol

This issue was fixed in the openstack/fuel-library ocata-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.