openvswitch systemd unit file ordering wrong

Looks we've been through this before with upstart:
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1084028

I have a the same problem.
In systemd, there are two files openvswitch-nonetwork.service openvswitch-switch.service which are supposed to fix this problem

I am testing OpenStack with VXLAN and require

sudo ovs-vsctl add-br br-eth2
sudo ovs-vsctl set port br-eth2 tag=2001
sudo ovs-vsctl add-port br-eth2 vxlan1
sudo ovs-vsctl set interface vxlan1 type=vxlan options:remote_ip=192.168.102.205
sudo ovs-vsctl add-port br-eth2 vxlan2
sudo ovs-vsctl set interface vxlan2 type=vxlan options:remote_ip=192.168.102.234
sudo ovs-vsctl add-br br-ex
sudo ovs-vsctl add-port br-ex eth0

then bring up the network from /etc/network/interfaces with

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
    up ifconfig $IFACE 0.0.0.0 up
    up ip link set $IFACE promisc on
    down ip link set $IFACE promisc off
    down ifconfig $IFACE down

auto br-ex
iface br-ex inet static
    address 192.168.102.206
    netmask 255.255.255.0
    gateway 192.168.102.36
    dns-nameservers 192.168.102.10 192.168.102.50
    up ip link set $IFACE promisc on
    down ip link set $IFACE promisc off

auto br-eth2
iface br-eth2 inet static
    address 10.1.0.11
    netmask 255.255.255.0
    up ip link set $IFACE promisc on
    down ip link set $IFACE promisc off
    mtu 1446

from "sudo systemctl list-unit-files" i get

openvswitch-nonetwork.service static
openvswitch-switch.service enabled

so I assume they are running

Status changed to 'Confirmed' because the bug affects multiple users.

Sorry, lost track of the bug as it fell into 100 papercuts...

if it helps I solved my ordering problem by modifying the
    /lib/systemd/sysytem/openvswitch-nonetwork.service
as follows

    [Unit]
    Description=Open vSwitch Internal Unit
    PartOf=openvswitch-switch.service

    # Without this all sorts of looping dependencies occur doh!
    DefaultDependencies=no

    #precedants pulled from isup@ service requirements
    After=apparmor.service local-fs.target systemd-tmpfiles-setup.service

    #subsequent to this service we need the network to start
    Wants=network-pre.target openvswitch-switch.service
    Before=network-pre.target openvswitch-switch.service

    [Service]
    Type=oneshot
    RemainAfterExit=yes
    EnvironmentFile=-/etc/default/openvswitch-switch
    ExecStart=/usr/share/openvswitch/scripts/ovs-ctl start \
          --system-id=random $OPTIONS
    ExecStop=/usr/share/openvswitch/scripts/ovs-ctl stop

This pulled up the services and allowed my configuration to work

openvswitch-nonetwork.service
● ├─openvswitch-switch.service
● └─network-pre.target
● ├─<email address hidden>
● ├─<email address hidden>
● ├─<email address hidden>
● ├─<email address hidden>
● ├─networking.service
● └─network.target
● ├─mysql.service
● ├─openvswitch-switch.service
● ├─rabbitmq-server.service
● ├─rc-local.service
● ├─ssh.service
● └─network-online.target
● ├─apache2.service
● ├─dns-clean.service
● └─kerneloops.service

(the dots are green :) )

There is lots of noise on the net about chickens and eggs, so I do not know if it solves some other case
The thing that took me so long to solve it (besides unfamiliarity of systemd) was the AUTOMATIC inclusion of dependencies, who on earth dreamt that up?

I was one of the people that helped get this fixed in Trusty and now, in a major case of deja vu, hit this same issue now with Xenial. It looks like Mark Dunn's fix above matches http://www.opencloudblog.com/?p=240 and I can confirm it fixed the problem on Xenial.

This bug was fixed in the package openvswitch - 2.6.1-0ubuntu1

---------------
openvswitch (2.6.1-0ubuntu1) zesty; urgency=medium

  * New upstream point release (LP: #1641956).
  * d/openvswitch-switch.openvswitch-nonetwork.service: Update Unit
    definition to ensure that openvswitch starts prior to configuration
    of any network interfaces (LP: #1448254). Thanks to Mark Dunn for
    this fix.

 -- James Page <email address hidden> Tue, 15 Nov 2016 14:15:06 +0000

Thanks for looking into this. Any chance of getting the fix backported to Xenial?

@Simon

Working on the backports

OK so I'm struggling a bit to produce a nice /e/n/i that reproduces the issue on a minimal/fresh install to help with SRU verification - any of the participants in this bug report have anything we can use for that?

James,

This bug seems to be weird to reproduce. My physical Xenial server worked for months without the proposed patch then last week it suddenly stopped receiving DHCP lease 2 days ago. After much troubleshooting I remembered I was on this bug report, used the proposed patch, and it works like a charm. Without it my server would never try to get a DHCP lease on startup but ifup worked fine once the machine was up.

I tried to recreate using a VM but the VM has no such issue; it grabs a DHCP lease without any problems. I don't know if it's some weird race condition or what.

The bug is still there on ubuntu 16.04
In the journalctl logs openvswitch complains about connecting to its db while trying to raise NIC defined in the /etc/network/interfaces or /etc/network/interfaces.d/* and it slow down the process and create errors.

Steps to reproduce:
-Create OVS interface in the /etc/network/interfaces
E.g.:
allow-ovs main_sw
auto main_sw
iface main_sw inet manual
        ovs_type OVSBridge
-Reboot your computer
-Look at journalctl -b0 -r.

The new patched systemd unit nonetwork for zesty works well on xenial.

I'm also affected by this on ubuntu 16.04. The patched systemd-unit-file does not work for me.

With the original unit-file the systems starts fine and all bridges and physical interfaces are configured by openvswitch, but the systems hangs during shutdown.

With patched unit-file the physical interfaces do not get attached to the openvswitch-bridges, but the system shutdown is working fine. Looks like ifup@ is turning the interfaces on before openvswitch consumes them.

I tried playing with the unit-dependencies, but so far without luck.

Is there already a bug report for xenial regarding this issue? I think it is a good idea to open bug against xenial, don't you think?

sry, i was wrong. I had some issues in my /etc/network/interfaces-file which caused the patched unit-file to malfunction. To be more precise: i did put a line "auto <ifname>" above any device enslaved by openvswitch-bridges or -bonds later on. That forced ifup@ to turn the devices on before they were enslaved by ovs...

 The patched unit-file is working fine now.

Can also confirm that Marks patch worked.
This was two years ago, why haven't this been fixed yet?

I might as well just add some facts and my logs.

First of all, I have one Xenial node where open vswitch worked, guessing that that time open vswitch daemon started before trying to setup network.
And then on other machines it doesn't work cause the daemon isn't started before trying to setup networking.

Here are the logs from a broken system.
```
Sep 12 16:11:07 vm10 systemd[1]: Reached target Network.
Sep 12 16:11:07 vm10 systemd[1]: networking.service: Failed with result 'exit-code'.
Sep 12 16:11:07 vm10 systemd[1]: networking.service: Unit entered failed state.
Sep 12 16:11:07 vm10 systemd[1]: Failed to start Raise network interfaces.
Sep 12 16:11:07 vm10 systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Sep 12 16:11:07 vm10 ntpdate[3027]: no servers can be used, exiting
Sep 12 16:11:07 vm10 ifup[2858]: Failed to bring up br0.
Sep 12 16:11:07 vm10 ifup[2858]: Cannot find device "br0"
Sep 12 16:11:07 vm10 ifup[2858]: Failed to bring up mgm.
Sep 12 16:11:07 vm10 ifup[2858]: Cannot find device "mgm"
Sep 12 16:11:07 vm10 ifup[2858]: mgm: ERROR while getting interface flags: No such device
Sep 12 16:11:07 vm10 ifup[2858]: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Sep 12 16:11:07 vm10 ovs-vsctl[2991]: ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Sep 12 16:11:07 vm10 ovs-vsctl[2991]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-exist add-port br0 mgm vlan_mode=access -- set Interface mgm type=internal -- set interface mgm external-ids:iface-id=vm10-mgm-vif
Sep 12 16:11:07 vm10 ifup[2858]: Failed to bring up bond0.
Sep 12 16:11:07 vm10 ifup[2858]: Cannot find device "bond0"
Sep 12 16:11:07 vm10 kernel: bnx2x 0000:01:00.1 em2: NIC Link is Up, 10000 Mbps full duplex, Flow control: none
Sep 12 16:11:06 vm10 kernel: bnx2x 0000:01:00.1 em2: using MSI-X IRQs: sp 50 fp[0] 52 ... fp[7] 59
Sep 12 16:11:06 vm10 kernel: bnx2x 0000:01:00.0 em1: NIC Link is Up, 10000 Mbps full duplex, Flow control: none
Sep 12 16:11:06 vm10 kernel: bnx2x 0000:01:00.0 em1: using MSI-X IRQs: sp 39 fp[0] 41 ... fp[7] 48
Sep 12 16:11:05 vm10 ifup[2858]: bond0: ERROR while getting interface flags: No such device
Sep 12 16:11:05 vm10 ifup[2858]: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Sep 12 16:11:05 vm10 ovs-vsctl[2922]: ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Sep 12 16:11:05 vm10 ovs-vsctl[2922]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --fake-iface add-bond br0 bond0 em1 em2 vlan_mode=native-untagged bond_mode=balance-tcp lacp=active other_config:lacp-time=fast --
Sep 12 16:11:05 vm10 ifup[2858]: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Sep 12 16:11:05 vm10 ovs-vsctl[2908]: ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Sep 12 16:11:05 vm10 ovs-vsctl[2908]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --may-ex...

Why it has not been backported to Xenial?

I agree, this should be backported to Xenial. Bionic does not support Openvswitch yet through netplan.io, so at least the previous LTS should support it correctly !

If you use the Ubuntu Cloud Archive, newer versions of Openvswitch are available backports to xenial. For more details see: https://wiki.ubuntu.com/OpenStack/CloudArchive

I ran across this today, apparently still not fixed. I was also surprised and shocked that netplan.io doesn't support it.

Hi Dale,

Please can you let us know what release you hit this on?

Thanks,
Corey

Running into the same issue on Cosmic

The OVS and ifupdown boot order is broken again in Bionic. Why the whole openvswitch-nonetwork.service unit was removed from Bionic without appropriate substitute? We have currently fixed it by modifying openvswitch-switch.service ovs-vswitchd.service and ovsdb-server.service units by moving network-pre.target from After to Before clause.

we fix it on bionic by following below tutorial: "Boot support (Ubuntu 16.04 – when using systemd)"
https://www.opencloudblog.com/?p=240