Bond with OVS bridging RuntimeError: duplicate mac found!
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-init (Ubuntu) |
Fix Released
|
Undecided
|
Dan Watkins |
Bug Description
When using bonds and OVS bridging cloud-init fails with
2021-01-22 18:44:08,094 - util.py[WARNING]: failed stage init
failed run of stage init
-------
Traceback (most recent call last):
File "/usr/lib/
ret = functor(name, args)
File "/usr/lib/
init.
File "/usr/lib/
self.
File "/usr/lib/
present_macs = self.get_
File "/usr/lib/
return net.get_
File "/usr/lib/
return get_interfaces_
File "/usr/lib/
raise RuntimeError(
RuntimeError: duplicate mac found! both 'br-ex.100' and 'br-ex' have mac 'e2:86:e6:60:4c:44'
snap-id: shY22YTZ3RhJJDO
tracking: 2.9/candidate
refresh-date: 3 days ago, at 20:03 UTC
channels:
2.9/stable: 2.9.1-9153-
2.9/candidate: ↑
2.9/beta: ↑
2.9/edge: 2.9.1-9156-
latest/stable: –
latest/candidate: –
latest/beta: –
latest/edge: 2.10.0~
2.8/stable: 2.8.2-8577-
2.8/candidate: 2.8.3~rc1-
2.8/beta: 2.8.3~rc1-
2.8/edge: 2.8.3~rc1-
2.7/stable: 2.7.3-8290-
2.7/candidate: ↑
2.7/beta: ↑
2.7/edge: 2.7.3-8294-
installed: 2.9.1-9153-
David Ames (thedac) wrote : | #1 |
David Ames (thedac) wrote : | #2 |
David Ames (thedac) wrote : | #3 |
Dan Watkins (oddbloke) wrote : | #4 |
This is the network config, pulled out of the log file:
bonds:
bond0:
interfaces:
- enp1s0
- enp7s0
macaddress: 52:54:00:28:fd:fd
mtu: 1500
parameters:
down-delay: 0
gratuitio
mii-
mode: active-backup
transmit-
up-delay: 0
bridges:
br-ex:
addresses:
- 192.168.151.18/24
gateway4: 192.168.151.1
interfaces:
- bond0
macaddress: 52:54:00:28:fd:fd
mtu: 1500
nameservers:
addresses:
- 192.168.151.5
search:
- maas
openvswitch: {}
parameters:
forward-
stp: false
ethernets:
enp1s0:
match:
macaddress: 52:54:00:28:fd:fd
mtu: 1500
set-name: enp1s0
enp7s0:
match:
macaddress: 52:54:00:7c:4e:85
mtu: 1500
set-name: enp7s0
version: 2
vlans:
br-ex.100:
dhcp4: true
id: 100
link: br-ex
mtu: 1500
Changed in cloud-init (Ubuntu): | |
assignee: | nobody → Dan Watkins (oddbloke) |
David Ames (thedac) wrote : | #5 |
Cloud init version:
$ dpkg -l |grep cloud-ini
ii cloud-init 20.4.1-
ii cloud-initramfs
ii cloud-initramfs
Dan Watkins (oddbloke) wrote : | #6 |
The network config below closely reproduces the issue when a LXD VM is launched with it, has openvswitch-switch installed on it (e.g. via a manual DHCP on enp5s0), and is `cloud-init clean --logs --reboot`ed.
The log does not contain the error message, but calling `cloudinit.
If the vlans definition is removed, the instance comes up with networking (after the same reboot process).
MAC_ADDRESS = "de:ad:be:ef:12:34"
NETWORK_CONFIG = """\
bonds:
bond0:
interfaces:
- enp5s0
macaddress: {0}
mtu: 1500
bridges:
ovs-br:
- bond0
mtu: 1500
ethernets:
enp5s0:
mtu: 1500
set-name: enp5s0
match:
version: 2
vlans:
ovs-br.100:
dhcp4: true
id: 100
link: ovs-br
mtu: 1500
""".format(
Dan Watkins (oddbloke) wrote : | #7 |
Also of note: the MAC address being reported as duplicated in both the reported log and in the exception I see is not present in the specified configuration. It's presumably being generated by OVS and applied to ovs-br (and therefore inherited by ovs-br.100?). I'm going to see if a more minimal vlans configuration reproduces.
Dan Watkins (oddbloke) wrote : | #8 |
OK, I have a suspicion of what's going on here. I've compared two systems: one launched with the network config above (and updated/rebooted), and one launched with that config minus the "openvswitch: {{}}" line.
When I compare /sys/class/
The function which raises the exception (get_interfaces
I'm not particularly familiar with OVS: is it somehow expected that this VLAN will not be presented via /sys/class/net as other VLANs are? Or does this suggest that there's a bug in something below cloud-init which isn't correctly configuring this VLAN (which cloud-init then cannot detect as a VLAN, and so fails)?
[0] https:/
[1] https:/
[2] https:/
[3] https:/
(As an aside: both my understanding of the problem and a test of a revert suggest that this isn't a regression caused by https:/
David Ames (thedac) wrote : | #9 |
I am not sure I have any definitivie answers but here are my thoughts.
Compare a VLAN device created with `ip link add`
ip link add link enp6s0 name enp6s0.100 type vlan id 100
cat /sys/class/
DEVTYPE=vlan
INTERFACE=
IFINDEX=3
To an OVS VLAN interface created with ovs-vsctl:
ovs-vsctl add-port br-ex vlan100 tag=200 -- set Interface vlan100 type=internal
cat /sys/class/
INTERFACE=br-ex.100
IFINDEX=7
I suspect this is down to the tooling. OVS is creating virtual devices so it may not be what `ip link` would create.
Could the `is_vlan` function check for the '.' followed by an integer which is the indication of a VLAN in all cases?
Ryan Harper (raharper) wrote : | #10 |
Thanks for doing most of the digging here @Oddbloke; I suspect as with bond and bridges for ovs, we'll need a special case to check if a vlan entry is also OVS, much like we did for bonds/bridges:
https:/
So our is_vlan change will need to see if link device is OVS and if so then say it's a vlan as well (since the DEVTYPE doesn't match) or something to that effect.
Dan Watkins (oddbloke) wrote : Re: [Bug 1912844] Re: Bond with OVS bridging RuntimeError: duplicate mac found! | #11 |
On Fri, Jan 22, 2021 at 10:48:56PM -0000, David Ames wrote:
> I am not sure I have any definitivie answers but here are my thoughts.
>
> Compare a VLAN device created with `ip link add`
>
> ip link add link enp6s0 name enp6s0.100 type vlan id 100
>
> cat /sys/class/
> DEVTYPE=vlan
> INTERFACE=
> IFINDEX=3
>
>
> To an OVS VLAN interface created with ovs-vsctl:
>
> ovs-vsctl add-port br-ex vlan100 tag=200 -- set Interface vlan100
> type=internal
>
> cat /sys/class/
> INTERFACE=br-ex.100
> IFINDEX=7
Thanks for isolating how these devices are created!
> I suspect this is down to the tooling. OVS is creating virtual devices
> so it may not be what `ip link` would create.
I don't believe that cloud-init has changed anything in this area, so I
would still like confirmation that this is a case that cloud-init
definitely needs to workaround. (Rather than it being the case that
there's an underlying bug which, being involved in networking early in
boot, we are the first to encounter.)
> Could the `is_vlan` function check for the '.' followed by an integer
> which is the indication of a VLAN in all cases?
Using device names is generally not reliable. In this specific case,
with this config passed to a LXD VM:
bridges:
br.100:
dhcp4: true
- enp5s0
mtu: 1500
ethernets:
enp5s0:
mtu: 1500
version: 2
it comes up with working networking and I see:
# cat /sys/class/
DEVTYPE=bridge
INTERFACE=br.100
IFINDEX=3
(While you (and I) can certainly question the wisdom of naming a non-VLAN
like this, cloud-init's code cannot assume that there aren't users out
there doing this, for whatever reason.)
Dan Watkins (oddbloke) wrote : | #12 |
On Fri, Jan 22, 2021 at 10:51:25PM -0000, Ryan Harper wrote:
> Thanks for doing most of the digging here @Oddbloke; I suspect as with
> bond and bridges for ovs, we'll need a special case to check if a vlan
> entry is also OVS, much like we did for bonds/bridges:
>
> https:/
>
> So our is_vlan change will need to see if link device is OVS and if so
> then say it's a vlan as well (since the DEVTYPE doesn't match) or
> something to that effect.
Unless I'm missing something, we don't have a network configuration to
reference in these codepaths: get_interfaces only takes a
blacklist_drivers parameter, and is_vlan only takes a devname.
So for the code to work as-architected, I believe we need to be able to
determine that this is a VLAN from examining the system (via
/sys/class/net, most likely) to be able to exclude it in get_interfaces.
As far as I (with my limited networking knowledge) can tell, we can
neither determine that this is a VLAN, nor that this is related to the
ovs-br interface by examining /sys/class/net: while the non-OVS VLAN has
a lower_ link to the bridge interface, the OVS VLAN does not.
Looking at everything in /sys/class/
*; do echo $f: $(cat $f); done 2>/dev/null`), here's the diff between
the two systems:
--- not-ovs 2021-01-25 13:15:34.560602978 -0500
+++ ovs 2021-01-25 13:15:23.400407103 -0500
@@ -1,26 +1,25 @@
-addr_assign_type: 2
+addr_assign_type: 3
addr_len: 6
-address: de:ad:be:ef:12:34
+address: 56:1d:35:09:77:47
broadcast: ff:ff:ff:ff:ff:ff
carrier: 1
-carrier_changes: 1
+carrier_changes: 0
carrier_
-carrier_up_count: 1
+carrier_up_count: 0
dev_id: 0x0
dev_port: 0
dormant: 0
duplex:
-flags: 0x1003
+flags: 0x1103
gro_flush_timeout: 0
ifalias:
-ifindex: 5
-iflink: 4
+ifindex: 6
+iflink: 6
link_mode: 0
-lower_br:
mtu: 1500
name_assign_type: 3
netdev_group: 0
-operstate: up
+operstate: unknown
phys_port_id:
phys_port_name:
phys_switch_id:
@@ -31,4 +30,4 @@
subsystem:
tx_queue_len: 1000
type: 1
-uevent: DEVTYPE=vlan INTERFACE=br.100 IFINDEX=5
+uevent: INTERFACE=
With addr_assign_type set to 3, and no DEVTYPE=vlan, and no lower_*
link, I don't see how we can tell that this is a VLAN. (I've checked
and the difference in flags is, if I did my bitmasking correctly,
whether the interface is in promiscuous mode or not.)
Frode Nordahl (fnordahl) wrote : | #13 |
Particulars about what role a Open vSwitch port/interface has is unfortunately not exposed through sysfs nor iproute2 tools POV. Open vSwitch and its data path driver would manage that based on configuration stored in Open vSwitch.
If it is sufficient to know that this is a Open vSwitch managed port I guess we could get that from udev, for example:
# udevadm info /sys/class/
P: /devices/
L: 0
E: DEVPATH=
E: INTERFACE=enp6s0.9
E: IFINDEX=7
E: SUBSYSTEM=net
E: USEC_INITIALIZE
E: ID_MM_CANDIDATE=1
E: ID_NET_
E: ID_NET_
E: ID_NET_
E: SYSTEMD_
E: TAGS=:systemd:
If you really need to know the specific role the port has you would have to query Open vSwitch:
Create a Open vSwitch vlan interface:
# ovs-vsctl add-br br0
# ovs-vsctl add-port br0 enp6s0
# ovs-vsctl add-port br0 enp6s0.9 tag=9 -- set interface enp6s0.9 type=internal
Alternatively you could do the same with "fake bridges" which allows for a slightly different CLI experience, I think netplan does the latter. Anyway, the underlying data structures inside Open vSwitch should be similar.
# ovs-vsctl find interface name=enp6s0.9
_uuid : 72ce2c2e-
admin_state : down
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
error : []
external_ids : {}
ifindex : 7
ingress_
ingress_
lacp_current : []
link_resets : 0
link_speed : []
link_state : down
lldp : {}
mac : []
mac_in_use : "d2:27:14:2d:60:88"
mtu : 1500
mtu_request : []
name : enp6s0.9
ofport : 2
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_missed_errors=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
status : {driver_
type : internal
# ovs-vsctl find port name=enp6s0.9
_uuid : 2c9e20de-
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
cvlans : []
external_ids : {}
fake_bridge : false
interfaces : [72ce2c2e-
lacp : []
mac : []
name : enp6s0.9
other_config : {}
protected : false
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 9
trunks : []
vlan_mode : []
# ovs-vsctl port-to-br enp6s0.9
br0
# ovs-vsctl find bridge name=br0
_uuid ...
Frode Nordahl (fnordahl) wrote : | #14 |
fwiw; Open vSwitch do distribute Python client libraries [0], but they may be a bit complex for this simple use case. We do maintain a SimpleOVSDB interface [1] and there is an example of how it could be used to iterate over stuff [2]. Feel free to use, steal or whatever suits your needs :)
0: https:/
1: https:/
2: https:/
Dan Watkins (oddbloke) wrote : | #15 |
I can confirm that udev does report the VLAN as OVS-managed:
# udevadm info /sys/class/
P: /devices/
L: 0
E: DEVPATH=
E: INTERFACE=
E: IFINDEX=5
E: SUBSYSTEM=net
E: USEC_INITIALIZE
E: ID_MM_CANDIDATE=1
E: ID_NET_
E: ID_NET_
E: ID_NET_
E: SYSTEMD_
E: TAGS=:systemd:
so this is a feasible approach, at least.
I have reservations about introducing a call to an external program in this codepath; I believe we'd have to call it for every interface (that wasn't excluded via some earlier check), and subprocess calls are much more expensive than reading files. Via local testing with timeit: is_vlan takes ~48.3 usec per loop, a subp of `udevadm info ...` takes ~2.48 msec per loop. This isn't too substantial in most systems, but in systems with more interfaces (or slower `udevadm`?), it could become something of an issue.
I suspect, however, that we can find a way of gating this check on the presence of OVS somehow: if openvswitch-switch is not installed, for example, then there's no reason to check `udevadm info`: the given interface will never be OVS-managed.
What's a reliable, cross-distro way of checking if a system might have OVS-managed interfaces?
Frode Nordahl (fnordahl) wrote : | #16 |
Those subprocess calls do indeed add up to be expensive quite quickly, and when I think of it it may actually not be a consistent way of determining if a interface belongs to Open vSwitch in all its configurations. Open vSwitch supports multiple datapath types, and depending on which one you use the interface may or may not be owned by the openvswitch driver.
However, this brings me to something we might use as a consistent cross-distro way of determining whether Open vSwitch is there and has bridges configured.
When Open vSwitch registers a datapath it also creates a virtual port for it, and we could possibly look for that to determine whether Open vSwitch is installed and actually has useful configuration.
The 'system' datapath is the kernel datapath as provided by the openvswitch kernel module, the 'netdev' datapath is used for alternative datapaths such as the Open vSwitch userspace implementation, DPDK, AF_XDP etc.
Example:
# ovs-vsctl show
5ef28194-
Bridge br1
Port br1
Bridge br0
Port br0
ovs_version: "2.13.1"
# ls -l /sys/class/
lrwxrwxrwx 1 root root 0 Feb 4 05:33 /sys/class/
lrwxrwxrwx 1 root root 0 Feb 4 06:06 /sys/class/
Dan Watkins (oddbloke) wrote : | #17 |
Thanks Frode, that's really helpful!
I don't see the `datapath_type` in my output:
e2d9c9b4-
Bridge ovs-br
fail_mode: standalone
Port ovs-br
Port bond0
Port ovs-br.100
tag: 100
ovs_version: "2.13.1"
but I _do_ see /sys/class/
# ls /sys/class/net/ -lah
total 0
drwxr-xr-x 2 root root 0 Feb 9 22:07 .
drwxr-xr-x 31 root root 0 Feb 9 22:07 ..
lrwxrwxrwx 1 root root 0 Feb 9 22:07 bond0 -> ../../devices/
-rw-r--r-- 1 root root 4.0K Feb 9 22:07 bonding_masters
lrwxrwxrwx 1 root root 0 Feb 9 22:07 enp5s0 -> ../../devices/
lrwxrwxrwx 1 root root 0 Feb 9 22:07 lo -> ../../devices/
lrwxrwxrwx 1 root root 0 Feb 9 22:07 ovs-br -> ../../devices/
lrwxrwxrwx 1 root root 0 Feb 9 22:07 ovs-br.100 -> ../../devices/
lrwxrwxrwx 1 root root 0 Feb 9 22:07 ovs-system -> ../../devices/
If I launch a separate system, and install openvswitch-switch, /sys/class/
This leads me to some questions: Will /sys/class/
Changed in cloud-init (Ubuntu): | |
status: | New → In Progress |
Frode Nordahl (fnordahl) wrote : | #18 |
The default `datapath_type` is 'system', so if it is not explicitly specified for a bridge it will not be visible in `ovs-vsctl show` output, but 'system' will still be the `datapath_type` used.
If you configure a system where all bridges have `datapath_type` 'netdev' /sys/class/
You can query Open vSwitch at runtime for which datapath types it supports with `ovs-vsctl get open_vswitch . datapath_types` which would produce '[netdev, system]' as output.
I guess we cannot foresee how this will be plumbed for all future Open vSwitch releases and possible new datapath types, but either `ovs-system` or `ovs-netdev` will always be there in current versions and use cases.
If we want to be opportunistic I guess iterating over the datapath_types list and check for /sys/class/
Dan Watkins (oddbloke) wrote : | #19 |
> The default `datapath_type` is 'system', so if it is not explicitly specified for a bridge it will not be visible in `ovs-vsctl show` output, but 'system' will still be the `datapath_type` used.
Great, I figured it wasn't material.
> You can query Open vSwitch at runtime for which datapath types it supports with `ovs-vsctl get open_vswitch . datapath_types` which would produce '[netdev, system]' as output.
Will `ovs-vsctl` always be available if OVS interfaces might be present on the system? Will it always be in (our systemd units') PATH?
(My _guess_ is that we can't be sure, about the latter at least: /opt/openvswitc
> If we want to be opportunistic I guess iterating over the datapath_types list and check for /sys/class/
I think this would be ideal, but we can avoid introducing a (runtime-detected, optional) dependency on `ovs-vsctl` if we hardcode the current options: as discussed above, this may even be necessary. Alternatively, we could opt into the more expensive lookup path if we find anything matching /sys/class/
What do you think?
Frode Nordahl (fnordahl) wrote : | #20 |
So I would think it to be odd to not have `ovs-vsctl` available if you're running Open vSwitch, but as you point out we have no way of knowing how or where other distributions or image builders would think to put the binary, so that is a fair point.
I'm leaning towards specifically checking for '/sys/class/
Dan Watkins (oddbloke) wrote : | #21 |
I've figured out why my LXD reproducer doesn't reproduce exactly: NoCloud runs at both local and net stages, so the code in question is called earlier in boot than the OpenStack data source is. For now, I'll proceed with the synthetic reproducer: calling the Python code which fails directly.
Dan Watkins (oddbloke) wrote : | #22 |
Hey Frode,
Now moving on from the "does this system have any OVS-managed interfaces?" to "how can I tell if a particular interface is managed by OVS?":
We discussed using `udevadm info` to determine if an interface is OVS-managed:
> If it is sufficient to know that this is a Open vSwitch managed port I guess we could get that from udev, for example:
>
> # udevadm info /sys/class/
> ...
> E: ID_NET_
> ...
But later we also discussed that OVS-managed interfaces may not be owned by the `openvswitch` kernel driver:
> Open vSwitch supports multiple datapath types, and depending on which one you use the interface may or may not be owned by the openvswitch driver.
It looks to me like these two statements are in opposition: if OVS is managing an interface via a different datapath, then it won't have ID_NET_
If this is the case, then I think we have a couple of options.
Firstly, we could scope this down to only handle system datapath interfaces, so that it _is_ true that all the interfaces we're handling are owned by the openvswitch. I don't know enough about how OVS is used (either by us or more generally) to know if this is a reasonable suggestion, so your guidance would be appreciated.
If we can't do that, then I think we need to ask OVS directly, presumably via `ovs-vsctl show` as discussed above. This would require it to be in PATH, of course; avoiding that is a nice-to-have at best, so I think that's fine.
Does that sound right to you?
Frode Nordahl (fnordahl) wrote : | #23 |
Hello Dan,
> It looks to me like these two statements are in opposition: if OVS is managing an interface via a different datapath, then it won't have ID_NET_
>
> If this is the case, then I think we have a couple of options.
That is correct, I remembered the other datapath types after suggesting the driver approach.
> Firstly, we could scope this down to only handle system datapath interfaces, so that it _is_ true that all the interfaces we're handling are owned by the openvswitch. I don't know enough about how OVS is used (either by us or more generally) to know if this is a reasonable suggestion, so your guidance would be appreciated.
For the specific use case we're testing with right now it would be safe to assume the system datapath, however our product does also support the use of DPDK acceleration which is hinged on the netdev datapath. So I'm afraid leaving that out of scope might come back and bite us before we know it. But I guess it would be reasonable to split the work up in bite sized chunks as long as we allow for supporting this in the design.
> If we can't do that, then I think we need to ask OVS directly, presumably via `ovs-vsctl show` as discussed above. This would require it to be in PATH, of course; avoiding that is a nice-to-have at best, so I think that's fine.
Yes, we would be able to find the interface in the runtime state through `ovs-vsctl` and friends. The `show` sub-command is mostly meant for human readable overview of configuration, so we will probably dive to the depths of the interface and port tables.
Many of the sub-commands support JSON output format, for example:
`ovs-vsctl -f json find interface name=testport`
`ovs-vsctl -f json find port name=testport`
Dan Watkins (oddbloke) wrote : | #24 |
To ensure that we understand the consequences of these changes, I've spent a bit of time tracking down everywhere this will affect in cloud-init by looking up the various call chains of `get_interfaces`:
Called by:
* `_get_current_
* `_rename_
* `apply_
* `Distro.
* `Init._
* `Init.apply_
* `get_ib_
* `helpers.
* {`ConfigDrive`, `OpenStack`, `IBMCloud`
* `get_interfaces
* `get_interfaces
* `OpenNebulaNetwork` -- used to determine physical NICs for network config generation
* Oracle -- a couple of ways in network config generation
* EC2 -- network config conversion (theirs to ours)
* `helpers.
* `helpers.
* `helpers.
* `net.get_
* `find_fallback_
* `find_fallback_
* `Networking.
* `Init.apply_
Most of these are related to converting a network configuration format provided by a cloud into our own formats. None of this generation includes handling for creating OVS config (unsurprisingly!), so those cases should all be unaffected by changes to `get_interfaces` around OVS (except for a very minor performance hit for any new checks). Others are only used on *BSD, so we also don't need to worry about OVS interfaces there.
Every other call originates from `Init.apply_
David Ames (thedac) wrote : | #25 |
- ovs-vsctl-failure.png Edit (11.5 KiB, image/png)
Running with Dan's PPA [0] on the final reboot cloud-init fails with the following. See attached image.
Stderr: ovs-vsctl: unix:/var/
I'll try and get the rest of the error content (not from console) for further debugging. But it seems the ovs command is being called before ovs is up, perhaps?
[0] https:/
David Ames (thedac) wrote : | #26 |
With the latest update to the PPA [0] I can deploy a full OpenStack with machines with two VLAN interfaces each and respective spaces.
For an OVN deploy this currently requires two PPAs [0] and [1].
[0] https:/
[1] https:/
Dan Watkins (oddbloke) wrote : | #27 |
> But I guess it would be reasonable to split the work up in bite sized chunks as long as we allow for supporting this in the design.
Having looked a little more, I don't think an incremental approach buys us much here: we'd have to replace the `udevadm` code with `ovs-vsctl` code in the next stage anyway (rather than extending it). We may as well just take the approach that will address both in the first place.
The next question is exactly which interfaces we should be excluding from the set of interfaces we consider. At the moment, my POC is excluding interfaces whose name matches an OVS bridge, determined via `ovs-vsctl list-br`. In an instance with an additional VLAN to the above configurations, I see:
# ovs-vsctl list-br
ovs-br
ovs-br.100
ovs-br.200
Does this seems appropriate? I also notice that querying for internal interfaces returns the same set:
# ovs-vsctl find interface type=internal | grep ^name
name : ovs-br
name : ovs-br.200
name : ovs-br.100
I don't think we want to exclude every interface known to OVS, because I believe that would regress bug 1898997. From an instance launched from the integration test for that bug:
cb6840fc-
Bridge ovs-br
fail_mode: standalone
Port enp5s0
Port ovs-br
ovs_version: "2.13.1"
We _do_ still want to consider enp5s0 in cloud-init's code, because it's a real interface that isn't (entirely?) configured by OVS.
Thoughts? (If this isn't a sufficient problem description, let me know!)
Dan Watkins (oddbloke) wrote : | #28 |
Another question: is there a canonical way to determine if OVS isn't up? Currently I'm trying to execute a command and looking for "database connection failed" in the output, is that appropriate?
Frode Nordahl (fnordahl) wrote : | #29 |
> The next question is exactly which interfaces we should be excluding from the set of interfaces we consider. At the moment, my POC is excluding interfaces whose name matches an OVS bridge, determined via `ovs-vsctl list-br`. In an instance with an additional VLAN to the above configurations, I see:
>
> # ovs-vsctl list-br
> ovs-br
> ovs-br.100
> ovs-br.200
>
> Does this seems appropriate? I also notice that querying for internal interfaces returns the same set:
While listing the bridges would work for configurations that make use
of the 'fake-bridge' paradigm, it would not work for listing
interfaces in configurations that do not use it. So I would rely on
querying the port/interface tables as entry point instead.
> # ovs-vsctl find interface type=internal | grep ^name
> name : ovs-br
> name : ovs-br.200
> name : ovs-br.100
The interface types I know about today are 'dpdk', 'system', and 'internal'.
Interfaces of type 'dpdk' would probably be invisible from the kernel
sysfs and netlink interfaces pov, interfaces of type 'system' have
their origin in the system and cloud-init would most likely already
know all about them. Interfaces of type 'internal' may be used for
other things than VLANs so depending on what you want to match on it
may or may not be precise enough. See below for some further
discussion.
> I don't think we want to exclude every interface known to OVS, because I believe that would regress bug 1898997. From an instance launched from the integration test for that bug:
>
> cb6840fc-
> Bridge ovs-br
> fail_mode: standalone
> Port enp5s0
> Interface enp5s0
> Port ovs-br
> Interface ovs-br
> type: internal
> ovs_version: "2.13.1"
>
> We _do_ still want to consider enp5s0 in cloud-init's code, because it's a real interface that isn't (entirely?) configured by OVS.
>
> Thoughts? (If this isn't a sufficient problem description, let me know!)
Do I understand correctly that the goal is to find any OVS managed
interface with a VLAN tag to exclude it from the duplicate MAC check?
If so the following may be an approach to find them:
ovs-vsctl find port 'tag>0'
From the port name you can find the associated interfaces and there is
also a shorthand to find on which bridge a port belongs:
ovs-vsctl port-to-br <PORT NAME>
Frode Nordahl (fnordahl) wrote : | #30 |
> Another question: is there a canonical way to determine if OVS isn't up? Currently I'm trying to execute a command and looking for "database connection failed" in the output, is that appropriate?
In the Ubuntu systemd service we assess the database socket:
https:/
But I guess assessing tools most likely to be in your path would be
more portable, so calling `ovs-vsctl` and assessing the result would
work ok.
One thing to keep in mind is that the OVS/OVN -ctl tools may get into
a situation where they wait indefinitely. That can for example happen
if the OVSDB server is running but the vSwitch is not.
To mitigate that you can use the -t or --timeout option when calling
`ovs-vsctl`.
To assess whether the `ovs-vswitchd` process is alive independently of
the OVSDB you could have an additional check that talks to it on its
control socket, for example:
`ovs-appctl -t ovs-vswitchd version`
Dan Watkins (oddbloke) wrote : | #31 |
> Interfaces of type 'internal' may be used for other things than VLANs so depending on what you want to match on it may or may not be precise enough.
So the cloud-init code in question is used in a couple of (relevant) ways: (a) to determine the state of any physical interfaces for which we should wait before proceeding to apply network configuration to the system (intended for the case where network devices have not yet been detected by the kernel, for a variety of reasons), and (b) to determine the current system state when renaming any such interfaces to match the specified network configuration.
The code in question is iterating through every interface in the system (as determined from /sys/class/net) and determining if it should be included in this set. For reference, the code excludes bridges, VLANs, bonds, any device that has a `/master` symlink that isn't a bridge/bond member, NET_FAILOVER devices, devices without a MAC, devices with a "stolen" MAC, and, on some clouds, interfaces owned by a particular device driver (the one case I recall is for Mellanox interfaces on Azure).
We're looking to answer the question of "is this interface one that cloud-init needs to know about", so I think what we want is to exclude any OVS interface that doesn't originate from the system: OVS will handle naming non-system interfaces correctly and we know that, even if currently absent, they will be present once "needed" (because OVS will create them).
> Interfaces of type 'dpdk' would probably be invisible from the kernel
> sysfs and netlink interfaces pov, interfaces of type 'system' have
> their origin in the system and cloud-init would most likely already
> know all about them. Interfaces of type 'internal' may be used for
> other things than VLANs so depending on what you want to match on it
> may or may not be precise enough.
This suggests to me that internal interfaces _are_ the right ones exclude. I considered excluding _all_ non-"system" interfaces, but in the example config we've been using here, I see:
# ovs-vsctl --columns type find interface name=bond0
type : ""
We'd exclude the bond0 interface anyway (for being a bond) but it makes me wonder if there are other interfaces that we _wouldn't_ otherwise exclude that won't have type=system. To avoid having to answer that, I think we can safely limit this to internal interfaces (as addressing this problem, and likely a class of related ones) and if we find cases that this doesn't cover then we can drive the required changes at that point.
> In the Ubuntu systemd service <snip>
Thanks, this is a bunch of useful info! I'll experiment with a few of these options.
Launchpad Janitor (janitor) wrote : | #32 |
This bug was fixed in the package cloud-init - 21.1-19-
---------------
cloud-init (21.1-19-
* d/cloud-
(LP: #1918303)
* New upstream snapshot.
- .travis.yml: generate an SSH key before running tests (#848)
- write passwords only to serial console, lock down cloud-init-
(#847) (LP: #1918303)
- Fix apt default integration test (#845)
- integration_tests: bump pycloudlib dependency (#846)
- commit f35181fa970453b
- archlinux: Fix broken locale logic (#841)
[Kristian Klausen] (LP: #1402406)
- Integration test for #783 (#832)
- integration_tests: mount more paths IN_PLACE (#838)
- Fix requiring device-number on EC2 derivatives (#836) (LP: #1917875)
- Remove the vi comment from the part-handler example (#835)
- net: exclude OVS internal interfaces in get_interfaces (#829)
(LP: #1912844)
- tox.ini: pass OS_* environment variables to integration tests (#830)
- integration_tests: add OpenStack as a platform (#804)
- Add flexibility to IMDS api-version (#793) [Thomas Stringer]
- Fix the TestApt tests using apt-key on Xenial and Hirsute (#823)
[Paride Legovini] (LP: #1916629)
- doc: remove duplicate "it" from nocloud.rst (#825) [V.I. Wood]
- archlinux: Use hostnamectl to set the transient hostname (#797)
[Kristian Klausen]
- cc_keys_
(#824) [dermotbradley]
- Update cc_set_hostname documentation (#818) [Toshi Aoyama]
-- James Falcon <email address hidden> Fri, 19 Mar 2021 14:32:13 -0500
Changed in cloud-init (Ubuntu): | |
status: | In Progress → Fix Released |
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote : | #33 |
Is this going to be backported to focal (see LP: #1919493)?
Dan Watkins (oddbloke) wrote : | #34 |
On Tue, Mar 23, 2021 at 07:53:59AM -0000, Alfonso Sanchez-Beato wrote:
> Is this going to be backported to focal (see LP: #1919493)?
Yep, the SRU process has been started already.
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote : | #35 |
@Dan, was this released to focal in the end? Thanks!
Screen capture of the network config in MAAS