Linux Bridge: can't change the VM's bridge and tap interface MTU at Compute node.

Fix proposed to branch: master
Review: https://review.openstack.org/174558

Patch as-is resulted in traceback:

2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1014, in daemon_loop
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent sync = self.process_network_devices(device_info)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 870, in process_network_devices
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent resync_a = self.treat_devices_added_updated(devices_added_updated)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 910, in treat_devices_added_updated
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent device_details['port_id']):
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 435, in add_interface
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent tap_device_name)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 401, in add_tap_interface
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent self.ensure_tap_mtu(tap_device_name, phy_dev_name)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 425, in ensure_tap_mtu
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent ip_lib.IPDevice(tap_dev_name).link.set_mtu(phy_dev_mtu)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 258, in set_mtu
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent self._as_root('set', self.name, 'mtu', mtu_size)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 242, in _as_root
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent kwargs.get('use_root_namespace', False))
2015-04-16 19:16:24.496 58732 TRACE neutron....

To demonstrate:

Before:

#ip link show br-vlan.440 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
br-vlan.440@br-vlan: mtu 1500 brq33fecdff-b3

#ip link show tap346c55b7-97 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap346c55b7-97: mtu 1500 brq33fecdff-b3

#ip link show tap59ddc0ac-7c | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap59ddc0ac-7c: mtu 1500 brq33fecdff-b3

Change physical MTU & Restart Agent:

# ip link set mtu 9000 dev br-vlan.440
# service neutron-linuxbridge-agent restart

After:

#ip link show br-vlan.440 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
br-vlan.440@br-vlan: mtu 9000 brq33fecdff-b3

#ip link show tap346c55b7-97 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap346c55b7-97: mtu 9000 brq33fecdff-b3

#ip link show tap59ddc0ac-7c | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap59ddc0ac-7c: mtu 9000 brq33fecdff-b3

oops - forgot I disabled rootwrap. Will do a new patchset.

Darragh, thanks for coming up the patch. We are working with Danny (the bug submitter), and actually trying to come up with a patch that would address the issue as a whole. But sort of late to assign the bug to ourself.

I looked at your patch, and felt that it doesn't seem to address the issue completely.
   -- the mtu on the vlan/vxlan interface should be set automatically per the existing config options (network_device_mtu or possibly veth_mtu or the MTU patches merged in Kilo)
   -- the bridge's MTU should be set properly as well.

Let us know if we can work together on this.

Robert: okay, but it is not clear to me what is missing. Can you please test this patch and report its short-comings.

This patch relies on several things about linux bridges and devices that are not obvious:

- With Nova/libvirt/QEMU/KVM, the tap devices are created with the same MTU as the linux bridge.
- The MTU on a linux bridge is always the smallest of all the MTUs of all the devices it enslaves.
- A VLAN device automatically gets the same MTU as its parent when it is created.
- A VXLAN devices gets an MTU 50 bytes lower than its parent.

# ip -o l | cut -d' ' -f2,5 | grep eth1
eth1: 9000

# ip link add link eth1 name eth1.10 type vlan id 10
# ip -o l | cut -d' ' -f2,5 | grep eth1
eth1: 9000
eth1.10@eth1: 9000

# brctl addbr br1
# ip -o l | cut -d' ' -f2,5 | grep br1
br1: 1500

# brctl addif br1 eth1.10
# ip -o l | cut -d' ' -f2,5 | grep br1
br1: 9000

# sudo ip link add vxlan1 type vxlan id 1 group 239.0.0.1 dev eth1
# ip -o l | cut -d' ' -f2,5 | grep vxlan1
vxlan1: 8950

# brctl addif br1 vxlan1
# ip -o l | cut -d' ' -f2,5 | grep br1
br1: 8950

# brctl show
bridge name bridge id STP enabled interfaces
br1 8000.080027eb45b4 no eth1.10
       vxlan1

Hi Darragh, I'll test the patch.

I thought you are going to post a new patch set to fix the tracebacks, or should I just use patch set #1?

Danny, yeah I though I needed to catch up on the recent changes to rootwrap, but maybe not. All the other places in the code do it the same way as the patch.

~$ grep -r link.set_mtu /opt/stack/neutron/ | tail -2
/opt/stack/neutron/neutron/agent/linux/interface.py: root_veth.link.set_mtu(self.conf.network_device_mtu)
/opt/stack/neutron/neutron/agent/linux/interface.py: ns_veth.link.set_mtu(self.conf.network_device_mtu)

James - what version are you using?

Sorry - should have mentioned it was Juno.

The patch does not have a rootwrap problem with trunk. There has been major changes in rootwrap since Juno.

Hi Darragh,

I'm using Kilo and did not see any traceback with your patch.

I'm using a multi-node setup. So I applied your patch to the Compute node and restarted the q-agt service.

When a VM is launched at the Compute node, the correct MTU 9000 is configured at all interfaces and devices:

1. VM eth0 interface:
$ ifconfig eth0 | grep MTU
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1

2. Compute node:
localadmin@qa5:/opt/stack/neutron$ brctl show
bridge name bridge id STP enabled interfaces
brqb5924fb1-0f 8000.6805ca30255b no eth5.300
       tap1425c7b3-f2
virbr0 8000.000000000000 yes

localadmin@qa5:/opt/stack/neutron$ ip -o l | cut -d' ' -f2,5 | grep eth5
eth5: 9000
eth5.300@eth5: 9000

localadmin@qa5:/opt/stack/neutron$ ip -o l | cut -d' ' -f2,5 | grep brq
brqb5924fb1-0f: 9000

localadmin@qa5:/opt/stack/neutron$ ip -o l | cut -d' ' -f2,5 | grep tap
tap1425c7b3-f2: 9000

Fair enough. Thanks for the quick turnaround.

Darragh, I realized that by setting the mtu on the physical interface, the sub interfaces will inherit from that. Thank you for pointing that out.

ok good. So with this patch I think the behavior is the same as nova-network, and is what people expect. The interfaces specified in device_mappings should have their MTUs set by the operating system startup scripts - these will usually be complex devices like bonds with many other parameters.

One last thing, you will need to set network_device_mtu in l3_agent.ini so the other side of the veth (the side in the namespace) gets the correct MTU,

Reviewed: https://review.openstack.org/174558
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6cf92011143eb55adda180ffac91886566fc7826
Submitter: Jenkins
Branch: master

commit 6cf92011143eb55adda180ffac91886566fc7826
Author: Darragh O'Reilly <email address hidden>
Date: Thu Apr 16 18:21:03 2015 +0000

    lb-agent: ensure tap mtu is the same as physical device

    On compute-nodes, Nova creates the bridge with the tap before
    the physical is in the bridge. This causes the tap to have the
    default 1500 MTU which may be different to what is on the physical.
    With this patch the linuxbridge agent ensures that the MTU on the
    tap device is the same as what is on the physical device.

    Change-Id: Id1a4f662ec33ca0333c15eb210366bc850d0d54c
    Closes-Bug: 1443607

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/207171

Reviewed: https://review.openstack.org/207171
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4f9409d0bc2bdaab20fd067a1ec0c522b5ef2784
Submitter: Jenkins
Branch: stable/kilo

commit 4f9409d0bc2bdaab20fd067a1ec0c522b5ef2784
Author: Darragh O'Reilly <email address hidden>
Date: Thu Apr 16 18:21:03 2015 +0000

    lb-agent: ensure tap mtu is the same as physical device

    On compute-nodes, Nova creates the bridge with the tap before
    the physical is in the bridge. This causes the tap to have the
    default 1500 MTU which may be different to what is on the physical.
    With this patch the linuxbridge agent ensures that the MTU on the
    tap device is the same as what is on the physical device.

    Change-Id: Id1a4f662ec33ca0333c15eb210366bc850d0d54c
    Closes-Bug: 1443607
    (cherry picked from commit 6cf92011143eb55adda180ffac91886566fc7826)

Is this done for openvswitch bridges?

Has anyone tested this on Juno ? I tried this today, and i dont see change happening ..