Wrong MTU values for container's NICs

Looks like juju detects correct settings from the machine (look for 192.168.33 subnet):

2017-11-21 00:51:24 DEBUG juju.worker.machiner machiner.go:172 observed network config updated for "machine-16" to [{1 127.0.0.0/8 65536 0 lo loopback false false loopback 127.0.0.1 [] [] []} {1 ::1/128 65536 0 lo loopback false false loopback ::1 [] [] []} {2 14:02:ec:42:38:dc 1500 0 eno1 ethernet false false manual [] [] []} {3 5c:b9:01:9a:3c:88 10.245.208.0/20 1500 0 eth4 ethernet false false static 10.245.209.20 [] [] []} {3 5c:b9:01:9a:3c:88 1500 0 eth4 ethernet false false manual [] [] []} {4 14:02:ec:42:38:dd 1500 0 eno2 ethernet false false manual [] [] []} {5 14:02:ec:42:38:de 1500 0 eno3 ethernet false false manual [] [] []} {6 5c:b9:01:9a:3c:89 9000 0 eth1 ethernet false false manual [] [] []} {7 14:02:ec:42:38:df 1500 0 eno4 ethernet false false manual [] [] []} {8 00:11:0a:66:2c:b4 9000 0 eth6 ethernet false false manual [] [] []} {9 00:11:0a:66:2c:b4 9000 0 eth7 ethernet false false manual [] [] []} {10 00:11:0a:66:2c:b4 9000 0 bond0 bond false false manual [] [] []} {11 00:11:0a:66:2c:b4 192.168.35.0/24 9000 0 bond0.2735 802.1q false false static 192.168.35.8 [] [] []} {11 00:11:0a:66:2c:b4 9000 0 bond0.2735 802.1q false false manual [] [] []} {12 00:11:0a:66:2c:b4 192.168.36.0/24 9000 0 bond0.2736 802.1q false false static 192.168.36.8 [] [] []} {12 00:11:0a:66:2c:b4 9000 0 bond0.2736 802.1q false false manual [] [] []} {13 5c:b9:01:9a:3c:89 192.168.33.0/24 9000 0 eth1.2733 802.1q false false static 192.168.33.127 [] [] []} {13 5c:b9:01:9a:3c:89 9000 0 eth1.2733 802.1q false false manual [] [] []} {14 5c:b9:01:9a:3c:89 1500 0 eth1.2734 802.1q false false manual [] [] []} {15 6a:43:a8:e2:cb:49 1500 0 lxdbr0 bridge false false manual [] [] []} {15 6a:43:a8:e2:cb:49 1500 0 lxdbr0 bridge false false manual [] [] []}]

2017-11-21 00:55:14 DEBUG juju.provisioner.lxd host_preparer.go:111 updating observed network config for "machine-16" to []params.NetworkConfig{...
params.NetworkConfig{DeviceIndex:22, MACAddress:"5c:b9:01:9a:3c:89", CIDR:"192.168.33.0/24", MTU:9000, ProviderId:"", ProviderSubnetId:"", ProviderSpaceId:"", ProviderAddressId:"", ProviderVLANId:"", VLANTag:0, InterfaceName:"br-eth1.2733", ParentInterfaceName:"", InterfaceType:"bridge", Disabled:false, NoAutoStart:false, ConfigType:"static", Address:"192.168.33.127", DNSServers:[]string(nil), DNSSearchDomains:[]string(nil), GatewayAddress:"", Routes:[]params.NetworkRoute(nil)}...

But however, creates a container with wrong MTU:

2017-11-21 00:56:32 DEBUG juju.cloudconfig.containerinit container_userdata.go:73 generating network config from container.NetworkConfig{NetworkType:"bridge", Device:"lxdbr0", MTU:0, Interfaces:[]network.Interfa
ceInfo{network.InterfaceInfo{DeviceIndex:0, MACAddress:"00:16:3e:4d:a2:ff", CIDR:"192.168.33.0/24", ProviderId:"227", ProviderSubnetId:"4", ProviderNetworkId:"", ProviderSpaceId:"", ProviderVLANId:"5037", Provid
erAddressId:"409", AvailabilityZones:[]string(nil), VLANTag:273...

We have confirmed that the bridge is created with proper MTU (9000), machiner is reporting proper value to controller, and this value is stored in database.
Then, when we create the e/n/i for the container in container_userdata.go the networkConfig has the MTU set to 1500
For tomorrow my next step is to verify all the steps between those two to see where do we loose the information about the proper MTU.

I’m more worried about MTU setting in LXC’s config itself, rather than
content of ENI. By setting mtu to 1500 in LXC’s config, veth pairs are
created and bridge moves to MTU 1500. This is before container’s ENI
content has any influence on the whole problem.
--
Ante Karamatić
<email address hidden>
Canonical

Source of this information is the same - it's just that information for e/n/i is more visible in the debug log - so the fix will be complete.

New data shows that MAAS had a space and vlan defined with MTU 1500:

    {
        "id": 3,
        "name": "internal-space",
        "subnets": [
            {
                "gateway_ip": null,
                "dns_servers": [],
                "resource_uri": "/MAAS/api/2.0/subnets/3/",
                "active_discovery": false,
                "managed": true,
                "id": 3,
                "allow_proxy": true,
                "vlan": {
                    "mtu": 1500,
                    "fabric": "default",
                    "relay_vlan": null,
                    "resource_uri": "/MAAS/api/2.0/vlans/5033/",
                    "primary_rack": null,
                    "external_dhcp": null,
                    "id": 5033,
                    "fabric_id": 30,
                    "name": "",
                    "dhcp_on": false,
                    "vid": 2733,
                    "space": "internal-space",
                    "secondary_rack": null
                },
                "name": "internal",
                "cidr": "192.168.33.0/24",
                "rdns_mode": 2,
                "space": "internal-space"
            }
        ],
        "vlans": [
            {
                "mtu": 1500,
                "fabric": "default",
                "relay_vlan": null,
                "resource_uri": "/MAAS/api/2.0/vlans/5033/",
                "primary_rack": null,
                "external_dhcp": null,
                "id": 5033,
                "fabric_id": 30,
                "name": "",
                "dhcp_on": false,
                "vid": 2733,
                "space": "internal-space",
                "secondary_rack": null
            }
        ],
        "resource_uri": "/MAAS/api/2.0/spaces/3/"
    },

When one tries to attach an interface to that VLAN, with MTU 9000, it should have raised an issue. Instead, MAAS happily connected two layer 2 objects (VLAN and a NIC) with different MTU values:

            {
                "mac_address": "5c:b9:01:9b:58:a9",
                "params": {
                    "mtu": 9000
                },
                "id": 109,
                "vlan": {
                    "fabric_id": 30,
                    "fabric": "default",
                    "mtu": 1500,
                    "relay_vlan": null,
                    "primary_rack": "abcfkh",
                    "resource_uri": "/MAAS/api/2.0/vlans/5031/",
                    "secondary_rack": "cmkt8t",
                    "name": "untagged",
                    "vid": 0,
                    "dhcp_on": true,
                    "id": 5031,
                    "space": "oam-space",
                    "external_dhcp": null
                },
                "type": "physical",
                "links": [],
                "system_id": "wn67a6",
                "enabled": true,
                "parents": [],
                "name": "eth1",
                "resource_uri": "/MAAS/api/2.0/nodes/wn67a6/interfaces/109/",
                "discovered": null,
                "tags": [
                    "sriov"
                ],
                "effective_mtu": 9000,
                "children": [
                    "eth1.27...

My 2 cents:

If you have both internal and external connectivity provided via the same l2 network (say via different interfaces attached to different vlans or even the same vlan but a different interface) you have to take MTU values further down the path to a remote host into account. Especially if PMTUD cannot be performed due to blocked ICMP on some hops.

So, there are use-cases for such behavior, however, it's best to detect mismatches like that and warn.

So long as hosts on the same link use consistent MTU for point-point connections there is generally no problem with using MTU <= MAX_SUPPORTED_LINK_MTU but we need better error detection and foolproof mechanisms.

Chris, can you give me links to the two follow-on bugs for this one (the foundation bug on us detecting this in a lightweight manner, and the bug against maas/juju/lxd in correctly detecting this sort of mismatch)?

The Foundation bugs are private, but for the record: https://bugs.launchpad.net/cpe-foundation/+bug/1733946 and https://bugs.launchpad.net/cpe-foundation/+bug/1736694

We are still missing an upstream bug (MAAS/Juju/LXD) which I'll file now.

Ah, here we are: https://bugs.launchpad.net/juju/+bug/1733944

I just found an incidence of this.

It appears that it's possible that the vlan definition MTU can be mismatched with the subnet definition of the MTU. This seems the root bug for my case.

$ maas admin networks read
...
    {
        "description": "ceph-access",
        "vlan_tag": 4004,
        "resource_uri": "/MAAS/api/2.0/networks/subnet-6/",
        "name": "subnet-6",
        "netmask": "255.255.0.0",
        "default_gateway": null,
        "dns_servers": [],
        "ip": "10.220.0.0"
    },
...
$ maas admin subnet read subnet-6
{
    "rdns_mode": 2,
    "allow_proxy": true,
    "id": 6,
    "dns_servers": [],
    "name": "ceph-access",
    "cidr": "10.220.0.0/16",
    "space": "ceph-access-space",
    "gateway_ip": null,
    "resource_uri": "/MAAS/api/2.0/subnets/6/",
    "vlan": {
        "vid": 4004,
        "mtu": 9000,
        "fabric": "default",
        "relay_vlan": null,
        "fabric_id": 9,
        "id": 5015,
        "name": "",
        "secondary_rack": null,
        "primary_rack": null,
        "space": "ceph-access-space",
        "external_dhcp": null,
        "resource_uri": "/MAAS/api/2.0/vlans/5015/",
        "dhcp_on": false
    },
    "active_discovery": false,
    "managed": true
}
$ maas admin spaces read
{
        "vlans": [
            {
                "vid": 4004,
                "external_dhcp": null,
                "relay_vlan": null,
                "name": "",
                "fabric": "default",
                "primary_rack": null,
                "resource_uri": "/MAAS/api/2.0/vlans/5015/",
                "fabric_id": 9,
                "secondary_rack": null,
                "dhcp_on": false,
                "id": 5015,
                "space": "ceph-access-space",
                "mtu": 9000
            }
        ],
        "subnets": [
            {
                "allow_proxy": true,
                "name": "ceph-access",
                "gateway_ip": null,
                "vlan": {
                    "vid": 4004,
                    "external_dhcp": null,
                    "relay_vlan": null,
                    "name": "",
                    "fabric": "default",
                    "primary_rack": null,
                    "resource_uri": "/MAAS/api/2.0/vlans/5015/",
                    "fabric_id": 9,
                    "secondary_rack": null,
                    "dhcp_on": false,
                    "id": 5015,
                    "space": "ceph-access-space",
                    "mtu": 9000
                },
                "active_discovery": false,
                "dns_servers": [],
                "resource_uri": "/MAAS/api/2.0/subnets/6/",
                "cidr": "10.220.0.0/16",
                "rdns_mode": 2,
                "id": 6,
                "space": "ceph-access-space",
                "managed": true
            }
        ],
        "id": 2,
        "name": "ceph-access-space",
        "resource_uri": "/MAAS/api/2.0/spaces/2/"
    },

.....

    {
        "vlans": [
            {
                "vid": 4004,
                "external_dhcp": null,
                "relay_vlan": null,
                "name": "4004",
                "fabric": "fabric-0",
       ...