Canonical Juju

agent cannot be up on LXD/Fan network on OpenStack OVN/geneve mtu=1442

Bug #1936842 reported by Nobuto Murata
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Medium
Joseph Phillips
Canonical Juju 2.9.18

Bug Description

When one runs OpenStack with MTU=1500 underlying network, Neutron/OVN by default will create a tenant/overlay network with MTU=1442(1500-58). However, when deploying a workload on top, the Fan network with Juju will use MTU=1450 which is actually bigger than that. Then, Juju agent cannot be up inside LXD/Fan on top of OpenStack.

OpenStack deployment is based on https://jaas.ai/openstack-base
and k8s as a workload on top of OpenStack is: https://jaas.ai/kubernetes-core

$ openstack network list
+--------------------------------------+----------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+----------+--------------------------------------+
| ba9880e3-5f07-4b49-aeda-20dfa8fe66ec | internal | 45efcc4c-fb99-4fe8-98a4-77195ec4aef1 |
| e27b1cef-c53b-42e0-b307-8ba2e622c2dd | ext_net | 5315d907-ce90-4605-ad72-b78229965b40 |
+--------------------------------------+----------+--------------------------------------+

$ openstack network show internal
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
...
| mtu | 1442 |
| name | internal |
...
| provider:network_type | geneve |
| provider:physical_network | None |
| provider:segmentation_id | 1410 |
...
+---------------------------+--------------------------------------+

$ juju machines -m k8s-on-openstack
Machine State DNS Inst id Series AZ Message
0 started 192.168.151.75 0e96a5b1-3665-44f6-bcb7-4851ab6cd22d focal nova ACTIVE
0/lxd/0 pending juju-b9bd8b-0-lxd-0 focal nova Container started
1 started 192.168.151.66 6bfa5d2e-24e6-42b8-b5ec-1f2a0d0e6b02 focal nova ACTIVE

[openstack instance/VM provisioned by Juju - ens3: mtu=1442, fan-252: mtu=1450]

juju-b9bd8b-k8s-on-openstack-0:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:48:85:85 brd ff:ff:ff:ff:ff:ff
3: fan-252: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 06:28:c7:b4:50:eb brd ff:ff:ff:ff:ff:ff
4: ftun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1392 qdisc noqueue master fan-252 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 92:d7:07:6e:b4:db brd ff:ff:ff:ff:ff:ff
5: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:03:0e:96 brd ff:ff:ff:ff:ff:ff
7: 0lxd0-0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master fan-252 state UP mode DEFAULT group default qlen 1000
    link/ether 06:28:c7:b4:50:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0

[lxc config of LXD container as juju machine 0/lxd/0]

juju-b9bd8b-k8s-on-openstack-0:~# lxc config show juju-b9bd8b-0-lxd-0
...
  user.user-data: |
    #cloud-config
    apt_mirror: ""
    bootcmd:
    - install -D -m 644 /dev/null '/etc/netplan/99-juju.yaml'
    - |-
      printf '%s\n' 'network:
        version: 2
        ethernets:
          eth0:
            match:
              macaddress: 00:16:3e:82:f9:44
            dhcp4: true
            nameservers:
              search: [openstack.internal]
              addresses: [8.8.8.8, 8.8.4.4]
            mtu: 1450
...
devices:
  eth0:
    host_name: 0lxd0-0
    hwaddr: 00:16:3e:82:f9:44
    mtu: "1450"
    name: eth0
    nictype: bridged
    parent: fan-252
    type: nic

[cloud-init-output.log - stuck at the initial apt update and also failing to fetch the agent binary]

Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~20.04.1 running 'modules:config' at Mon, 19 Jul 2021 03:27:32 +0000. Up 53.99 seconds.
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Err:2 http://security.ubuntu.com/ubuntu focal-security InRelease
  Connection failed [IP: 192.168.151.1 8000]
Err:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Connection failed [IP: 192.168.151.1 8000]
Err:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Connection failed [IP: 192.168.151.1 8000]
Ign:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
Ign:6 http://archive.ubuntu.com/ubuntu focal/universe Translation-en
Ign:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 c-n-f Metadata
Ign:8 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages
Ign:9 http://archive.ubuntu.com/ubuntu focal/multiverse Translation-en
Ign:10 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 c-n-f Metadata
Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]

+ printf Attempt 5 to download agent binaries from %s...\n https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64
Attempt 5 to download agent binaries from https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64...
+ curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --connect-timeout 20 --noproxy * --insecure -o /var/lib/juju/tools/2.9.8-ubuntu-amd64/tools.tar.gz https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64
curl: (28) Operation timed out after 20000 milliseconds with 0 out of 0 bytes received
agent binaries from https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64 downloaded: HTTP 000; time 20.000946s; size 0 bytes; speed 0.000 bytes/s + echo Download failed, retrying in 15s
Download failed, retrying in 15s
+ sleep 15

[manually run tracepath to get pmtu]

juju-b9bd8b-0-lxd-0:~# tracepath -n -m 5 192.168.151.1
 1?: [LOCALHOST] pmtu 1450
 1: 252.16.0.1 0.346ms
 1: 252.16.0.1 0.051ms
 2: 252.16.0.1 0.053ms pmtu 1442
 2: no reply
 3: 192.168.151.1 3.283ms reached
     Resume: pmtu 1442 hops 3 back 3

Revision history for this message
Nobuto Murata (nobuto) wrote :

For the record, by lowering MTU of eth0 on LXD on top of OpenStack VM by hand as `ip link set eth0 mtu 1442`, that makes binary download successful.

+ printf Attempt 17 to download agent binaries from %s...\n https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64
Attempt 17 to download agent binaries from https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64...
+ curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --connect-timeout 20 --noproxy * --insecure
 -o /var/lib/juju/tools/2.9.8-ubuntu-amd64/tools.tar.gz https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64
curl: (28) Operation timed out after 20001 milliseconds with 0 out of 0 bytes received
agent binaries from https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64 downloaded: HTTP 000; time 20.001599s; size 0 bytes; speed 0.000 bytes/s + echo Download
failed, retrying in 15s
Download failed, retrying in 15s
+ sleep 15
+ n=18
+ true
+ printf Attempt 18 to download agent binaries from %s...\n https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64
Attempt 18 to download agent binaries from https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64...
+ curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --connect-timeout 20 --noproxy * --insecure
 -o /var/lib/juju/tools/2.9.8-ubuntu-amd64/tools.tar.gz https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64
agent binaries from https://192.168.151.101:17070/model/9e5ed9b4-224d-4486-8dac-b56a70b9bd8b/tools/2.9.8-ubuntu-amd64 downloaded: HTTP 200; time 2.124569s; size 124661048 bytes; speed 58691642.000 bytes/s +
echo Agent binaries downloaded successfully.
Agent binaries downloaded successfully.
+ break
+ sha256sum /var/lib/juju/tools/2.9.8-ubuntu-amd64/tools.tar.gz
+ grep 635a3524acc09a557d92c53d3d3a6fa0b3b428fbd504c8d1b8e0d3dbefbf3176 /var/lib/juju/tools/2.9.8-ubuntu-amd64/juju2.9.8-ubuntu-amd64.sha256
635a3524acc09a557d92c53d3d3a6fa0b3b428fbd504c8d1b8e0d3dbefbf3176 /var/lib/juju/tools/2.9.8-ubuntu-amd64/tools.tar.gz

Revision history for this message
Nobuto Murata (nobuto) wrote :

https://bugs.launchpad.net/charmed-kubernetes-bundles/+bug/1919296

Revision history for this message
Joseph Phillips (manadart) wrote :

Looking at /usr/sbin/fanctl, it looks like the Fan MTU is always either 1480 or (usually) 1450.

case "$C_flag_type" in
ipip*)
        C_tun_control="type ipip"
        C_bridge_mtu=1480
        ;;
vxlan)
        # XXX: check that the overlay width is not more than 24 bits.

        # Work out the vxlan ID, which is our overlay >> 8 bits to fix in the
        # vxlan nid.
        local vxlan_id="$(( $overlay_ipnum >> 8 ))"

        C_tun_control="type vxlan id $vxlan_id dev $C_underlay_dev dstport 0"
        C_bridge_mtu=1450
        ;;
esac

The Juju container manager configures the LXD NIC with the MTU of the parent device, so we will have to look at options here.

Changed in juju:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Joseph Phillips (manadart)
Revision history for this message
Joseph Phillips (manadart) wrote :

I've added the ubuntu-fan package, as fanctl should accommodate the underlay MTU.

There is a work-around for the Juju case, which I will look to implement.

Joseph Phillips (manadart)
Changed in juju:
status: Triaged → In Progress
milestone: none → 2.9.17
Revision history for this message
Joseph Phillips (manadart) wrote :

Hmm. Now I see that on Canonistack, this is all set up correctly by default.

2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:5f:24:be brd ff:ff:ff:ff:ff:ff
    inet 10.48.130.242/17 brd 10.48.255.255 scope global dynamic ens2
       valid_lft 85790sec preferred_lft 85790sec
    inet6 fe80::f816:3eff:fe5f:24be/64 scope link
       valid_lft forever preferred_lft forever
3: fan-252: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1408 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:91:cf:78:a0:20 brd ff:ff:ff:ff:ff:ff
    inet 252.5.228.1/8 scope global fan-252
       valid_lft forever preferred_lft forever
    inet6 fe80::891:cfff:fe78:a020/64 scope link
       valid_lft forever preferred_lft forever
4: ftun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1408 qdisc noqueue master fan-252 state UNKNOWN group default qlen 1000
    link/ether 0a:91:cf:78:a0:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::891:cfff:fe78:a020/64 scope link
       valid_lft forever preferred_lft forever

Revision history for this message
Nobuto Murata (nobuto) wrote :

Hmm, I'm not sure where the difference comes from. With Juju 2.9.16 I still see mtu=1442 on VM NIC (expected) and mtu=1450 (bigger than underlying NIC) on fan-252 bridge.

ubuntu@juju-913ba4-k8s-on-openstack-0:~$ brctl show
bridge name bridge id STP enabled interfaces
fan-252 8000.0653e0778a0a no ftun0
                                                        veth966fdd48
lxdbr0 8000.00163e2848ef no
ubuntu@juju-913ba4-k8s-on-openstack-0:~$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:27:18:ce brd ff:ff:ff:ff:ff:ff
3: fan-252: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 06:53:e0:77:8a:0a brd ff:ff:ff:ff:ff:ff
4: ftun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1392 qdisc noqueue master fan-252 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 82:8a:c4:02:b5:77 brd ff:ff:ff:ff:ff:ff
5: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:28:48:ef brd ff:ff:ff:ff:ff:ff
7: veth966fdd48@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master fan-252 state UP mode DEFAULT group default qlen 1000
    link/ether 06:53:e0:77:8a:0a brd ff:ff:ff:ff:ff:ff link-netnsid 0

Revision history for this message
Nobuto Murata (nobuto) wrote :

Let me know what log / log level you want to see to compare. I'm attaching the machine log of the VM for the time being.

Canonical Juju QA Bot (juju-qa-bot)
Changed in juju:
milestone: 2.9.17 → 2.9.18
Revision history for this message
Joseph Phillips (manadart) wrote :

I've removed ubuntu-fan.

I'm working on a fix for this.

What needs to happen is for the container NIC to use the MTU of the VXLAN accompanying the Fan bridge (which appears to be correctly offset from the underlay) rather than the bridge itself.

no longer affects: ubuntu-fan (Ubuntu)
Revision history for this message
Joseph Phillips (manadart) wrote :

This should be addressed by the following patches:
https://github.com/juju/juju/pull/13419
https://github.com/juju/juju/pull/13412

Joseph Phillips (manadart)
Changed in juju:
status: In Progress → Fix Committed
Canonical Juju QA Bot (juju-qa-bot)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.