test_101_neutron_sriov_config hangs on xenial-mitaka

test_101_neutron_sriov_config hang on xenial-mitaka

For example, see timeout at: https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/799266

juju status
-----------

Model Controller Cloud/Region Version SLA Timestamp
zaza-f6089ab94abb coreycb-serverstack serverstack/serverstack 2.8.11 unsupported 13:20:05Z

App Version Status Scale Charm Store Rev OS Notes
glance 12.0.0 active 1 glance jujucharms 306 ubuntu
keystone 9.3.0 active 1 keystone jujucharms 325 ubuntu
neutron-api 8.4.0 active 1 neutron-api jujucharms 295 ubuntu
neutron-gateway 8.4.0 active 1 neutron-gateway jujucharms 292 ubuntu
neutron-openvswitch 8.4.0 blocked 2 neutron-openvswitch local 0 ubuntu
nova-cloud-controller 13.1.4 active 1 nova-cloud-controller jujucharms 355 ubuntu
nova-compute 13.1.4 active 2 nova-compute jujucharms 330 ubuntu
percona-cluster 5.6.37 active 1 percona-cluster jujucharms 298 ubuntu
rabbitmq-server 3.5.7 active 1 rabbitmq-server jujucharms 280 ubuntu

Unit Workload Agent Machine Public address Ports Message
glance/0* active idle 4 10.5.3.240 9292/tcp Unit is ready
keystone/0* active idle 3 10.5.0.98 5000/tcp Unit is ready
neutron-api/0* active idle 2 10.5.3.101 9696/tcp Unit is ready
neutron-gateway/0* active idle 5 10.5.1.174 Unit is ready
nova-cloud-controller/0* active idle 6 10.5.2.127 8774/tcp Unit is ready
nova-compute/0* active idle 7 10.5.1.67 Unit is ready
  neutron-openvswitch/0* blocked idle 10.5.1.67 Services not running that should be: neutron-sriov-agent
nova-compute/1 active idle 8 10.5.0.202 Unit is ready
  neutron-openvswitch/1 blocked idle 10.5.0.202 Services not running that should be: neutron-sriov-agent
percona-cluster/0* active idle 0 10.5.1.207 3306/tcp Unit is ready
rabbitmq-server/0* active idle 1 10.5.1.151 5672/tcp Unit is ready

Machine State DNS Inst id Series AZ Message
0 started 10.5.1.207 a0a916c4-4fa8-4023-b32c-dc765f91694f xenial nova ACTIVE
1 started 10.5.1.151 f49d2a4b-e4e9-46ec-a595-c8d500ce3e82 xenial nova ACTIVE
2 started 10.5.3.101 3bb92c88-b771-4b6e-9712-1f90d6ffcfa1 xenial nova ACTIVE
3 started 10.5.0.98 f89ce7c4-8235-41c4-98cc-4c5aa2321de5 xenial nova ACTIVE
4 started 10.5.3.240 58b1aa43-3e6d-4888-9df7-d92adc534e6b xenial nova ACTIVE
5 started 10.5.1.174 0167d813-51f3-42b4-8a84-9d805c56bc86 xenial nova ACTIVE
6 started 10.5.2.127 55ca3f32-5398-4219-9485-c0848f61a607 xenial nova ACTIVE
7 started 10.5.1.67 734fd7b9-5fcb-4f05-b3ed-2fafb9357cbf xenial nova ACTIVE
8 started 10.5.0.202 94e5f12e-7b13-45f3-ba81-02c93977b8fa xenial nova ACTIVE

/var/log/neutron/neutron-sriov-agent.log
----------------------------------------

2021-07-15 12:46:11.446 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.eswitch_manager [-] Failed to get devices for eth42
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [-] Agent Initialization Failed
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent Traceback (most recent call last):
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 455, in main
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent polling_interval)
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 114, in __init__
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent exclude_devices)
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 191, in setup_eswitch_mgr
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent self.eswitch_mgr.discover_devices(device_mappings, exclude_devices)
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 345, in discover_devices
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent exclude_devices.get(dev_name, set()))
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 348, in _create_emb_switch
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent embedded_switch = EmbSwitch(phys_net, dev_name, exclude_devices)
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 118, in __init__
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent self._load_devices(exclude_devices)
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 125, in _load_devices
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent scanned_pci_list = PciOsWrapper.scan_vf_devices(self.dev_name)
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 51, in scan_vf_devices
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent reason=_("Device not found"))
2021-07-15 12:46:11.448 22101 ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent InvalidDeviceError: Invalid Device eth42: Device not found

ip a
----

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8958 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:59:75:b1 brd ff:ff:ff:ff:ff:ff
    inet 10.5.1.67/16 brd 10.5.255.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe59:75b1/64 scope link
       valid_lft forever preferred_lft forever
3: fan-252: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8908 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:12:0b:7b:97:a5 brd ff:ff:ff:ff:ff:ff
    inet 252.1.67.1/8 scope global fan-252
       valid_lft forever preferred_lft forever
    inet6 fe80::3812:bff:fe7b:97a5/64 scope link
       valid_lft forever preferred_lft forever
4: ftun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8908 qdisc noqueue master fan-252 state UNKNOWN group default qlen 1000
    link/ether 3a:12:0b:7b:97:a5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::3812:bff:fe7b:97a5/64 scope link
       valid_lft forever preferred_lft forever
5: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 96:95:39:15:17:2c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9495:39ff:fe15:172c/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link
       valid_lft forever preferred_lft forever
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
    link/ether 0e:80:6c:98:1d:51 brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST> mtu 1458 qdisc noop state DOWN group default qlen 1
    link/ether 02:0e:e2:0a:10:4a brd ff:ff:ff:ff:ff:ff
10: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
    link/ether be:e3:1d:d6:45:4a brd ff:ff:ff:ff:ff:ff
11: br-data: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
    link/ether fe:c5:4d:16:82:45 brd ff:ff:ff:ff:ff:ff
12: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
    link/ether a2:30:fa:e6:b6:40 brd ff:ff:ff:ff:ff:ff
13: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
14: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
16: gre_sys@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
    link/ether 2e:7c:6a:18:4e:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2c7c:6aff:fe18:4e02/64 scope link
       valid_lft forever preferred_lft forever
17: qbr846e7272-be: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc noqueue state UP group default qlen 1000
    link/ether 16:bf:13:81:95:24 brd ff:ff:ff:ff:ff:ff
18: qvo846e7272-be@qvb846e7272-be: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1458 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether ee:a3:47:aa:f1:31 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::eca3:47ff:feaa:f131/64 scope link
       valid_lft forever preferred_lft forever
19: qvb846e7272-be@qvo846e7272-be: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1458 qdisc noqueue master qbr846e7272-be state UP group default qlen 1000
    link/ether 16:bf:13:81:95:24 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::14bf:13ff:fe81:9524/64 scope link
       valid_lft forever preferred_lft forever
20: tap846e7272-be: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc pfifo_fast master qbr846e7272-be state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:5b:cc:ec brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe5b:ccec/64 scope link
       valid_lft forever preferred_lft forever