get_ifname_by_pci_address failed

Hi, ubuntu 20.04 amd64 5.4.0-70-generic,
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt probe_vf=0 apparmor=0 selinux=0

root@hp1111:/home/ubuntu# ls -l /sys/class/net | grep 05:10.0
lrwxrwxrwx 1 root root 0 Mar 28 09:42 enp5s0f0v126 -> ../../devices/pci0000:00/0000:00:02.0/0000:05:10.0/net/enp5s0f0v126

root@hp1111:/home/ubuntu# ip link show enp5s0f0v126
132: enp5s0f0v126: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 66:2a:39:9e:e5:16 brd ff:ff:ff:ff:ff:ff

  nova-compute:
    charm: cs:nova-compute
    bindings:
      "": *internal-space
      internal: *internal-space
    options:
      config-flags: default_ephemeral_format=ext4
      enable-live-migration: true
      enable-resize: true
      migration-auth-type: ssh
      openstack-origin: *openstack-origin
      ceph-osd-replication-count: *ceph-osd-replication-count
      libvirt-image-backend: *libvirt-image-backend
      verbose: *verbose
      use-internal-endpoints: True
      restrict-ceph-pools: False
      flat-interface: physnet1 physnet2 prov_sriov_1
      virt-type: kvm
      hugepages: 60%
      reserved-host-memory: 16384
      pci-passthrough-whitelist: '{ "vendor_id": "15b3", "product_id": "1016", "physical_network": "prov_sriov_1" }'

2021-03-28 15:13:58.256 465334 DEBUG nova.network.os_vif_util [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Converting VIF {"id": "9f7aae43-8e80-44fb-b626-f021423e6063", "address": "fa:16:3e:10:ae:3a", "network": {"id": "c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c", "bridge": "br-int", "label": "ext_net2", "subnets": [{"cidr": "192.0.3.0/24", "dns": [], "gateway": {"address": "192.0.3.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "192.0.3.50", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"dhcp_server": "192.0.3.1"}}], "meta": {"injected": false, "tenant_id": "80088d4e3eea4f90958175e58eb7ddbe", "mtu": 1500, "physical_network": "prov_sriov_1", "tunneled": false}}, "type": "ovs", "details": {"port_filter": true}, "devname": "tap9f7aae43-8e", "ovs_interfaceid": "9f7aae43-8e80-44fb-b626-f021423e6063", "qbh_params": null, "qbg_params": null, "active": false, "vnic_type": "direct", "profile": {"capabilities": ["switchdev"], "physical_network": "prov_sriov_1", "pci_vendor_info": "15b3:1016", "pci_slot": "0000:05:10.0"}, "preserve_on_delete": true, "meta": {}} nova_to_osvif_vif /usr/lib/python3/dist-packages/nova/network/os_vif_util.py:501
2021-03-28 15:13:58.257 465334 DEBUG nova.network.os_vif_util [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Converted object VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) nova_to_osvif_vif /usr/lib/python3/dist-packages/nova/network/os_vif_util.py:538
2021-03-28 15:13:58.258 465334 DEBUG os_vif [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Unplugging vif VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True) unplug /usr/lib/python3/dist-packages/os_vif/__init__.py:109
2021-03-28 15:13:58.258 465334 ERROR os_vif [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] Failed to unplug vif VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True): vif_plug_ovs.exception.PciDeviceNotFoundById: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 77, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif plugin.plug(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 305, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif self._plug_vf(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 269, in _plug_vf
2021-03-28 15:13:58.258 465334 ERROR os_vif pf_ifname = linux_net.get_ifname_by_pci_address(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 355, in get_ifname_by_pci_address
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.PciDeviceNotFoundById(id=pci_addr)
2021-03-28 15:13:58.258 465334 ERROR os_vif vif_plug_ovs.exception.PciDeviceNotFoundById: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 680, in _plug_os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif os_vif.plug(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 82, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif raise os_vif.exception.PlugException(vif=vif, err=err)
2021-03-28 15:13:58.258 465334 ERROR os_vif os_vif.exception.PlugException: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found

2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2392, in _build_and_run_instance
2021-03-28 15:13:58.258 465334 ERROR os_vif self.driver.spawn(context, instance, image_meta,
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3743, in spawn
2021-03-28 15:13:58.258 465334 ERROR os_vif self._create_guest_with_network(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6735, in _create_guest_with_network
2021-03-28 15:13:58.258 465334 ERROR os_vif self._cleanup_failed_start(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-28 15:13:58.258 465334 ERROR os_vif self.force_reraise()
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-28 15:13:58.258 465334 ERROR os_vif six.reraise(self.type_, self.value, self.tb)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-03-28 15:13:58.258 465334 ERROR os_vif raise value
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6700, in _create_guest_with_network
2021-03-28 15:13:58.258 465334 ERROR os_vif self.plug_vifs(instance, network_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1169, in plug_vifs
2021-03-28 15:13:58.258 465334 ERROR os_vif self.vif_driver.plug(instance, vif)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 704, in plug
2021-03-28 15:13:58.258 465334 ERROR os_vif self._plug_os_vif(instance, vif_obj)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/virt/libvirt/vif.py", line 684, in _plug_os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.InternalError(msg)
2021-03-28 15:13:58.258 465334 ERROR os_vif nova.exception.InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2218, in _do_build_and_run_instance
2021-03-28 15:13:58.258 465334 ERROR os_vif self._build_and_run_instance(context, instance, image,
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2488, in _build_and_run_instance
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.RescheduledException(
2021-03-28 15:13:58.258 465334 ERROR os_vif nova.exception.RescheduledException: Build of instance bd1a7d09-cf67-406c-8d0a-92256eb1daf8 was re-scheduled: Failure running os_vif plugin plug method: Failed to plug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif During handling of the above exception, another exception occurred:
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.258 465334 ERROR os_vif Traceback (most recent call last):
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/os_vif/__init__.py", line 110, in unplug
2021-03-28 15:13:58.258 465334 ERROR os_vif plugin.unplug(vif, instance_info)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 380, in unplug
2021-03-28 15:13:58.258 465334 ERROR os_vif self._unplug_vf(vif)
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/ovs.py", line 343, in _unplug_vf
2021-03-28 15:13:58.258 465334 ERROR os_vif pf_ifname = linux_net.get_ifname_by_pci_address(
2021-03-28 15:13:58.258 465334 ERROR os_vif File "/usr/lib/python3/dist-packages/vif_plug_ovs/linux_net.py", line 355, in get_ifname_by_pci_address
2021-03-28 15:13:58.258 465334 ERROR os_vif raise exception.PciDeviceNotFoundById(id=pci_addr)
2021-03-28 15:13:58.258 465334 ERROR os_vif vif_plug_ovs.exception.PciDeviceNotFoundById: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.258 465334 ERROR os_vif
2021-03-28 15:13:58.260 465334 WARNING nova.compute.manager [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] [instance: bd1a7d09-cf67-406c-8d0a-92256eb1daf8] Cleaning up VIFs failed for instance. Error: Failure running os_vif plugin unplug method: Failed to unplug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found: nova.exception.InternalError: Failure running os_vif plugin unplug method: Failed to unplug VIF VIFHostDevice(active=False,address=fa:16:3e:10:ae:3a,dev_address=0000:05:10.0,dev_type='ethernet',has_traffic_filtering=True,id=9f7aae43-8e80-44fb-b626-f021423e6063,network=Network(c5ffad64-b4aa-4925-8c6a-31c35a8b8b5c),plugin='ovs',port_profile=VIFPortProfileOVSRepresentor,preserve_on_delete=True). Got error: PCI device 0000:05:10.0 not found
2021-03-28 15:13:58.260 465334 DEBUG nova.compute.manager [req-306e2d84-8d39-4e08-8df2-ba87baa323ae aa48b560d18541489031f10410fe33c7 80088d4e3eea4f90958175e58eb7ddbe - c35b9b534cf14b36802207ec524932e0 c35b9b534cf14b36802207ec524932e0] [instance: bd1a7d09-cf67-406c-8d0a-92256eb1daf8] Deallocating network for instance _deallocate_network /usr/lib/python3/dist-packages/nova/compute/manager.py:2052