32bit pci domain number is not supported

The libvirt fix[2] landed in libvirt 5.7.0. Do we need / do we have similar fix in qemu as well?

If both qemu and libvirt supports 32bit PCI domains then I don't see any reason why nova should reject 32bit PCI domains.

Could you please confirm that QEMU works with these 32bit PCI domains? I'm setting this bug to Incomplete until the QEMU support is confirmed.

[2] https://gitlab.com/libvirt/libvirt/-/commit/d19c21429fd1acfdade23c903804d6e279d1e00e

Looking at the QEMU code it seems to me that QEMU would simply reject PCI devices with domain bigger than 0xFFFF [1]. I'm not sure if this limitation also valid for QEMU with accel=kvm.

[1] https://github.com/qemu/qemu/blob/f2a1cf9180f63e88bb38ff21c169da97c3f2bad5/hw/core/qdev-properties.c#L993

Thanks for the confirmation.

I tried it by actually running qemu with accel=kvm.
On my environment (linux 3.10.0-957.27.2.el7.x86_64), the vfio-pci driver
failed to bind the 32bit PCI domain device before booting the VM. So, it
seems to me that there are a few other things we need to check besides
QEMU to assign 32bit PCI domain device to the VM. Please let me know if
the procedure is wrong.

Anyway, if nova doesn't support the 32bit PCI domain number,
the nova-compute service itself won't be able to start,
regardless of whether we actually use 32bit PCI number device or not.
So, I'd like to send a patch if possible.

# 16bit PCI domain number, booted successfully
$ sudo qemu-system-x86_64 -machine pc-i440fx-2.11,accel=kvm -nographic \
  -device vfio-pci,host=0000:5e:02.5 cirros-0.5.1-x86_64-disk.img

# 32bit PCI domain number, failed to boot
$ dmesg
# vfio-pci: probe of 10000:01:00.0 failed with error -22
$ sudo qemu-system-x86_64 -machine pc-i440fx-2.11,accel=kvm -nographic \
  -device vfio-pci,host=10000:01:00.0 cirros-0.5.1-x86_64-disk.img

Thanks for confirming that qemu also fails with such PCI device.

Does nova only fail to start because you added a 32bit PCI address to the whitelist?

I'm a bit reluctant to allow adding 32bit PCI addresses in the whitelist as it suggest that deployment support it, but right now qemu will fail when you want to boot a VM with such device. So I think first we have to make sure that there is end-to-end support for 32 bit addresses and then we can enable the use of them in nova.

This issue is reproduced when only 16bit PCI domain address is registered
in pci.passthrough_whitelist. Note that it is not reproduced if the
pci.passthrough_whitelist is empty.

So I think there may be a bug in the PCI domain address matching part.

2020-10-08 11:23:13.316 10735 DEBUG oslo_service.service [req-bef9c36c-53c2-45ef-9b8e-4e3f602172cb - - - - -] pci.alias = [] log_opt_values /opt/nova/lib/python2.7/site-packages/oslo_config/cfg.py:3027
2020-10-08 11:23:13.317 10735 DEBUG oslo_service.service [req-bef9c36c-53c2-45ef-9b8e-4e3f602172cb - - - - -] pci.passthrough_whitelist = ['{ "address": "0000:5e:01.5" }'] log_opt_values /opt/nova/lib/python2.7/site-packages/oslo_config/cfg.py:3027
2020-10-08 11:23:14.142 10735 ERROR nova.compute.manager [req-e22b43ef-0f19-4ba1-9711-87a7ea9ca8f1 - - - - -] Error updating resources for node XXXXXX.jp.: PciConfigInvalidWhitelist: Invalid PCI devices Whitelist config: property domain (10000) is greater than the maximum allowable value (FFFF).

Now that is interesting. So in a system where there are devices with 32bit PCI domain but they are not listed in the passthrough_whitelist nova still fails. I'm setting this back to New. I think it would make sense to try to create a reproduction unit test. My feeling is that we iterate all the host devices when filtering with the whitelist and there we encounter the 32bit domain.

I tested my above assumption and it seems it is the root of the problem.

The nova.pci.devspec.WhitelistPciAddress.match()[1] is called for every PCI device on the host at startup, including the ones with 32bit domain. And this reuses the nova.pci.devspec.PciAddressSpec._set_pci_dev_info() utility with maxval=0xFFFF for domains. So I can confirm that if the host has a PCI device with 32bit domain then nova compute will refuse to start. If the passthrough_whitelist config is empty then there is a shortcut in the startup sequence[3] skipping the iteration of the PCI devices on the host hence the problem disappear.

Marking this bug Confirmed. I will try to push a fix soon.

[1] https://github.com/openstack/nova/blob/2745e685376abbc4c32516837f6074a3de23aa24/nova/pci/devspec.py#L217
[2] https://github.com/openstack/nova/blob/2745e685376abbc4c32516837f6074a3de23aa24/nova/pci/devspec.py#L51
[3] https://github.com/openstack/nova/blob/2745e685376abbc4c32516837f6074a3de23aa24/nova/compute/manager.py#L1393

To be precise the shortcut I mentioned in comment #7 is a shortcut because it creates 0 device specification and when the host devices are iterated it is matched against the existing specs. If no spec exists there is no attempt to match and therefore no PCI address parsing happen.

Fix proposed to branch: master
Review: https://review.opendev.org/756696

Fix proposed to branch: master
Review: https://review.opendev.org/756697

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/791767

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/791768

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/791770

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/791771

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/792116

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/792117

Reviewed: https://review.opendev.org/c/openstack/nova/+/791767
Committed: https://opendev.org/openstack/nova/commit/0354d4d9f47354e2b4fc0b2343c27e734fe2e494
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 0354d4d9f47354e2b4fc0b2343c27e734fe2e494
Author: Balazs Gibizer <email address hidden>
Date: Thu Oct 8 14:13:38 2020 +0200

    Reproduce bug 1897528

    The nova-compute fails to start if the hypervisor has PCI addresses
    32bit domain.

    Change-Id: I48dcb7faa17fe9f8346445a1746cff5845baf358
    Related-Bug: #1897528
    (cherry picked from commit 976ac722d36439d16ea4ec1bf5037c482c89ef55)

Reviewed: https://review.opendev.org/c/openstack/nova/+/791768
Committed: https://opendev.org/openstack/nova/commit/90ffc553d7f4152a6a4a8708787150d3c3c40b03
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 90ffc553d7f4152a6a4a8708787150d3c3c40b03
Author: Balazs Gibizer <email address hidden>
Date: Thu Oct 8 14:27:44 2020 +0200

    Ignore PCI devices with 32bit domain

    Nova and QEMU[1] supports PCI devices with a PCI address that has 16 bit
    domain. However there are hypervisors that reports PCI addresses with
    32 bit domain. While today we cannot assign these to guests this should
    not prevent the nova-compute service to start.

    This patch changes the PCI manager to ignore such PCI devices.

    Please note that this patch does not change fact that nova does not
    allow specifying PCI addresses with 32bit domain in the
    [pci]/passthrough_whitelist configuration. Such configuration is still
    rejected at nova-compute service startup.

    Closes-Bug: #1897528

    [1] https://github.com/qemu/qemu/blob/f2a1cf9180f63e88bb38ff21c169da97c3f2bad5/hw/core/qdev-properties.c#L993

    Change-Id: I59a0746b864610b6a314078cf5661d3d2b84b1d4
    (cherry picked from commit 8c9d6fc8f073cde78b79ae259c9915216f5d59b0)

Reviewed: https://review.opendev.org/c/openstack/nova/+/791770
Committed: https://opendev.org/openstack/nova/commit/8e9859b95c537738d97ade41a8d09670de27ef8d
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 8e9859b95c537738d97ade41a8d09670de27ef8d
Author: Balazs Gibizer <email address hidden>
Date: Thu Oct 8 14:13:38 2020 +0200

    Reproduce bug 1897528

    The nova-compute fails to start if the hypervisor has PCI addresses
    32bit domain.

    Change-Id: I48dcb7faa17fe9f8346445a1746cff5845baf358
    Related-Bug: #1897528
    (cherry picked from commit 976ac722d36439d16ea4ec1bf5037c482c89ef55)
    (cherry picked from commit 0354d4d9f47354e2b4fc0b2343c27e734fe2e494)

Reviewed: https://review.opendev.org/c/openstack/nova/+/791771
Committed: https://opendev.org/openstack/nova/commit/22daca3e7f33e89b937f03a89db63cf25489e3c9
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 22daca3e7f33e89b937f03a89db63cf25489e3c9
Author: Balazs Gibizer <email address hidden>
Date: Thu Oct 8 14:27:44 2020 +0200

    Ignore PCI devices with 32bit domain

    Nova and QEMU[1] supports PCI devices with a PCI address that has 16 bit
    domain. However there are hypervisors that reports PCI addresses with
    32 bit domain. While today we cannot assign these to guests this should
    not prevent the nova-compute service to start.

    This patch changes the PCI manager to ignore such PCI devices.

    Please note that this patch does not change fact that nova does not
    allow specifying PCI addresses with 32bit domain in the
    [pci]/passthrough_whitelist configuration. Such configuration is still
    rejected at nova-compute service startup.

    Changes:
      nova/pci/manager.py

    NOTE(stephenfin): We need to replace a use of 'str(exc)' with
    'six.text_type(exc)' to keep pep8/Python 2.7 happy.

    Closes-Bug: #1897528

    [1] https://github.com/qemu/qemu/blob/f2a1cf9180f63e88bb38ff21c169da97c3f2bad5/hw/core/qdev-properties.c#L993

    Change-Id: I59a0746b864610b6a314078cf5661d3d2b84b1d4
    (cherry picked from commit 8c9d6fc8f073cde78b79ae259c9915216f5d59b0)
    (cherry picked from commit 90ffc553d7f4152a6a4a8708787150d3c3c40b03)

This issue was fixed in the openstack/nova 22.2.2 release.

This issue was fixed in the openstack/nova 21.2.2 release.

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/816656

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/816682

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/816682
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/816656
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/792116
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/792117
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.