lxc c2-m2 focal VM causes KVM internal error during PCI init

I attached gdb to the VM[*] and retrieved the following backtrace from the guest kernel:

(gdb) bt full
#0 pci_conf1_read (seg=<optimized out>, bus=<optimized out>, devfn=<optimized out>, reg=16, len=4, value=0xffffc9000001fa0c) at /home/dannf/linux-kvm-5.4.0/arch/x86/pci/direct.c:35
        flags = 582
        __dummy = <optimized out>
        __dummy2 = <optimized out>
        __dummy = <optimized out>
        __dummy2 = <optimized out>
#1 0xffffffff815a1964 in raw_pci_read (domain=<optimized out>, bus=<optimized out>, devfn=<optimized out>, reg=<optimized out>, len=<optimized out>, val=<optimized out>) at /home/dannf/linux-kvm-5.4.0/arch/x86/pci/common.c:46
No locals.
#2 0xffffffff815a19a7 in pci_read (bus=<optimized out>, devfn=<optimized out>, where=<optimized out>, size=<optimized out>, value=<optimized out>) at /home/dannf/linux-kvm-5.4.0/arch/x86/include/asm/pci.h:45
No locals.
#3 0xffffffff8144f4c5 in pci_bus_read_config_dword (bus=<optimized out>, devfn=<optimized out>, pos=<optimized out>, value=0xffffc9000001fa58) at /home/dannf/linux-kvm-5.4.0/drivers/pci/access.c:65
        res = -2147480816
        flags = <optimized out>
        data = 0
#4 0xffffffff8144f7f2 in pci_read_config_dword (dev=<optimized out>, where=<optimized out>, val=<optimized out>) at /home/dannf/linux-kvm-5.4.0/drivers/pci/access.c:550
No locals.
#5 0xffffffff8145142a in __pci_read_base (dev=0xffff888000d8d000, type=pci_bar_unknown, res=0xffff888000d8d278, pos=16) at /home/dannf/linux-kvm-5.4.0/drivers/pci/probe.c:196
        l = 0
        sz = 0
        mask = <optimized out>
        l64 = <optimized out>
        sz64 = <optimized out>
        mask64 = <optimized out>
        orig_cmd = 7
        region = {start = 18446683600570153584, end = 18446744071584749924}
        inverted_region = {start = 18446683600570153600, end = 18446744071584749991}
#6 0xffffffff81451757 in pci_read_bases (dev=0xffff888000d8d000, howmany=2, rom=56) at /home/dannf/linux-kvm-5.4.0/drivers/pci/probe.c:334
        res = <optimized out>
        pos = 0
        reg = <optimized out>
        res = <optimized out>
#7 0xffffffff81451dd4 in pci_setup_device (dev=0xffff888000d8d000) at /home/dannf/linux-kvm-5.4.0/drivers/pci/probe.c:1854
        class = 1540
        cmd = 34750
        hdr_type = <optimized out>
        pos = <optimized out>
        region = {start = 18446683600670950407, end = 18446744071583369946}
        res = <optimized out>
#8 0xffffffff81452950 in pci_scan_device (devfn=<optimized out>, bus=<optimized out>) at /home/dannf/linux-kvm-5.4.0/drivers/pci/probe.c:2301
        dev = 0xffff888000d8d000
        l = 793398
        dev = <optimized out>
        l = <optimized out>
#9 pci_scan_single_device (devfn=<optimized out>, bus=<optimized out>) at /home/dannf/linux-kvm-5.4.0/drivers/pci/probe.c:2474
        dev = <optimized out>
        dev = <optimized out>
#10 pci_scan_single_device (bus=0xffff888000c76c00, devfn=11) at /home/dannf/linux-kvm-5.4.0/drivers/pci/probe.c:2464
        dev = 0x0 <fixed_percpu_data>
#11 0xffffffff81452a02 in pci_scan_slot (bus=0xffff888000c76c00, devfn=8) at /home/dannf/linux-kvm-5.4.0/drivers/pci/probe.c:2560
        fn = 3
       ...

I filed this issue on the kernel because it initially appeared to follow the host kernel version. While it's very reproducible w/ v5.4, I wasn't able to reproduce it with v5.8 or v5.7. However, I've now found that was a false negative - it is still reproducible with v5.13, so the kernel may not be the correct component.

I got back to this a couple days ago, here's what I've learned.

I tried decoding the QEMU syndrome - but be warned, I've never done this before. I believe what it is reporting is an EPT[*] misconfiguration.

I noticed that the 5.4 -generic kernel does not reproduce the issue, while the -kvm one does. I tried comparing the configs, but there's really nothing obvious there. Disabling DYNAMIC_FTRACE did seem to cause the -generic kernel to fail, but turning it on did not fix the -kvm kernel, so that seems like a red herring.

I noticed that newer guest kernels do not seem to reproduce the problem. I bisected the guest kernels and hit this commit:

  f1d4d47c5851 x86/setup: Always reserve the first 1M of RAM

This was introduced upstream in v5.13 and appears to be a workaround for BIOSes that corrupt memory under 1M. By default, both the -generic and -kvm kernels build with CONFIG_X86_RESERVE_LOW=64, which avoids using the first 64K of memory. So this suggests that something might be getting corrupted in the 64K->1M region. There's also a kernel parameter you can use to modify this called "reservelow", and if I add "reservelow=1024k" to the cmdline, the -kvm kernel no longer crashes.

It seems like some kind of corruption maybe going on - possibly by edk2/ovmf. As a next step I think I'll try to figure out if I can run the VM under GDB and have it trap writes to that memory area.

[*] https://en.wikipedia.org/wiki/Second_Level_Address_Translation#Extended_Page_Tables

Adding linux-kvm to the bug. It looks like if we can have the commit above backported, it would take care of this issue for most users.

fyi, I do recall trying what I suggested in Comment #3, trying to debug under GDB. However, I didn't find a way to trap writes to that region, and didn't have any other good ideas.