OpenStack Compute (nova)

Failure to detach volume during Tempest test cleanup due to APIC related kernel panic within the guest OS

Bug #1939108 reported by Lee Yarwood on 2021-08-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Confirmed	High	Lee Yarwood
	devstack	Fix Released	Undecided	Lee Yarwood

Bug Description

Creating a fresh bug for this as we haven't seen it in a while since the libvirt event rewrite landed but it did just cause a gate failure...

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5d6/801990/6/gate/tempest-integrated-compute/5d6a7ec/testr_results.html

13 Aug 05 16:01:08.964854 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f is timed out.
47114 Aug 05 16:01:08.969793 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Failed to detach device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Libvirt did not re port any error but the device is still in the config. {{(pid=104299) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2394}}
47115 Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. De vice is still attached to the guest.
47116 Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Devi ce detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
47117 Aug 05 16:01:09.024848 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG oslo_concurrency.lockutils [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tem pest-AttachVolumeNegativeTest-1260221999-project] Lock "ce4b444f-1587-43e9-b708-d2ae2c9cd59f" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 201.180s {{(p id=104299) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:367}}
47118 Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 temp est-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.

$ logsearch storedsearch bug-xxx-detach
Running stored search:
  bug-xxx-detach:
    branches:
    - master
    files:
    - controller/logs/screen-n-cpu.txt
    job-groups:
    - nova-devstack
    limit: 100
    project: openstack/nova
    regex: from the live domain config. Device is still attached to the guest.
    result: FAILURE
[..]
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47115:Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47116:Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47118:Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47170:Aug 05 16:01:09.101298 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.

Builds with matching logs 1/100:
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
| uuid | finished | review | pipeline | job |
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
| 5d6a7ec36f104f10a1f49fe44909aa23 | 2021-08-05T16:25:36 | https://review.opendev.org/801990 | gate | tempest-integrated-compute |
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+

Tags:

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-08-18:

Download full text (7.5 KiB)

[    0.000000] Linux version 5.3.0-26-generic (buildd@lgw01-amd64-039) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 (Ubuntu 5.3.0-26.28~18.04.1-generic 5.3.13)
[    0.000000] Command line: LABEL=cirros-rootfs ro console=tty1 console=ttyS0
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai  
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007fdcfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000007fdd000-0x0000000007ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 1.13.0-1ubuntu1.1 04/01/2014
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2394.452 MHz processor
[    0.026481] last_pfn = 0x7fdd max_arch_pfn = 0x400000000
[    0.028543] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.059293] found SMP MP-table at [mem 0x000f5c90-0x000f5c9f]
[    0.080002] check: Scanning 1 areas for low memory corruption
[    0.087928] RAMDISK: [mem 0x0798e000-0x07fccfff]
[    0.088855] ACPI: Early table checksum verification disabled
[    0.089662] ACPI: RSDP 0x00000000000F5AA0 000014 (v00 BOCHS )
[    0.090108] ACPI: RSDT 0x0000000007FE153C 000030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
[    0.091295] ACPI: FACP 0x0000000007FE1418 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
[    0.092403] ACPI: DSDT 0x0000000007FE0040 0013D8 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
[    0.092593] ACPI: FACS 0x0000000007FE0000 000040
[    0.092735] ACPI: APIC 0x0000000007FE148C 000078 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
[    0.092800] ACPI: HPET 0x0000000007FE1504 000038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
[    0.096602] No NUMA configuration found
[    0.096673] Faking a node at [mem 0x0000000000000000-0x0000000007fdcfff]
[    0.097804] NODE_DATA(0) allocated [mem 0x07963000-0x0798dfff]
[    0.104051] Zone ranges:
[    0.104153]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.104229]   DMA32    [mem 0x0000000001000000-0x0000000007fdcfff]
[    0.104250]   Normal   empty
[    0.104279]   Device   empty
[    0.104302] Movable zone start for each node
[    0.104370] Early memory node ranges
[    0.104422]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.104679]   node   0: [mem 0x0000000000100000-0x0000000007fdcfff]
[    0.105220] Zeroed struct page in unavailable ranges: 98 pages
[    0.105486] Initmem setup node 0 [mem 0x0000000000001000-0x0000000007fdcfff]
[    0.113353] ACPI: PM-Timer IO Port: 0x608
[    0.114155] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.114823] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[    0.115040] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.115571] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.115640] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.115877] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.115926] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.116354] Using ACPI (MADT) for SMP configuration information
[    0.116446] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.117087] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.118442] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.118560] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.118604] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.118620] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.118805] [mem 0x08000000-0xfffbffff] available for PCI devices
[    0.118874] Booting paravirtualized kernel on bare hardware
[    0.119527] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.120341] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[    0.132893] percpu: Embedded 54 pages/cpu s184320 r8192 d28672 u2097152
[    0.137042] Built 1 zonelists, mobility grouping on.  Total pages: 32102
[    0.137081] Policy zone: DMA32
[    0.137390] Kernel command line: LABEL=cirros-rootfs ro console=tty1 console=ttyS0
[    0.139937] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.140361] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.142922] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.154216] Memory: 88564K/130540K available (14339K kernel code, 2370K rwdata, 4668K rodata, 2660K init, 5076K bss, 41976K reserved, 0K cma-reserved)
[    0.155470] random: get_random_u64 called from __kmem_cache_create+0x41/0x540 with crng_init=0
[    0.169726] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.172015] ftrace: allocating 42927 entries in 168 pages
[    0.312650] rcu: Hierarchical RCU implementation.
[    0.312817] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
[    0.312948] 	Tasks RCU enabled.
[    0.313023] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.313067] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.345147] NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
[    0.363395] Console: colour VGA+ 80x25
[    0.368516] printk: console [tty1] enabled
[    0.387973] printk: console [ttyS0] enabled
[    0.389502] ACPI: Core revision 20190703
[    0.397050] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.399659] APIC: Switch to symmetric I/O mode setup
[    0.405993] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.473819] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[    0.474106] ...trying to set up timer (IRQ0) through the 8259A ...
[    0.474428] ..... (found apic 0 pin 2) ...
[    0.541777] ....... failed.
[    0.542187] ...trying to set up timer as Virtual Wire IRQ...
[    0.609633] ..... failed.
[    0.609861] ...trying to set up timer as ExtINT IRQ...
[    2.015089] ..... failed :(.
[    2.015512] Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.
[    2.016274] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.3.0-26-generic #28~18.04.1-Ubuntu
[    2.016575] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.13.0-1ubuntu1.1 04/01/2014
[    2.017025] Call Trace:
[    2.018195]  dump_stack+0x6d/0x95
[    2.018429]  panic+0xfe/0x2d4
[    2.018634]  check_timer+0x5e8/0x685
[    2.018799]  ? radix_tree_lookup+0xd/0x10
[    2.019005]  setup_IO_APIC+0x182/0x1ca
[    2.019167]  apic_intr_mode_init+0x1f5/0x1f8
[    2.019351]  x86_late_time_init+0x1b/0x22
[    2.019518]  start_kernel+0x4cb/0x58b
[    2.019725]  x86_64_start_reservations+0x24/0x26
[    2.019901]  x86_64_start_kernel+0x74/0x77
[    2.020095]  secondary_startup_64+0xa4/0xb0
[    2.021474] ---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option. ]---
}}}

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-08-18:

https://lkml.org/lkml/2020/4/2/252

Changed in nova:
status:	New → Confirmed
importance:	Undecided → High
assignee:	nobody → Lee Yarwood (lyarwood)
tags:	added: volumes

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-08-18:

https://review.opendev.org/c/openstack/nova/+/96090
https://bugs.launchpad.net/cirros/+bug/1312199

Lee Yarwood (lyarwood) on 2021-08-18

summary:

- tempest.api.compute.volumes.test_attach_volume_negative.AttachVolumeNegativeTest
- fails due to detach failure during cleanup
+ Failure to detach volume during Tempest test cleanup due to APIC related
+ kernel panic within the guest OS

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-08-18:

Apologies, I forgot to list the example failures in c#1:

https://fd4898d1ad323f562564-d9130fa6474aa8d33290246fdda852b9.ssl.cf2.rackcdn.com/803778/2/gate/tempest-integrated-compute/1645c07/testr_results.html

https://ad8b39086a6983ba968f-dd957a1616adc7212cb23145a9d50168.ssl.cf1.rackcdn.com/764435/6/gate/tempest-integrated-compute/8213711/testr_results.html

Both came up today as openstack/nova gate failures.

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-08-19:

https://github.com/cirros-dev/cirros/issues/69 - raised to see if there's any interest in adding the kernel command line arg by default to Cirros.

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-08-19:

http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024236.html

Lee Yarwood (lyarwood) on 2021-08-20

Changed in devstack:
assignee:	nobody → Lee Yarwood (lyarwood)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-21: Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/766043
Committed: https://opendev.org/openstack/nova/commit/ec48e1523dfdd0f1031e1a70c28c56962db66e8a
Submitter: "Zuul (22348)"
Branch: master

commit ec48e1523dfdd0f1031e1a70c28c56962db66e8a
Author: Sean Mooney <email address hidden>
Date: Tue Dec 8 16:42:52 2020 +0000

workarounds: Add libvirt_disable_apic

This change adds a new workaround config option to allow disabling the
guest apic.

    Co-authored-by: Lee Yarwood <email address hidden>
    Related-Bug: #1939108
    Change-Id: I2da867f2734b590a884b1fe1200c402cbf7e9e1c

Changed in devstack:
status:	New → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-21: Fix merged to devstack (master)

Reviewed: https://review.opendev.org/c/openstack/devstack/+/805219
Committed: https://opendev.org/openstack/devstack/commit/1e86a25cc28e34d7f73a4c6ccbbc3fc667598d50
Submitter: "Zuul (22348)"
Branch: master

commit 1e86a25cc28e34d7f73a4c6ccbbc3fc667598d50
Author: Lee Yarwood <email address hidden>
Date: Thu Aug 19 14:24:28 2021 +0100

nova: Enable apic removal workaround for bug #1939108 when using QEMU

    This change enables [workarounds]libvirt_disable_apic when devstack is
    deployed using the libvirt virt driver and qemu virt type in an effort
    to avoid issues outlined in bug #1939108 caused by the older kernel
    currently used in Cirros 0.5.2.

    Depends-On: https://review.opendev.org/c/openstack/nova/+/766043
    Closes-Bug: #1939108
    Change-Id: Ibb6c34133bb1c95ef11cc59d9b12a0f65502c61b

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Related fix proposed to nova (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/805628

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Related fix proposed to nova (stable/victoria)

#10

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/805629

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Related fix proposed to nova (stable/ussuri)

#11

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/805630

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Related fix proposed to nova (stable/train)

#12

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/805631

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Fix proposed to devstack (stable/wallaby)

#13

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/devstack/+/805632

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Fix proposed to devstack (stable/victoria)

#14

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/devstack/+/805633

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Fix proposed to devstack (stable/ussuri)

#15

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/devstack/+/805634

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-23: Fix proposed to devstack (stable/train)

#16

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/devstack/+/805636

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-09-06:

#17

Still seeing the odd hit of this but nothing like we were a few weeks ago:

$ logsearch storedsearch bug-apic
Running stored search:
  bug-apic:
    branches:
    - master
    files:
    - testr_results.html
    job-groups:
    - nova-devstack
    limit: 100
    project: openstack/nova
    regex: 'Kernel panic - not syncing: IO-APIC'
    result: FAILURE
[..]
Searching logs:
30946b8e90fb4aeaa200d7847b253780:.logsearch/30946b8e90fb4aeaa200d7847b253780/testr_results.html:514:[ 1.834866] Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option.
30946b8e90fb4aeaa200d7847b253780:.logsearch/30946b8e90fb4aeaa200d7847b253780/testr_results.html:529:[ 1.841148] ---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. ]---

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-10-07:

#18

Revision history for this message

Attila Fazekas (afazekas) wrote on 2022-02-09:

#19

Kernel probably has blind spto between pci enumeration and hotplug enable.
Initiation of pci rescan after the device addition can make the device show up.
https://stackoverflow.com/questions/32334870/how-to-do-a-true-rescan-of-pcie-bus

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-02-28: Related fix merged to nova (stable/wallaby)

#20

Reviewed: https://review.opendev.org/c/openstack/nova/+/805628
Committed: https://opendev.org/openstack/nova/commit/80572cef67d691ff3f2de50ea2214f0d0c4725c4
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 80572cef67d691ff3f2de50ea2214f0d0c4725c4
Author: Sean Mooney <email address hidden>
Date: Tue Dec 8 16:42:52 2020 +0000

workarounds: Add libvirt_disable_apic

This change adds a new workaround config option to allow disabling the
guest apic.

    NOTE(lyarwood): Conflict caused by I66a8e94f6d04cc89785646aa9c3e138f22a51f60
    not being present in stable/wallaby (test_driver.py), and
    Ie904d1513b5cf76d6d5f6877545e8eb378dd5499 merged in different order in
    different branches (workarounds.py).

    Conflicts:
        nova/tests/unit/virt/libvirt/test_driver.py
        nova/conf/workarounds.py

    Co-authored-by: Lee Yarwood <email address hidden>
    Related-Bug: #1939108
    Change-Id: I2da867f2734b590a884b1fe1200c402cbf7e9e1c
    (cherry picked from commit ec48e1523dfdd0f1031e1a70c28c56962db66e8a)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-03-02: Fix merged to devstack (stable/wallaby)

#21

Reviewed: https://review.opendev.org/c/openstack/devstack/+/805632
Committed: https://opendev.org/openstack/devstack/commit/566896e22cd70b9c946d05a68106db8c4fd821b7
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 566896e22cd70b9c946d05a68106db8c4fd821b7
Author: Lee Yarwood <email address hidden>
Date: Thu Aug 19 14:24:28 2021 +0100

nova: Enable apic removal workaround for bug #1939108 when using QEMU

    Depends-On: https://review.opendev.org/c/openstack/nova/+/805628
    Closes-Bug: #1939108
    Change-Id: Ibb6c34133bb1c95ef11cc59d9b12a0f65502c61b
    (cherry picked from commit 1e86a25cc28e34d7f73a4c6ccbbc3fc667598d50)

Revision history for this message

LIU Yulong (dragon889) wrote on 2022-07-12:

#22

Download full text (13.5 KiB)

Hi there:

We got same error after disable APIC for upstream cloud image: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-2111.qcow2

OpenStack version: rocky

host kernel: 3.10.0-1160.53.1.el7.x86_64

VM console log:
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.10.0-1160.45.1.el7.x86_64 (<email address hidden>) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Oct 13 17:20:51 UTC 2021
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.45.1.el7.x86_64 root=UUID=44a6a613-4e21-478b-a909-ab653c9d39df ro console=tty0 console=ttyS0,115200n8 crashkernel=auto net.ifnames=0 console=ttyS0 LANG=en_US.UTF-8
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffdbfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007ffdc000-0x000000007fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 1.11.0-2.el7 04/01/2014
[ 0.000000] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x400000000
[ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC
[ 0.000000] found SMP MP-table at [mem 0x000f63a0-0x000f63af] mapped at [ffffffffff2003a0]
[ 0.000000] RAMDISK: [mem 0x32419000-0x35204fff]
[ 0.000000] Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 00000000000f6170 00014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 000000007ffe14f7 0002C (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 000000007ffe140b 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 000000007ffe0040 013CB (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACS 000000007ffe0000 00040
[ 0.000000] ACPI: APIC 000000007ffe147f 00078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007ffdbfff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x7ffb5000-0x7ffdbfff]
[ 0.000000] Reserving 161MB of memory at 640MB for crashkernel (System RAM: 2047MB)
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00001000-0x0009efff]
[ 0.000000] node 0: [mem 0x00100000-0x7ffdbfff]
[ 0.000000] Initmem setup node 0 [mem 0x00001000-0x7ffdbfff]
[ 0.000000] ACPI: PM-Timer IO Port: 0x608
[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.000000] ACPI: IOAPIC (id[0x00] ad...

Hi there:

We got same error after disable APIC for upstream cloud image: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-2111.qcow2

OpenStack version: rocky

host kernel: 3.10.0-1160.53.1.el7.x86_64

VM console log:
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-1160.45.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Oct 13 17:20:51 UTC 2021
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.45.1.el7.x86_64 root=UUID=44a6a613-4e21-478b-a909-ab653c9d39df ro console=tty0 console=ttyS0,115200n8 crashkernel=auto net.ifnames=0 console=ttyS0 LANG=en_US.UTF-8
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffdbfff] usable
[    0.000000] BIOS-e820: [mem 0x000000007ffdc000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 1.11.0-2.el7 04/01/2014
[    0.000000] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x400000000
[    0.000000] PAT configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- UC  
[    0.000000] found SMP MP-table at [mem 0x000f63a0-0x000f63af] mapped at [ffffffffff2003a0]
[    0.000000] RAMDISK: [mem 0x32419000-0x35204fff]
[    0.000000] Early table checksum verification disabled
[    0.000000] ACPI: RSDP 00000000000f6170 00014 (v00 BOCHS )
[    0.000000] ACPI: RSDT 000000007ffe14f7 0002C (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
[    0.000000] ACPI: FACP 000000007ffe140b 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
[    0.000000] ACPI: DSDT 000000007ffe0040 013CB (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
[    0.000000] ACPI: FACS 000000007ffe0000 00040
[    0.000000] ACPI: APIC 000000007ffe147f 00078 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000007ffdbfff]
[    0.000000] NODE_DATA(0) allocated [mem 0x7ffb5000-0x7ffdbfff]
[    0.000000] Reserving 161MB of memory at 640MB for crashkernel (System RAM: 2047MB)
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009efff]
[    0.000000]   node   0: [mem 0x00100000-0x7ffdbfff]
[    0.000000] Initmem setup node 0 [mem 0x00001000-0x7ffdbfff]
[    0.000000] ACPI: PM-Timer IO Port: 0x608
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.000000] e820: [mem 0x80000000-0xfffbffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] Detected CPU family 6h model 13
[    0.000000] Warning: AMD Processor - this hardware has not undergone upstream testing. Please consult http://wiki.centos.org/FAQ for more information
[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] percpu: Embedded 38 pages/cpu s118784 r8192 d28672 u2097152
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 515941
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.45.1.el7.x86_64 root=UUID=44a6a613-4e21-478b-a909-ab653c9d39df ro console=tty0 console=ttyS0,115200n8 crashkernel=auto net.ifnames=0 console=ttyS0 LANG=en_US.UTF-8
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Memory: 1832372k/2097008k available (7796k kernel code, 392k absent, 264244k reserved, 5947k data, 1980k init)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.
[    0.000000] NR_IRQS:327936 nr_irqs:256 0
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [ttyS0] enabled
[    0.000000] allocated 16777216 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] tsc: Fast TSC calibration failed
[    0.000000] tsc: Unable to calibrate against PIT
[    0.000000] tsc: using PMTIMER reference calibration
[    0.000000] tsc: Detected 3696.055 MHz processor
[    0.005000] Calibrating delay loop (skipped), value calculated using timer frequency.. 7392.11 BogoMIPS (lpj=3696055)
[    0.005000] pid_max: default: 32768 minimum: 301
[    0.005000] Security Framework initialized
[    0.005000] SELinux:  Initializing.
[    0.005000] Yama: becoming mindful.
[    0.005000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.006000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.006000] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.006000] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.007000] Initializing cgroup subsys memory
[    0.007000] Initializing cgroup subsys devices
[    0.007000] Initializing cgroup subsys freezer
[    0.007100] Initializing cgroup subsys net_cls
[    0.007370] Initializing cgroup subsys blkio
[    0.007618] Initializing cgroup subsys perf_event
[    0.007922] Initializing cgroup subsys hugetlb
[    0.008000] Initializing cgroup subsys pids
[    0.008000] Initializing cgroup subsys net_prio
[    0.009000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[    0.009000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0
[    0.009000] tlb_flushall_shift: -1
[    0.009000] FEATURE SPEC_CTRL Not Present
[    0.009000] FEATURE IBPB_SUPPORT Not Present
[    0.009000] Spectre V1 : Mitigation: Load fences, usercopy/swapgs barriers and __user pointer sanitization
[    0.010000] Spectre V2 : Vulnerable: Retpoline without IBPB
[    0.010000] Speculative Store Bypass: Vulnerable
[    0.039000] Freeing SMP alternatives: 28k freed
[    0.040000] ACPI: Core revision 20130517
[    0.043000] ACPI: All ACPI Tables successfully acquired
[    0.043000] ftrace: allocating 29688 entries in 116 pages
[    0.049000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.051000] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[    0.051000] ...trying to set up timer (IRQ0) through the 8259A ...
[    0.051000] ..... (found apic 0 pin 2) ...
[    0.052000] ....... failed.
[    0.052000] ...trying to set up timer as Virtual Wire IRQ...
[    0.054000] ..... failed.
[    0.054000] ...trying to set up timer as ExtINT IRQ...
[    0.057000] ..... failed :(.
[    0.057000] Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.
[    0.057000] 
[    0.057000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-1160.45.1.el7.x86_64 #1
[    0.057000] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.11.0-2.el7 04/01/2014
[    0.057000] Call Trace:
[    0.057000]  [<ffffffff9b583539>] dump_stack+0x19/0x1b
[    0.057000]  [<ffffffff9b57d241>] panic+0xe8/0x21f
[    0.057000]  [<ffffffff9bba4e89>] check_timer+0x454/0x625
[    0.057000]  [<ffffffff9bba5883>] setup_IO_APIC+0x27f/0x2a0
[    0.057000]  [<ffffffff9bba12ca>] native_smp_prepare_cpus+0x342/0x3cc
[    0.057000]  [<ffffffff9bb8b2d8>] kernel_init_freeable+0xe8/0x249
[    0.057000]  [<ffffffff9b572330>] ? rest_init+0x80/0x80
[    0.057000]  [<ffffffff9b57233e>] kernel_init+0xe/0x100
[    0.057000]  [<ffffffff9b595de4>] ret_from_fork_nospec_begin+0xe/0x21
[    0.057000]  [<ffffffff9b572330>] ? rest_init+0x80/0x80

VM XML:

<domain type='qemu' id='14'>
  <name>instance-0000004c</name>
  <uuid>dd7294d2-f226-4a5c-901d-9d13d1c090f5</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="18.2.4"/>
      <nova:name>centos7-vm-2</nova:name>
      <nova:creationTime>2022-07-12 10:51:32</nova:creationTime>
      <nova:flavor name="max2">
        <nova:memory>2048</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="37ca630955904090b7ea59748b180a20">admin</nova:user>
        <nova:project uuid="8e957fe9c73c41b08252bc5943c195fa">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="2eb774b2-19fd-4dab-9040-7283c85c0a08"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>18.2.4</entry>
      <entry name='serial'>6f3f861f-f232-b744-b136-aae4c63cea49</entry>
      <entry name='uuid'>dd7294d2-f226-4a5c-901d-9d13d1c090f5</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/opt/stack/data/nova/instances/dd7294d2-f226-4a5c-901d-9d13d1c090f5/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/opt/stack/data/nova/instances/_base/f545bc754fcefe3a6762fdc4ef415b970679bf94'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:55:8f:bc'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='55e16efe-848e-4438-bd38-594e1d52a2d2'/>
      </virtualport>
      <target dev='tap55e16efe-84'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <mtu size='1500'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/10'/>
      <log file='/opt/stack/data/nova/instances/dd7294d2-f226-4a5c-901d-9d13d1c090f5/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/10'>
      <source path='/dev/pts/10'/>
      <log file='/opt/stack/data/nova/instances/dd7294d2-f226-4a5c-901d-9d13d1c090f5/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5908' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

Revision history for this message

norman shen (jshen28) wrote on 2023-02-08:

#23

Hi, what qemu version is used? Is there any reliable way to reproduce it? thank you.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-01: Change abandoned on nova (stable/train)

#24

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/805631
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.