failure to migrate virtual machines with pc-i440fx-wily type to ubuntu 20.04

Forgot to mention that we managed to get around this bug by creating a libvirt hook that checks XML during migration and adds following arguments to qemu process for pc-i440fx-wily guests:

-global virtio-balloon-pci.any_layout=on -global virtio-serial-pci.any_layout=on

Thanks for this good report!
And also thanks for providing the workaround for anyone else running into this.

The old types should (tm) not change due to the compat layers that on any change ensure that former types have the old values set. That is an issue within qemu.

I don't see yet why that would be specific to windows guests, but maybe their drivers trigger something Linux wont.

P.S. I know you now got it migrated, still I'd strongly encourage to bump to a more recent machine type at e.g. a planned service downtime window or such. You just don't want to carry things forever.

I spawned "pc-i440fx-wily" (which is actually xenial in very early days due to a bug back then) guests on Bionic and Focal.

New types like Bionic and Focal all have "any_layout = true" in all slots.
Which is reasonable as they all are post 2.3 where this was later introduced as new default.

Defining the type to "pc-i440fx-wily" on Bionic and Focal shows the issue as reported:

Bionic:
      dev: virtio-balloon-pci, id "balloon0"
        bus: virtio-bus
          dev: virtio-balloon-device, id ""
            any_layout = true
Focal:
      dev: virtio-balloon-pci, id "balloon0"
        bus: virtio-bus
          dev: virtio-balloon-device, id ""
            any_layout = false

As already reported the hw_compat_2_3 should disable that.
But since the wily/xenial confusion was exactly that maybe something was lost there when porting to newer code. There was a change how types got handled later on which might have been the source of this.

The history of the odd Wily type is in these bugs:
- bug 1621042 Xenial used a wily type
- bug 1829868 wily type was a hybrid of 2.4 / 2.3 types

Therefore the "modern" pc_i440fx_wily_machine_options use this mix to reflect that:
 + pc_i440fx_2_4_machine_options(m);
 + m->hw_version = "2.4.0";
 + pcmc->broken_reserved_end = true;
 + compat_props_add(m->compat_props, hw_compat_2_3, hw_compat_2_3_len);
 + compat_props_add(m->compat_props, pc_compat_2_3, pc_compat_2_3_len);

Seems like you have found a new incarnation of this old bug (from before my time here even) to continue to haunt us *sigh*.

I have checked further other than the error suggests it isn't console but "virtio-balloon-pci" that changes and that is indeed known to be rather different in windows - that could explain why they behave different.

The following minimal steps can be used to test:

$ cat > layout-test.xml << EOF
<domain type='kvm'>
  <name>layout-test</name>
  <memory unit='MiB'>256</memory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-wily'>hvm</type>
    <boot dev='hd'/>
  </os>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>
EOF
$ virsh define layout-test.xml
$ virsh start layout-test
$ virsh qemu-monitor-command layout-test --hmp info qtree | grep any_layout
            any_layout = false

With that you can easily get the result that used to be true on Xenial.

Xenial = true
Bionic = true
C/D no more available
Eoan = false
Focal = false

BTW all of this is doable without libvirt via:
$ qemu-system-x86_64 -m 128 -M pc-i440fx-wily,accel=kvm --nodefaults --nographic --monitor stdio -device virtio-balloon-pci,id=balloon0
(qemu) info qtree

You can find the same "any_layout = false" at the device
      dev: virtio-balloon-pci, id "balloon0"
            any_layout = false

The type does:
1197 compat_props_add(m->compat_props, hw_compat_2_3, hw_compat_2_3_len);

Defined as:
 198 GlobalProperty hw_compat_2_3[] = {
 199 { "virtio-blk-pci", "any_layout", "off" },
 200 { "virtio-balloon-pci", "any_layout", "off" },
 201 { "virtio-serial-pci", "any_layout", "off" },
 202 { "virtio-9p-pci", "any_layout", "off" },
 203 { "virtio-rng-pci", "any_layout", "off" },
 204 { TYPE_PCI_DEVICE, "x-pcie-lnksta-dllla", "off" },
 205 { "migration", "send-configuration", "off" },
 206 { "migration", "send-section-footer", "off" },
 207 { "migration", "store-global-state", "off" },
 208 };
 209 const size_t hw_compat_2_3_len = G_N_ELEMENTS(hw_compat_2_3);

311 static inline void
312 compat_props_add(GPtrArray *arr,
313 GlobalProperty props[], size_t nelem)
314 {
315 int i;
316 for (i = 0; i < nelem; i++) {
317 g_ptr_array_add(arr, (void *)&props[i]);
318 }
319 }

So in debug on the init of the wily type we should see
   { "virtio-balloon-pci", "any_layout", "off" }
be added.

$ apt install qemu-system-x86-dbgsym
$ apt source qemu
$ cd qemu-4.2
$ gdb /usr/bin/qemu-system-x86_64
(gdb) b compat_props_add
Breakpoint 1 at 0x4aadd1: compat_props_add. (83 locations)
(gdb) b pc_i440fx_wily_machine_options
Breakpoint 2 at 0x4acdcb: file ./hw/i386/pc_piix.c, line 1198.
(gdb) run -m 128 -M pc-i440fx-wily,accel=kvm --nodefaults --nographic --monitor stdio -device virtio-balloon-pci,id=balloon0

Unfortunately on a normal build these functions like pc_i440fx_wily_machine_options and compat_props_add are very much inlined and arguments optimized out - so we might need a debug build to be able to track where this is lost more easily.

At init of pc_i440fx_wily_machine_options the referred compat is ok:

compat_props_add(m->compat_props, hw_compat_2_3, hw_compat_2_3_len);

It contains any_layout off for virtio-balloon-pci

(gdb) p hw_compat_2_3[1]
$7 = {driver = 0x555555ea725c "virtio-balloon-pci", property = 0x555555ea2c91 "any_layout", value = 0x555555ee6853 "off", used = false, optional = false}

Prior to applying 2_3 115-120 are the last entries:
(gdb) p (m->compat_props).pdata[115]
$58 = (gpointer) 0x5555564d0400 <hw_compat_2_4>
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[115])
$61 = {driver = 0x555555e93d89 "virtio-blk-device", property = 0x555555e9cb8f "scsi", value = 0x555555eece85 "true", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[116])
$62 = {driver = 0x555555ea8ffe "e1000", property = 0x555555ee7f21 "extra_mac_registers", value = 0x555555ee6853 "off", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[117])
$63 = {driver = 0x555555e978cb "virtio-pci", property = 0x555555ee7f35 "x-disable-pcie", value = 0x555555eeb3e5 "on", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[118])
$64 = {driver = 0x555555e978cb "virtio-pci", property = 0x555555ee7f44 "migrate-extra", value = 0x555555ee6853 "off", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[119])
$65 = {driver = 0x555555ee7f52 "fw_cfg_mem", property = 0x555555eb0b55 "dma_enabled", value = 0x555555ee6853 "off", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[120])
$66 = {driver = 0x555555ee7f5d "fw_cfg_io", property = 0x555555eb0b55 "dma_enabled", value = 0x555555ee6853 "off", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[121])
Cannot access memory at address 0x0

Eventually [137] will be the last entry of pc_compat_2_4 that was added.

Then hw_compat_2_3 is added.
Thread 1 "qemu-system-x86" hit Breakpoint 2, compat_props_add (nelem=9, props=<optimized out>, arr=0x5555566121a0) at ./include/hw/qdev-core.h:289

After that we have:

(gdb) p (m->compat_props).pdata[138]
$89 = (gpointer) 0x5555564d02e0 <hw_compat_2_3>
...
(gdb) p (m->compat_props).pdata[146]
$93 = (gpointer) 0x5555564d03e0 <hw_compat_2_3+256>
(gdb) p (m->compat_props).pdata[147]
$94 = (gpointer) 0x0

And content:
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[138])
$96 = {driver = 0x555555ea755c "virtio-blk-pci", property = 0x555555ea2c91 "any_layout", value = 0x555555ee6853 "off", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[139])
$97 = {driver = 0x555555ea725c "virtio-balloon-pci", property = 0x555555ea2c91 "any_layout", value = 0x555555ee6853 "off", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[140])
$98 = {driver = 0x555555ea778a "virtio-serial-pci", property = 0x555555ea2c91 "any_layout", value = 0x555555ee6853 "off", used = false, optional = false}
(gdb) p (*(GlobalProperty*)(m->compat_props).pdata[141])
$99 = {driver = 0x555555ea735a "virtio-9p-pci", property = 0x555555ea2c91 "any_layout", value = 0x555555ee6853 "off", used = false, optional = false}
...

$ gdb /usr/bin/qemu-system-x86_64
(gdb) run -m 128 -M pc-i440fx-wily,accel=kvm --nodefaults --nographic --monitor stdio -device virtio-balloon-pci,id=balloon0
(gdb) handle SIGUSR1 noprint nostop

/*
 * Global property defaults
 * Slot 0: accelerator's global property defaults
 * Slot 1: machine's global property defaults
 * Each is a GPtrArray of of GlobalProperty.
 * Applied in order, later entries override earlier ones.
 */
static GPtrArray *object_compat_props[2];

^^^ this is where the compat data lands on init with a wily type machine.

Thread 1 "qemu-system-x86" hit Breakpoint 1, qemu_init_main_loop (errp=0x7fffffffe468) at ./util/main-loop.c:148
148 {
(gdb) p (*(GlobalProperty*)(object_compat_props[1]).pdata[139])
$17 = {driver = 0x555555ea725c "virtio-balloon-pci", property = 0x555555ea2c91 "any_layout", value = 0x555555ee6853 "off", used = false, optional = false}

So it got into the type correctly.

Oh I forgot - false is actually the value we thought to be wrong according to the report.
But things seem right to be "false" by just looking at the Focal code.

I think I need to look at the very same in Bionic where it should end up being "true" according to the tests done so far.

Problem:
- initial broken as HW_COMPAT_2_4 + pc_compat_2_3
- then some options moved between HW_COMPAT_2_4 <-> pc_compat_2_3
- HW_COMPAT_WILY needed to be HW_COMPAT_2_4 + some things that moved to pc_compat_2_3
- when this was further reworked in later releases this was partially lost
  and is right now using hw_version = "2.4.0"; and pc_i440fx_2_4_machine_options (right)
  but also hw_compat_2_3 + pc_compat_2_3 (wrong)
- I think we need to make it use hw_compat_2_4 + those that moved to pc_compat_2_3 later.
  In the past we have identified these as
      .property = "send-configuration",\
      .property = "send-section-footer",\
      .property = "store-global-state",\

We can use the debug steps above with GDB or the "info qtree" on the monitor to gather a full list of compat attributes that are registered to the fully initialized wily machine.
They should match with a fix applied.

$ gdb /usr/bin/qemu-system-x86_64
(gdb) handle SIGUSR1 noprint nostop
(gdb) b qemu_init_main_loop
(gdb) run -m 128 -M pc-i440fx-wily,accel=kvm --nodefaults --nographic --monitor stdio -device virtio-balloon-pci,id=balloon0
Thread 1 "qemu-system-x86" hit Breakpoint 3, qemu_init_main_loop (errp=0x7fffffffe468) at ./util/main-loop.c:148
(gdb) set $i=62
(gdb) p (*(GlobalProperty*)(object_compat_props[1]).pdata[$i++])

# 56 is where hw_compat_2_10 starts in Focal and that tail is what needs to match in Bionic and earlier

Effective compats in place for wily at 4.2 as of today (from gdb):
"virtio-mouse-device" "wheel-axis" "false", used = false, optional = false
"virtio-tablet-device" "wheel-axis" "false", used = false, optional = false
"e1000" "romfile" "compat-256k-efi-e1000.rom", used = false, optional = false
"ne2000" "romfile" "compat-256k-efi-ne2k_pci.rom", used = false, optional = false
"pcnet" "romfile" "compat-256k-efi-pcnet.rom", used = false, optional = false
"rtl8139" "romfile" "compat-256k-efi-rtl8139.rom", used = false, optional = false
"virtio-net-pci" "romfile" "compat-256k-efi-virtio.rom", used = false, optional = false
"x86_64-cpu" "x-hv-max-vps" "0x40", used = false, optional = false
"i440FX-pcihost" "x-pci-hole64-fix" "off", used = false, optional = false
"q35-pcihost" "x-pci-hole64-fix" "off", used = false, optional = false
"pci-bridge" "shpc" "off", used = false, optional = false
"intel-iommu" "pt" "off", used = false, optional = false
"virtio-net-device" "x-mtu-bypass-backend" "off", used = false, optional = false
"pcie-root-port" "x-migrate-msix" "false", used = false, optional = false
"mch" "extended-tseg-mbytes" "0", used = false, optional = false
"fw_cfg_mem" "x-file-slots" "0x10", used = false, optional = false
"fw_cfg_io" "x-file-slots" "0x10", used = false, optional = false
"pflash_cfi01" "old-multiple-chip-handling" "on", used = false, optional = false
"pci-bridge" "shpc" "on", used = false, optional = false
"pci-device" "x-pcie-extcap-init" "off", used = false, optional = false
"virtio-pci" "x-pcie-deverr-init" "off", used = false, optional = false
"virtio-pci" "x-pcie-lnkctl-init" "off", used = false, optional = false
"virtio-pci" "x-pcie-pm-init" "off", used = false, optional = false
"cirrus-vga" "vgamem_mb" "8", used = false, optional = false
"isa-cirrus-vga" "vgamem_mb" "8", used = false, optional = false
"x86_64-cpu" "tcg-cpuid" "off", used = false, optional = false
"kvmclock" "x-mach-use-reliable-get-clock" "off", used = false, optional = false
"ICH9-LPC" "x-smi-broadcast" "off", used = false, optional = false
"x86_64-cpu" "vmware-cpuid-freq" "off", used = false, optional = false
"Haswell-x86_64-cpu" "stepping" "1", used = false, optional = false
"virtio-pci" "page-per-vq" "on", used = false, optional = false
"virtio-serial-device" "emergency-write" "off", used = false, optional = false
"ioapic" "version" "0x11", used = false, optional = false
"intel-iommu" "x-buggy-eim" "true", used = false, optional = false
"virtio-pci" "x-ignore-backend-features" "on", used = false, optional = false
"x86_64-cpu" "l3-cache" "off", used = false, optional = false
"x86_64-cpu" "full-cpuid-auto-level" "off", used = false, optiona...

Bionic

$ gdb /usr/bin/qemu-system-x86_64
(gdb) handle SIGUSR1 noprint nostop
(gdb) b machine_register_compat_props
(gdb) run -m 128 -M pc-i440fx-wily,accel=kvm --nodefaults --nographic --monitor stdio -device virtio-balloon-pci,id=balloon0
(gdb) fin

(gdb) set $c=(GList *)global_props
(gdb) define report
>p *((GlobalProperty*)$c->data)
>set $c=$c->next
>end
(gdb) report

On qemu 2.11 delivered

"virtio-mouse-device" "wheel-axis" "false", user_provided = false, used = false
"virtio-tablet-device" "wheel-axis" "false", user_provided = false, used = false
"e1000" "romfile" "compat-256k-efi-e1000.rom", user_provided = false, used = false
"ne2000" "romfile" "compat-256k-efi-ne2k_pci.rom", user_provided = false, used = false
"pcnet" "romfile" "compat-256k-efi-pcnet.rom", user_provided = false, used = false
"rtl8139" "romfile" "compat-256k-efi-rtl8139.rom", user_provided = false, used = false
"virtio-net-pci" "romfile" "compat-256k-efi-virtio.rom", user_provided = false, used = false
"x86_64-cpu" "x-hv-max-vps" "0x40", user_provided = false, used = true
"i440FX-pcihost" "x-pci-hole64-fix" "off", user_provided = false, used = true
"q35-pcihost" "x-pci-hole64-fix" "off", user_provided = false, used = false
"pci-bridge" "shpc" "off", user_provided = false, used = false
"intel-iommu" "pt" "off", user_provided = false, used = false
"virtio-net-device" "x-mtu-bypass-backend" "off", user_provided = false, used = false
"pcie-root-port" "x-migrate-msix" "false", user_provided = false, used = false
"mch" "extended-tseg-mbytes" "0", user_provided = false, used = false
"fw_cfg_mem" "x-file-slots" "0x10", user_provided = false, used = false
"fw_cfg_io" "x-file-slots" "0x10", user_provided = false, used = true
"pflash_cfi01" "old-multiple-chip-handling" "on", user_provided = false, used = false
"pci-bridge" "shpc" "on", user_provided = false, used = false
"pci-device" "x-pcie-extcap-init" "off", user_provided = false, used = true
"virtio-pci" "x-pcie-deverr-init" "off", user_provided = false, used = true
"virtio-pci" "x-pcie-lnkctl-init" "off", user_provided = false, used = true
"virtio-pci" "x-pcie-pm-init" "off", user_provided = false, used = true
"cirrus-vga" "vgamem_mb" "8", user_provided = false, used = false
"isa-cirrus-vga" "vgamem_mb" "8", user_provided = false, used = false
"x86_64-cpu" "tcg-cpuid" "off", user_provided = false, used = true
"kvmclock" "x-mach-use-reliable-get-clock" "off", user_provided = false, used = true
"ICH9-LPC" "x-smi-broadcast" "off", user_provided = false, used = false
"x86_64-cpu" "vmware-cpuid-freq" "off", user_provided = false, used = true
"Haswell-x86_64-cpu" "stepping" "1", user_provided = false, used = false
"virtio-pci" "page-per-vq" "on", user_provided = false, used = true
"virtio-serial-device" "emergency-write" "off", user_provided = false, used = false
"ioapic" "version" "0x11", user_provided = false, used = false
"intel-iommu" "x-buggy-eim" "true", user_provided = false, used = false
"virtio-pci" "x-ignore-backend-features" "on", user_provided = false, used = true
"x86_64-cpu" "l3-cache" "off", user_provided = false, used = true
"x86_64-cpu" "full-cpuid-auto-level" "off", user_provided = false, used = true
"Opteron_G...

4.2 misses:
the three known "moved between HW and PC compat
"migration" "send-configuration" "off", user_provided = false, used = true
"migration" "send-section-footer" "off", user_provided = false, used = true
"migration" "store-global-state" "off", user_provided = false, used = true

4.2 has "too much"
Those are probably all or mostly due to using hw_compat_2.3 instead of hw_compat_2.4+the-three-above
"virtio-blk-pci" "any_layout" "off", used = false, optional = false
"virtio-balloon-pci" "any_layout" "off", used = false, optional = false
"virtio-serial-pci" "any_layout" "off", used = false, optional = false
"virtio-9p-pci" "any_layout" "off", used = false, optional = false
"virtio-rng-pci" "any_layout" "off", used = false, optional = false
"pci-device" "x-pcie-lnksta-dllla" "off", used = false, optional = false
^^ the above are missing
"migration" "send-configuration" "off", used = false, optional = false
"migration" "send-section-footer" "off", used = false, optional = false
"migration" "store-global-state" "off", used = false, optional = false
^^ those three it got from hw_compat_2_3 but need to get on top of hw_compat_2_3 instead

vv those cpu attributes below are from pc_compat_2_3 and should be pc_compat_2_4
"qemu32-x86_64-cpu" "model-id" "QEMU Virtual CPU version 2.3.0", used = false, optional = false
"qemu64-x86_64-cpu" "model-id" "QEMU Virtual CPU version 2.3.0", used = false, optional = false
"athlon-x86_64-cpu" "model-id" "QEMU Virtual CPU version 2.3.0", used = false, optional = false
"x86_64-cpu" "arat" "off", used = false, optional = false
"qemu64-x86_64-cpu" "min-level" "4", used = false, optional = false
"kvm64-x86_64-cpu" "min-level" "5", used = false, optional = false
"pentium3-x86_64-cpu" "min-level" "2", used = false, optional = false
"n270-x86_64-cpu" "min-level" "5", used = false, optional = false
"Conroe-x86_64-cpu" "min-level" "4", used = false, optional = false
"Penryn-x86_64-cpu" "min-level" "4", used = false, optional = false
"Nehalem-x86_64-cpu" "min-level" "4", used = false, optional = false
"n270-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Penryn-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Conroe-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Nehalem-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Westmere-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"SandyBridge-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"IvyBridge-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Haswell-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Haswell-noTSX-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Broadwell-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"Broadwell-noTSX-x86_64-cpu" "min-xlevel" "0x8000000a", used = false, optional = false
"x86_64-cpu" "kvm-no-smi-migration" "on", used = false, optional = false

TODO:
- create hw_compat_2.4+the-three-migration-attribs
- wily to use hw_compat_2.4+the-three-migration-attribs
- wily to use pc_compat_2.4

Once patched and built for Focal re-te...

Changes:
- "any_layout" "off" no more present (all six entries of the bad compat are gone)
- 2.4 values in "qemu32-x86_64-cpu" "model-id" (matches Bionic now, formerly had 2.4+2.3 set)
- "migration" "send-configuration" (still present as before and as in Bionic)
- all other implied compat changes between 2.3/2.4 changed as well e.g. it now has "virtio-blk-device" "scsi" "true" (which is 2.4)

The above is great - all compat entries that Bionic had are now present and the bad ones that were too much are gone \o/.
But it isn't perfect yet there is a "tail" that repeats some of the compat entries.

The following "tail" is registered in the wily compat type twice now:
"virtio-blk-device" "scsi" "true"
"e1000" "extra_mac_registers" "off"
"virtio-pci" "x-disable-pcie" "on"
"virtio-pci" "migrate-extra" "off"
"fw_cfg_mem" "dma_enabled" "off"
"fw_cfg_io" "dma_enabled" "off"
"qemu32-x86_64-cpu" "model-id" "QEMU Virtual CPU version 2.4.0"
"qemu64-x86_64-cpu" "model-id" "QEMU Virtual CPU version 2.4.0"
"athlon-x86_64-cpu" "model-id" "QEMU Virtual CPU version 2.4.0"
"Haswell-x86_64-cpu" "abm" "off"
"Haswell-noTSX-x86_64-cpu" "abm" "off"
"Broadwell-x86_64-cpu" "abm" "off"
"Broadwell-noTSX-x86_64-cpu" "abm" "off"
"host-x86_64-cpu" "host-cache-info" "on"
"x86_64-cpu" "check" "off"
"qemu64-x86_64-cpu" "sse4a" "on"
"qemu64-x86_64-cpu" "abm" "on"
"qemu64-x86_64-cpu" "popcnt" "on"
"qemu32-x86_64-cpu" "popcnt" "on"
"Opteron_G2-x86_64-cpu" "rdtscp" "on"
"Opteron_G3-x86_64-cpu" "rdtscp" "on"
"Opteron_G4-x86_64-cpu" "rdtscp" "on"
"Opteron_G5-x86_64-cpu" "rdtscp" "on"

While functionally that should be no difference it needs to be analyzed as I hope to much to not touch this broken old type again that we should not let it slip this time int hat regard.

This is due to applying pc_i440fx_2_4_machine_options + pc_i440fx_wily_machine_options. The former does 2.4 proper and the latter the special wily things.
But since wily is this weird hybrid it needs "only" wily and no normal 2.4 to be called.
I think I have this fixed now, rebuilding for another test to be sure ...

Good - now the compat attributes in Focal for a wily type machine fully match what was present in Bionic.

@Trya - could I ask you to try qemu - 1:4.2-3ubuntu6.9~ppa3 from PPA [1] on Focal and do a migration of the kind of windows-guest that formerly failed onto a Focal target that has that installed?

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/debug-qemu-4.2

Thank you for your time!

The fix from PPA works. It fixes both problems that I described in original message:
- problem with migration of windows guests from bionic to focal
- problem with cpu feature 'arat' when migrating from focal to bionic

---

on the unrelated note, I wish that I had known earlier that 'wily' in 'pc-i440fx-wily' is a codename of an old ubuntu release. I always thought 'pc-i440fx-wily' is some fancy name from qemu project. If I would have known the truth earlier, then I would ditch this machine type years ago.

Yeah I understand that it is bad i you realize only rather later that it is a release, but the full name references the Ubuntu release quite literally.

$ qemu-system-x86_64 -M ? | grep wily
pc-i440fx-wily Ubuntu 15.04 PC (i440FX + PIIX, 1996)

We can't change the actual type names but the description is a bit of free-form at least for future types.

Do you think there would be a better text making it more obvious, or should we forget about that part and just focus on the actual issue of the bad type here?

Also thanks for testing the PPA @Trya!
Once this has become an SRU you'll be asked once more to test it, but until then I need to first fix it in the current Ubuntu release in Development and then do SRUs.

FYI: I added the SRU template in preparation and opened three MPs for the required packaging changes. Waiting for review on those now.

the fix in 21.04 is in -proposed but will (due to a compiler bug) only fully release with the 5.1 upload of hopefully next week.
Never the less it is in hirsute-proposed and that is enough to get the SRUs started.

I uploaded the fix to F/G -unapproved queue

Hello trya, or anyone else affected,

Accepted qemu into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:5.0-5ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello trya, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted qemu (1:4.2-3ubuntu6.9) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/245.4-4ubuntu3.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted qemu (1:5.0-5ubuntu9.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.694 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

@trya - do you have a chance to test focal-proposed and groovy-proposed as a migration target with your windows machines that are affected?
It would be the preferred way to validate this, if you can't please let me know (then I'll do at least the GDB based attribute check that I did before to ensure they match).

Issue in livecd-rootfs/2.694 resolved.
The one in systemd likely as well (known to be flaky) but it is still running.

Autopkgtest on systemd/245.4-4ubuntu3.3 (amd64) resolved as well now.

First of all - the upgrades to the versions in proposed worked just fine.

Groovy
root@g-wily:~# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  alsa-ucm-conf apport apt apt-utils busybox-initramfs busybox-static libapt-pkg6.0 libasound2 libasound2-data python3-apport python3-problem-report qemu-block-extra qemu-system-common
  qemu-system-data qemu-system-gui qemu-system-x86 qemu-system-x86-dbgsym qemu-utils
18 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 60.5 MB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libapt-pkg6.0 amd64 2.1.11 [848 kB]
Get:2 http://ddebs.ubuntu.com groovy-proposed/main amd64 qemu-system-x86-dbgsym amd64 1:5.0-5ubuntu9.1 [44.1 MB]
Get:3 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 apt amd64 2.1.11 [1305 kB]
Get:4 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 apt-utils amd64 2.1.11 [213 kB]
Get:5 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 busybox-static amd64 1:1.30.1-4ubuntu9.1 [915 kB]
Get:6 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 alsa-ucm-conf all 1.2.2-1ubuntu5.1 [25.8 kB]
Get:7 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 python3-problem-report all 2.20.11-0ubuntu50.2 [9968 B]
Get:8 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 python3-apport all 2.20.11-0ubuntu50.2 [84.6 kB]
Get:9 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 apport all 2.20.11-0ubuntu50.2 [129 kB]
Get:10 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 busybox-initramfs amd64 1:1.30.1-4ubuntu9.1 [167 kB]
Get:11 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libasound2 amd64 1.2.3.2-1ubuntu3.1 [334 kB]
Get:12 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 libasound2-data all 1.2.3.2-1ubuntu3.1 [19.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 qemu-utils amd64 1:5.0-5ubuntu9.1 [996 kB]
Get:14 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 qemu-system-common amd64 1:5.0-5ubuntu9.1 [1790 kB]
Get:15 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 qemu-block-extra amd64 1:5.0-5ubuntu9.1 [56.9 kB]
Get:16 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 qemu-system-data all 1:5.0-5ubuntu9.1 [1092 kB]
Get:17 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 qemu-system-gui amd64 1:5.0-5ubuntu9.1 [44.3 kB]
Get:18 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 qemu-system-x86 amd64 1:5.0-5ubuntu9.1 [8335 kB]
Fetched 60.5 MB in 19s (3156 kB/s)
(Reading database ... 47451 files and directories currently installed.)
Preparing to unpack .../libapt-pkg6.0_2.1.11_amd64.deb ...
Unpacking libapt-pkg6.0:amd64 (2.1.11) over (2.1.10) ...
Setting up libapt-pkg6.0:amd64 (2.1.11) ...
(Reading database ...

The originally reported check via qtree (just looking at
virtio-balloon-pci) now correctly reports true for any_layout.

Focal
      dev: virtio-balloon-pci, id "balloon0"
...
        bus: virtio-bus
          type virtio-pci-bus
          dev: virtio-balloon-device, id ""
...
              any_layout = true

Groovy
      dev: virtio-balloon-pci, id "balloon0"
...
        bus: virtio-bus
          type virtio-pci-bus
          dev: virtio-balloon-device, id ""
...
            any_layout = true

GDB check attributes of the wily type in gdb in regard to
virtio-blk-pci
virtio-balloon-pci
virtio-serial-pci
virtio-9p-pci
virtio-rng-pci
migration send-configuration
migration send-section-footer
migration store-global-state

Focal (2.10 is starting at 62)
(gdb) p (GlobalProperty*)(object_compat_props[1]).pdata[62]
$1 = (GlobalProperty *) 0x555556477940 <hw_compat_2_10>

Groovy (2.10 is starting at 75)
(gdb) p (GlobalProperty*)(object_compat_props[1]).pdata[75]
$1 = (GlobalProperty *) 0x555556436340 <hw_compat_2_10>

Both of those fully match the correct attributes as listed in comment #13 for Bionic.
See tarball for detailed data

Setting verified tags as I confirmed in GDB and qtree.

@trya - still if there would be a chance to also try the windows guests on these before we release that would be great.

Everything works. I tested migration to focal and groovy.

focal: migration fails on 1:4.2-3ubuntu6.8, works on 1:4.2-3ubuntu6.9
groovy: migration fails on 1:5.0-5ubuntu9, works on 1:5.0-5ubuntu9.1

On Wed, Nov 18, 2020 at 8:15 PM trya uuum <email address hidden> wrote:
>
> Everything works. I tested migration to focal and groovy.

Thank you!

This bug was fixed in the package qemu - 1:5.1+dfsg-4ubuntu1

---------------
qemu (1:5.1+dfsg-4ubuntu1) hirsute; urgency=medium

  * Merge with Debian testing, remaining changes:
    Fixes qemu-arm-static Assertion `guest_base != 0' failed (LP: #1897854)
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type (LP: 1304107 1621042)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types
      - d/qemu-system-x86.NEWS Info on fixed machine type definitions
        for host-phys-bits=true (LP: 1776189)
      - add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - provide pseries-bionic-2.11-sxxm type as convenience with all
        meltdown/spectre workarounds enabled by default. (LP: 1761372).
      - ubuntu-q35 alias added to auto-select the most recent q35 ubuntu type
    - Enable nesting by default
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - improved dependencies
      - Make qemu-system-common depend on qemu-block-extra
      - Make qemu-utils depend on qemu-block-extra
      - let qemu-utils recommend sharutils
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/control-in: Disable capstone disassembler library support (universe)
    - d/qemu-system-x86.README.Debian: add info about updated nesting changes
    - d/control*, d/rules: disable xen by default, but provide universe
      package qemu-system-x86-xen as alternative
      [includes compat links changes of 5.0-5ubuntu4]
    - allow qemu to load old modules post upgrade (LP 1847361)
      - d/qemu-block-extra.*.in, d/qemu-system-gui.*.in: save shared objects on
        upgrade
      - d/rules: generate maintainer scripts matching package version on build
      - d/rules: enable --enable-module-upgrades where --enable-modules is set
    - d/control: regenerate debian/control out of control-in
  * Dropped changes [in Debian or no more needed]
    - d/control-in: disable pmem on ppc64 as it is currently considered
      experimental on that architecture (pmdk v1.8-1)
    - d/rules: makefile definitions can't be recursive - sys_systems for s390x
    - d/rules: report config log from the correct subdir
    - d/control-in: disable rbd support unavailable on riscv (LP: 1872931)
    - Pick further changes for groovy from debian/master since 5.0-5
      - ati-vga-check-mm_index-before-recursive-call-CVE-2...

I see that the armhf binaries FTBFS with the latest version. Could someone take a look at it?

Thanks for the ping Łukasz,
  during RTL pass: reload
  /<<PKGBUILDDIR>>/fpu/softfloat.c: In function ‘soft_f64_muladd’:
  /<<PKGBUILDDIR>>/fpu/softfloat.c:1535:1: internal compiler error: Segmentation fault
   1535 | }
        | ^
that is bug 1890435 which should be flaky in groovy.

I'd really prefer not to switch to gcc-9 in groovy for this.
In 21.04 this became 100% non buildable and we had no other chance until bug 1890435 is resolved there. But in Groovy I'd hope we get away with re-triggering the build.
I did that and will give it three tries before we have to consider the gcc-9 treatment.

The armhf build is resolved (third was a charm)

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:5.0-5ubuntu9.1

---------------
qemu (1:5.0-5ubuntu9.1) groovy; urgency=medium

  * d/p/ubuntu/define-ubuntu-machine-types.patch: update to fix 15.04 wily
    machine type to match how it originally was released (LP: #1902654)

 -- Christian Ehrhardt <email address hidden> Mon, 09 Nov 2020 08:19:07 +0100

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.9

---------------
qemu (1:4.2-3ubuntu6.9) focal; urgency=medium

  * d/p/ubuntu/define-ubuntu-machine-types.patch: update to fix 15.04 wily
    machine type to match how it originally was released (LP: #1902654)

 -- Christian Ehrhardt <email address hidden> Wed, 04 Nov 2020 15:34:47 +0100