[Xenial] KVM trusty guest 3.13.0-68 raid6-pq panic in raid6_avx21_gen_syndrome() while probing grub devices [was: Xenial KVM: updating Trusty guest from 3.13.0-68 to 3.13.0-71 causes kernel exception]

For what it's worth, unchecking "[ ] Copy host CPU configuration" in virt-manager and selecting "Hypervisor default" for the CPU is a workaround. (/proc/cpuinfo reports "QEMU Virtual CPU version 2.4.0")

Also, the hypervisor's /proc/cpuinfo reports the following CPU configuration (8 cores of the same):

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1e
cpu MHz : 2500.097
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bugs :
bogomips : 4988.49
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

And the hypervisor's kernel is version 4.2.0-19-generic.

This change was made by a bot.

I just tried again with two logical CPUs and the core set to "Westmere" (I think that was the default), and didn't see the bug. So this seems related specifically to "[x] Copy host CPU configuration" being checked. Here's the /proc/cpuinfo from the "virtual Westmere" where it works:

http://paste.ubuntu.com/13836572/

So the host exposing some of the physical cpu characteristics is triggering use of this specific syndrome generator (which makes sense as it is h/w specific) raid6_avx21_gen_syndrome().

I will note that this has thrown an illegal instruction trap while doing this:

    [ 522.963634] invalid opcode: 0000 [#1] SMP

So it is likely that dispite offering up the avx2 flags to the guest the host is not actually providing the instructions?

Ok I attempted to reproduce without success. What I tried:
- Xenial on Xenial on machine/guest with avx2
- Trusty 3.13.0-{68,71} on Xenial on machine/guest with avx2

In both of these instances I modprobed raid6_pq, in the trusty instance I triggered an upgrade from 68 to 71 and neither of these caused a segfault.

The following information would be useful in debugging this:

1) The differences between the guests /proc/cpuinfo with it reproducing and not-reproducing. This can verify the avx2 bit is to blame.
2) Can you simply 'sudo modprobe -r raid6_pq && sudo modprobe raid6_pq' to trigger this issue?
3) Can you test with a xenial guest and a similar configuration to see if this still triggers the issue?
4) If you have the ability testing with different versions of hypervisor (trusty vs xenial) might also be useful.

Thanks,
--chris j arges

On my Trusty VM with "[x] Copy host CPU configuration" checked in virt-manager, running 'sudo modprobe -r raid6_pq && sudo modprobe raid6_pq' is enough to reproduce the kernel exception in dmesg (though I didn't see it print "Segmentation Fault", so that may be somewhat of a red herring).

Here's /proc/cpuinfo on the guest:
http://paste.ubuntu.com/13868715/

And here's /proc/cpuinfo on the hypervisor host:
http://paste.ubuntu.com/13868744/

I have a separate issue (which I have not filed, because I haven't triaged, and am not sure I can reproduce yet) that was blocking me from creating a Xenial guest, but I can try again shortly.

Status changed to 'Confirmed' because the bug affects multiple users.

It affects me, trying to bootstrap xenial vm on xenial, getting OOPS like above.

# virt-install --name $SHORT \
        --ram $MEM \
        --vcpus $CPUS \
        --location=http://hk.archive.ubuntu.com/ubuntu/dists/xenial/main/installer-amd64/ \
        --os-type=linux --os-variant=virtio26 \
        --bridge=siec,model=virtio,mac=$MAC \
        --disk path=$DISK \
        --cpuset=0,1,2 --cpu=host \
        --autostart --nographics \
        --extra-args="auto=true priority=critical locale=pl_PL keymap=pl debian-installer/allow_unauthenticated_ssl=true console-setup/ask_detect=false netcfg/hostname=$HOST url=https://domain.com/ubuntu.pre4seed.cfg console=tty0 console=ttyS0,115200n8 serial"

To be honest, it's pretty annoying bug, as I am not able to install any xenial VM :-(

@Serge, A workaround for me is to use "hypervisor default" in virt-manager. I'm not sure what the equivalent is in virt-install, but maybe using --cpu=host-model-only would be a workaround?

From the man page:

               Expose the host CPUs configuration to the guest. This enables the guest to take advantage of many
               of the host CPUs features (better performance), but may cause issues if migrating the guest to a
               host without an identical CPU.

           --cpu host-model-only
               Expose the nearest host CPU model configuration to the guest. It is the best CPU which can be
               used for a guest on any of the hosts.

           Use --cpu=? to see a list of all available sub options. Complete details at
           <http://libvirt.org/formatdomain.html#elementsCPU>

The bug persists in kernel 4.3.0-2-generic #11-Ubuntu.

@Mike - thanks for a workaround, it works, indeed :-)

--cpu=host-model-only

for virt-manager.

If you need avx2 support, --cpu Haswell-noTSX,-x2apic works on Haswell desktop/laptop chips.

The most confusing problem is that qemu's definition of "Haswell" is actually Haswell-E, -EP and -EX; Haswell itself lacks x2apic, which qemu's Haswell requires. x2apic dates back to Nehalem, but qemu's CPU definitions only include it back to Sandy Bridge, so a standard desktop or laptop Haswell CPU falls all the back to Westmere and then adds flags including avx, avx2 and xsave on top of that[0].

Advertising support for AVX and AVX2 is just a matter of setting CPUID.1:ECX.AVX and CPUID.7:EBX.AVX2, but the instructions won't actually work unless XCR0.AVX is set, and kvm_load_guest_xcr0 only sets XCR0 from the guest if the guest's XR4.OSXSAVE is set. The guest's fpu__init_cpu_xstate only sets CR4.OSXSAVE when xfeatures_mask is non-zero, and xfeatures_mask is calculated by fpu__init_system_xstate from CPUID.(EAX=0DH,ECX=0), which doesn't exist on qemu's Westmere (level=0xb), so XCR0.AVX remains unset and AVX2 instructions #UD.

The bug is probably that raid6_have_avx2 only checks that AVX2 is supported in CPUID, not that it's enabled. Checking for X86_FEATURE_OSXSAVE might work, though I'm not sure if the value checked by boot_cpu_has is stored too early for that.

I am also a little suspicious of kvm_load_guest_xcr0's CR4.OSXSAVE guard. The Intel manuals state that XSAVE, XSRSTOR, XGETBV and XSETBV require the flag to be set, but KVM won't restore a guest's existing XCR0 unless OSXSAVE is still set.

[0] "-cpu Westmere,+invpcid,+erms,+bmi2,+smep,+avx2,+bmi1,+fsgsbase,+abm,+rdtscp,+pdpe1gb,+rdrand,+f16c,+avx,+osxsave,+xsave,+tsc-deadline,+movbe,+pcid,+pdcm,+xtpr,+fma,+tm2,+est,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme"

One could argue that libvirt should exclude x2apic from the host-model checks, as it's emulated by qemu whether or not the host supports it.

I am seeing the same issue on some of my OpenStack compute nodes, interestingly those which seem to have a newer CPU than others.

Affected CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
Mapped in guest to: Intel Core i7 9xx (Nehalem Class Core i7)

Unaffected Host CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
Mapped in guest to: Intel Xeon E312xx (Sandy Bridge)

Booting a Xenial guest on the affected host crashes during boot, see http://paste.ubuntu.com/15204908/. A Wily guest runs fine at first, but generates a similar traceback as soon as I modprobe raid6_pq.

Both hosts are running nova-compute from Wily. Please let me know if you need any further details.

Just hit this same issue with nova-compute on Xenial and creating a Xenial instance in nova. I worked around the issue with:

juju set-config nova-compute cpu-mode=host-passthrough

Hit this issue when booting Ubuntu Trusty VM on an Ubuntu host, worked around via editing VM XML by adding this in <cpu> section:

<feature policy='disable' name='avx2'/>

This + destroy->start the VM and it booted fine.

I've hit this bug as well, OpenStack Nova Compute 2:12.0.4-0ubuntu1~cloud1, KVM 1:2.3+dfsg-5ubuntu9.4~cloud1. Hypervisor is running Ubuntu 14.04.4 with CPU flags:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid

(NOTE "avx2")

CPU model is Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz

Guest running Ubuntu Xenial (16.04.1) - started up with the following arguments (note again "avx2"):

libvirt+ 7221 1 5 14:22 ? 00:01:14 qemu-system-x86_64 -enable-kvm -name instance-00000002 -S -machine pc-i440fx-vivid,accel=kvm,usb=off -cpu Nehalem,+invpcid,+erms,+bmi2,+smep,+avx2,+bmi1,+fsgsbase,+abm,+rdtscp,+pdpe1gb,+rdrand,+f16c,+avx,+osxsave,+xsave,+tsc-deadline,+movbe,+x2apic,+dca,+pcid,+pdcm,+xtpr,+fma,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d1d07811-e82c-4fc0-9df0-31b7992b098b -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=12.0.4,serial=00000000-0000-0000-0000-0cc47a4e5b5a,uuid=d1d07811-e82c-4fc0-9df0-31b7992b098b,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000002.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/disk/by-path/ip-10.101.104.1:3260-iscsi-iqn.2010-10.org.openstack:volume-7ad6788f-fdb0-4e50-a470-525bf36a8f1e-lun-1,if=none,id=drive-virtio-disk0,format=raw,serial=7ad6788f-fdb0-4e50-a470-525bf36a8f1e,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d5:97:04,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/home/zerostack/data/openstack/nova/instances/d1d07811-e82c-4fc0-9df0-31b7992b098b/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

Symptom: VM Crashes on boot up with similar stack trace as documented here.