libvirt/qemu guests can't boot with -no-kvm

Description of problem:

using 'no hardware acceleration' option in qemu (no-kvm) results a situation where vm is stuck on booting from hard drive.
when running same vm but using the '-enable-kvm' option, vm boots from hard drive and os is loaded.

qemu commands:

 - qemu process with kvm enabled - vm boots from hard drive as expected

PID TTY STAT TIME COMMAND
31956 ? Sl 0:07 /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu qemu64,+cx16,+ssse3,-svm -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name libvirt-pool-07 -uuid 9362025d-0906-4587-9b0e-0561c25aaa53 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/libvirt-pool-07.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=2010-4-20T16:23:34 -boot c -drive file=/rhev/data-center/606d043c-ef9c-4c6f-848b-5bd89325c78d/d9124e52-d42a-4b0c-8657-523bc5b6733b/images/ad757a3f-bce5-4629-a91d-9d81bef59377/afd1ef06-a03d-4751-9348-2a4c1467f78f,if=none,id=drive-ide0-0-0,boot=on,format=qcow2,serial=29-a91d-9d81bef59377,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:23:71:17,bus=pci.0,addr=0x4 -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
  PID TTY STAT TIME COMMAND

 - qemu process with kvm disabled - vm doesn't boot from hard drive and stuck
   on 'Booting from Hard disk...'

32322 ? Rl 0:08 /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu qemu64,+cx16,+ssse3,-svm -no-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name libvirt-pool-07 -uuid 9362025d-0906-4587-9b0e-0561c25aaa53 -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/libvirt-pool-07.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=2010-4-20T16:24:49 -boot cn -drive file=/rhev/data-center/606d043c-ef9c-4c6f-848b-5bd89325c78d/d9124e52-d42a-4b0c-8657-523bc5b6733b/images/ad757a3f-bce5-4629-a91d-9d81bef59377/afd1ef06-a03d-4751-9348-2a4c1467f78f,if=none,id=drive-ide0-0-0,boot=on,format=qcow2,serial=29-a91d-9d81bef59377,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:23:71:17,bus=pci.0,addr=0x4 -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

versions:

vdsm-4.9-6.1.x86_64
qemu-kvm-0.12.1.2-2.59.el6.x86_64
libvirt-0.8.1-6.el6.x86_64
2.6.32-21.el6.x86_64 (RHEL 6)

How reproducible: always

Steps to Reproduce:
1. start vm and make sure -no-kvm arg is provided
2. access vm via vnc

I'm experiencing very similar thing. I'm not sure if it's the same, but in my case without kvm supported hardware, booting freezes on 'GRUB Loading'. I've done some debugging and if I run command provided by libvirt (minus -S), I get the same behaviour. But, if I replace boot=on with boot=off in -drive, it boots and works as expected.

I'm experiencing this on Ubuntu 10.04 (libvirt 0.7.5).

As a dirty workaround, I've replaced all occurances of boot=on with boot=off in whole of libvirt code. After that everything works and I guess that's the source of the problem.

Where this requirement comes from? extboot does not and never worked without KVM. It relies on some KVM limitation to function that does not exists in plane QEMU.

With RHEL5's kvm we could boot guests with -no-kvm. Unless stated otherwise, we would like to keep this ability with RHEL6's kvm+libvirt.

Is there a workaround, or is this feature dead?

Don't use boot=on then. (In reply to comment #6)
> With RHEL5's kvm we could boot guests with -no-kvm. Unless stated otherwise, we
Really? With boot=on?

> would like to keep this ability with RHEL6's kvm+libvirt.
Then libvirt shouldn't specify boot=on.

>
> Is there a workaround, or is this feature dead?
don't use boot=on

Daniel, please ask Gleb if libvirt should avoid setting boot=on for drives while using -no-kvm? Maybe if he repeats his suggestion 3 more time we would understand the problem better.

> > With RHEL5's kvm we could boot guests with -no-kvm.
> Really? With boot=on?

Yes, that works fine with guests I'm running on RHEL5

eg this one

/usr/bin/qemu-system-x86_64 -S -M pc -no-kvm -m 409 -smp 2 -name rhel5x86_64 -uuid a29d663e-3083-1096-b8ba-a11c081e77e0 -no-kvm-pit-reinjection -monitor pty -pidfile /var/run/libvirt/qemu//rhel5x86_64.pid -boot c -drive file=/var/lib/libvirt/images/rhel5x86_64.img,if=ide,index=0,boot=on -drive file=/var/lib/libvirt/images/extra1.img,if=virtio,index=0 -drive file=/var/lib/libvirt/images/extra11.img,if=virtio,index=1 -net nic,macaddr=54:52:00:3a:a8:af,vlan=0 -net tap,fd=18,script=,vlan=0,ifname=vnet0 -net nic,macaddr=54:52:00:3a:a8:bf,vlan=1,model=e1000 -net tap,fd=19,script=,vlan=1,ifname=vnet1 -net nic,macaddr=54:52:00:3a:a8:cf,vlan=2,model=e1000 -net tap,fd=20,script=,vlan=2,ifname=vnet2 -net nic,macaddr=54:52:00:3a:a8:df,vlan=3,model=virtio -net tap,fd=21,script=,vlan=3,ifname=vnet3 -net nic,macaddr=54:52:00:3a:a8:ef,vlan=4,model=virtio -net tap,fd=22,script=,vlan=4,ifname=vnet4 -serial pty -parallel none -usb -vnc 127.0.0.1:1 -soundhw ac97

> > would like to keep this ability with RHEL6's kvm+libvirt.
> Then libvirt shouldn't specify boot=on.

If QEMU reports that it has support for 'boot=on', then libvirt will use it.
If this flag isn't supposed to be used anymore then it needs to be removed from QEMU, otherwise the regression wrt previous behaviour needs to be fixed. libvirt can't second guess the fact that the RHEL6 kvm binary reports boot=on support, which then doesn't work in certain scenarios.

> Either libvirt needs to stop using ,boot=on for anything but virtio disks or we
> need to make qemu not act on ,boot=on for anything but virtio. The second
> option doesn't sound too good.

This has worked without trouble ever since we added it. Before we consider removing anything, we need to know why there is a regression in KVM behaviour in RHEL-6 vs RHEL5 where it was working.

(In reply to comment #13)
> > Either libvirt needs to stop using ,boot=on for anything but virtio disks or we
> > need to make qemu not act on ,boot=on for anything but virtio. The second
> > option doesn't sound too good.
>
> This has worked without trouble ever since we added it. Before we consider
> removing anything, we need to know why there is a regression in KVM behaviour
> in RHEL-6 vs RHEL5 where it was working.

The problem was analyzed a long time ago:
http://<email address hidden>/msg29091.html
I haven't check why old qemu works, my guess is old BIOS haven't locked all BIOS memory.

extboot was always just a hack to overcome bios shortcomings (not being able to boot from virtio). It was never meant to be used with ide and rhev-m people respected that. And -no-kvm option is unsupported in RHEL6, so any use of it may cause data lose, hair lose or worse.

RHEL6 bios now have native ability to boot from virtio and since scsi is unsupported we should drom boot=on and extboot entirely.

> RHEL6 bios now have native ability to boot from virtio and since scsi is
> unsupported we should drom boot=on and extboot entirely.

If boot=on is no longer required or recommended, then it should be removed from KVM command line & thus libvirt won't detect its existence and thus won't use it.

(In reply to comment #15)
> > RHEL6 bios now have native ability to boot from virtio and since scsi is
> > unsupported we should drom boot=on and extboot entirely.
>
> If boot=on is no longer required or recommended, then it should be removed from
> KVM command line & thus libvirt won't detect its existence and thus won't use
> it.

boot=on was a bad name. It actually means extboot=on.
It should only we used for ide interface. With the current qemu cmdline it is too hard to change. I asked libvirt to change that also in rhel5.
I'm sure there is a BZ against libvirt somewhere.

Only now saw that it was already a libvirt bug, reopen + change component back.

Unfortunately just dropping boot=on on drive device when --no-kvm is
used doesn't work in general, I just tried, if using

-drive file=/var/lib/libvirt/images/test6.img,if=none,id=drive-virtio-disk0,boot=on,format=raw

then booting fails with 'Booting from Hard drive' and gets stuck there
but if that boot=on option is dropped, and using

-drive file=/var/lib/libvirt/images/test6.img,if=none,id=drive-virtio-disk0,format=raw

the same domain now fail to boot with

"Booting from Hard Disk..."
Boot failed: could not read the boot disk

No bootable device.

Of course that same exact domain boots fine if using kvm instead of qemu
everything else being equal.
It seems to me the recipe to remove "boot=on" in case of --no-kvm is being
used is not workable, this must be done in a more fine-grained fashing, maybe
per device type.
I'm waiting for complete description of the change to do, in the current state
the suggested approach just does not work for me.

Daniel

What about '-boot c' ?

Actually after updating to seabios-0.5.1-2 the domain will boot ...
very slowly because of the lack of acceleration but that's normal.

Daniel

w.r.t. comment #19

Yaniv, the -boot c option was present in both cases

Daniel

Daniel, do you need more information from me on this issue?

Yes, basically the problem seems not related to ide drive but to -no-kvm being
incompatible with boot=on in any kind of disk. I reproduced the problem
with a domain using only one virtio device.
So the fix for this issue seems to desactivate boot=on if -no-kvm is used

It's independant of the fact that ide drives don't need boot=on

For RHEL-6 I plan to fix only the first problem, because that's the only
one leading to non-booting domains, i.e. really need to be fixed at this point

Daniel

Patch posted upstream and internally

https://www.redhat.com/archives/libvir-list/2010-July/msg00690.html

Daniel

libvirt-0_8_1-21_el6 has been built in RHEL-6-candidate with the fix.

Dave

*** Bug 631655 has been marked as a duplicate of this bug. ***

Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.