virtualization/kvm_check_vm test corrupts pre-downloaded image

Steps to reproduce:

bzr branch lp:checkbox
wget http://cloud-images.ubuntu.com/raring/current/raring-server-cloudimg-i386-disk1.img
cp raring-server-cloudimg-i386-disk1.bak
# edit /etc/checkbox.d/virtualization.cfg and put the following:
# image: /path/to/raring-server-cloudimg-i386-disk1.img
# timeout: 30
sudo checkbox/checkbox-old/scripts/virtualization kvm --debug
# Will Boot successfully the first time
sudo checkbox/checkbox-old/scripts/virtualization kvm --debug

Expected result: "Booted successfully" for the second time
Actual result: KVM instance failed to boot

this is due to a problem mounting filesystems, as seen in virt_debug:

[ 2.754950] init: mountall main process (284) killed by FPE signal
General error mounting filesystems.
A maintenance shell will now be started.
CONTROL-D will terminate this shell and reboot the system.

also, as Yung reported, md5sum raring-server-* will show that the image is different from what the original one had, although that doesn't necessarily mean it was corrupted, maybe it was not unmounted properly and is now damaged.

OK, this was a fun one to debug.

The test was designed to download a fresh cloud image every time, that's why we hadn't noticed the behavior where the VM is "corrupted" after the first boot. This problem is evident when using a local cloud image, as it's a writable disk image so changes are persistent.

What happens is that our test waits a fixed amount of time, then simply terminates qemu forcefully, the equivalent of shutting down the VM ungracefully, which for some reason (outside the scope of our purposes) causes it to spit those errors out when it reboots.

This was easily solved by amending our cloud-init file to have a power_state stanza. This causes the VM to boot, perform the initialization process, then power_state halts it (not powering off, just want it to leave it in a state where yanking the plug won't damage the disk), and then when we do exactly that, the disk is left in a state that will boot successfully the next time.

Once that was fixed, the VM still "failed to boot". The reason is that the script looks for the "END SSH HOST KEYS" string to determine whether the VM booted correctly. However, this string is only produced on the *first* boot, when creating those keys; subsequent boots will NOT have this string. This was fixed by using the "final_message" cloud-init option, so we have a consistent message when boot is completed successfully. I'm still checking for END SSH HOST KEYS to identify the first boot.

With the proposed changes, the VM can be reused many times for testing.

Another option, which I didn't implement, is to simply copy the .img file to a temporary file, and use that to perform the testing. That way we always have the "pristine" image as a starting point. But this is just a proof of concept which I didn't really work on as it felt more complicated.