machine-types trusty and utopic are not unique (depend on the qemu version)

After more investigation (comparation of the machine description between the ubuntu qemu 2.3 and 2.5), I have seen that in 2.3, the utopic machine is a kvm-2.3 machine while it's a kvm-2.5 machine is 2.5 when it should be kvm-2.3 too.

I will provide a debdiff once I have validated my change.

It's not a debdiff but just the patch I have applied as-is.

I'm afraid there is no "perfect" fix for it - so I need your help evaluating.
And as I assumed it is worse than it seemed at first.

The reason for your finding "I have seen that in 2.3, the utopic machine is a kvm-2.3 machine while it's a kvm-2.5 machine is 2.5 when it should be kvm-2.3 too" is the one being complex.
I assume you likely checked that in the past it was 2.3 on a Vivid host where qemu was on 2.3.

The problem is that the definition always semed to just inherit "latest".
So on Xenial being 2.5 it got "2.5".
On Vivid where it was 2.3 it got 2.3.
IMHO that is wrong back to when it started - utopic actually was "2.1" so that is what it should be (and always have been).

Your fix to make it a 2.3 therefore is only correct for your case where your guest was "last started" on a qemu 2.3 if I'm not mistaken.
It should truly be a 2.1 type.

The worst in all of this is this: If you happened to create that guest back in a day it was 2.1.
But if in the meantime you restarted the guest on let's say the qemu 2.3 Host then it "became" a 2.3 guest (which is wrong). The same is true to the even more relevant trusty type.
But everybody out there might have restarted on different types - so I have to assume that I can find Trusty types on any of 2.0 - 2.5 out there (the same applies to utopic types).

So if we now make it "correctly" a 2.1 type we might still not help you and on top break others :-/
But if we make it a 2.3 type we might only help you and others in the same case as you.
But at the same time we break all others :-/

TL;DR - do fix for those that never restarted; or if not which release of the "restarters" do we fix :-/++

Still thinking ....

ATM - I can't think of a really good way out.
We can fix X/Y/Z releases to expect the "correct" types when seeing the old machine-types.
On top one would have to to the same for all qemus in UCA.

But everybody that had (re)-started a guest "in between" with a newer qemu - think of Ubuntu Cloud Archive that can give Trusty all varieties of Qemu - might have a different version on his guests.

Trying to summarize:
- Affected machine types: trusty and utopic
- Effect: those types are not non-ambiguous, they can be any hw_version from 2.0-2.5 depending on
  under which qemu they were started last time.

Status today - a migration forward fails in some combinations but not in others.
- Trusty as-is (2.0) to Xenial works even with the bug in place.
- Trusty + UCA-Liberty (2.3) to Xenial fails due to the issue.
- other combinations have to be evaluated

Fixing the types to the "correct" ones will fix anybody that never has restarted the guests.
But it might break those that have restarted guests as they have accidentally picked up a too new guest type.

I'll create a ppa for the reporter to check on his case but before getting closer to a real upload we need not only this test but extend the current migration tests to more UCA combinations.

I can do the latter but it needs a bit of time to run and sort out potential unrelated issues.

Luckly the compat function on Trusty was correct on pc_compat_2_0 which might be the reason this works for the trusty type in my tests so far.

Notes for testing:
 - extend to all T-UCA versions
 - Spawn the native T-UCA guest version in those cases to migrate
 - Spawn at least the faulty case with a T-UCA-L + utopic type (from there go to NxM matrix as
   needed and time permits)

I prepped a ppa with the "correct" fix (utopic type being 2.1) at:
https://launchpad.net/~paelzer/+archive/ubuntu/bug-1641532-tu-machine-type-ambiguous

All three Releases (Xenial, Yakkety, Zesty) are building right now in the ppa and should be available in about an hour.
Note @ UCA Team - if that works we will need to coordinate on similar fixes for UCA-I through UCA-L.

@sileht: I'd be interested if that version in the ppa would make it work for you as well or if we are really in the deadlock of your source being 2.3 and therefore needing a 2.3 in your case.

I still got the same error with the ppa:

2016-11-14T13:11:15.479395Z qemu-system-x86_64: Unknown ramblock "/rom@etc/acpi/rsdp", cannot accept migration
2016-11-14T13:11:15.479419Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
2016-11-14T13:11:15.479613Z qemu-system-x86_64: load of migration failed: Invalid argument

I think my utopic type is a 2.3, because I have a custom qemu package that use 2.3 instead of 2.1 for utopic and live migration works with it.

Sorry it took so long, but I'm happy that I finally found the right people to talk about it, summarizing the issue once more and then subscribing a bunch of people to make the right call for the UCA on this.

Affected:
- Trusty type guests in Liberty/Mitaka/Xenial/Yakkety/Zesty
- Utopic type guests in Liberty/Mitaka/Xenial

Status today - a migration forward fails only in some combinations but not in others.
- Trusty as-is (2.0) to Xenial works even with the bug in place (accidentally/fortunately)
- Trusty + UCA-Liberty (2.3) to Xenial fails due to the issue (reported case).
- other combinations have to be evaluated

Cause:
- the way machine types are defined got changed several times in qemu along the updates Trusty to Xenial
- It seem that there was an error added when porting the machine types to Wily
- This issue got carried forward when the conversion to Xenial was made
- Due to that those types are not non-ambiguous, they inherit the current qemus version
- So instead of Trusty being always 2.0 as it should be, it is 2.x matching the hosts qemu version
- The same applies to the utopic type

Challenges on fixing that:
- Testing showed that fixing on Xenial did not help - especially for the reported case - it expected 2.1 (utopic) instead ot 2.5 after the fix, but the source was Liberty which is 2.3)
- Since it seems it needs fixing on the source of the migration as well it needs
- In general to "pick up" the fix you need to (re-)start the guest
- This has the additional impact that the guest visible virtual hardware will change, so e.g. once we fix Xenial/Liberty to expect trusty to be 2.0 as it should be - any guests that were started as 2.5 type before will restart as 2.0 type and could cause further issues which makes that fix super critical. And especially going down versions might be critical.
- Due to the fact that Trusty->Xenial works fine, we might think on doing only a fix to UCA-Liberty and avoid these new issues by fixing the old one.
- There might be other even more complex approaches, but then equally more error prone, we have to decide on what we need here

Actions that are to be evaluated:
- Create a fix for Liberty (and eventually Kilo)
- Test if that fixes it without restarting a guest (unlikely but might help and would be great)
- Test if that fixes it with restarting the guest (that should work for sure at least)
- After that fix of the more immediate issue we have to think on >=Xenial preparation with the lowest impact to users due to restarting into lower machine type levels or later migration.

Note: restart ~ start/stop in this context. If you keep a QEMU process alive it is not going to change the machine type. It is clear from the comments but just to make sure.

Testing of migrations from liberty with the following patch were successful. However we may need to hold off on uploading to liberty until later releases are fixed. http://paste.ubuntu.com/23827800/

This bug was fixed in the package qemu - 1:2.8+dfsg-2ubuntu1

---------------
qemu (1:2.8+dfsg-2ubuntu1) zesty; urgency=medium

  * Merge with Debian; remaining changes:
    - add qemu-kvm init script and defaults file
      (d/qemu-system-common.qemu-kvm.*)
    - d/rules, d/qemu-kvm-init: add and install script loading kvm
      modules and handling /etc/default/qemu-kvm
    - qemu-system-common.preinst: add kvm group if needed
    - Enable nesting by default on intel.
      - set default module option
      - re-load kvm_intel.ko if it was loaded without nested=1
      - d/p/ubuntu/expose-vmx_qemu64cpu.patch: enable nested kvm by
        default in qemu64 cpu type.
    - Enable svm by default for qemu64 on amd
    - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
      types to ease future live vm migration.
    - Make qemu-system-common depend on qemu-block-extra
    - Make qemu-utils depend on qemu-block-extra
    - s390x support
      - Create qemu-system-s390x package
      - Include s390-ccw.img firmware
    - qemu-system-common.postinst:
      - change acl placed by udev, and add udevadm trigger.
      - d/control-in: change dependencies for fix of wrong acl for newly
        created device node on ubuntu
    - have qemu-system-arm suggest: qemu-efi; this should be a stronger
      relationship, but qemu-efi is still in universe right now.
    - d/qemu-kvm-init, d/kvm.powerpc, d/control-in: check SMT on ppc64el
    - Several changes were applied but missing in the changelog so far
      - d/qemu-system-ppc.links provide usr/bin/qemu-system-ppc64le symlink
      - arch aware kvm wrapper
      - update VCS links
      - no more skip disable libiscsi on Ubuntu
      - let qemu-utils recommend sharutils
      - disable x32 architecture
  * Dropped Changes:
    - Several changes were applied but missing in the changelog so far
      but are no more needed
      - no pie for relocatable LD calls, with toolchain defaulting to
        pie (fixed upstream)
      - enable libnuma-dev (now in Debian)
      - transition for moved init scripts (can be dropped after LTS
        containing >=2.5 which is Xenial)
      - --enable-seccomp related whitespace change (had no effect)
    - apport hook for qemu source package (In Debian)
    - add upstart script (d/qemu-system-common.qemu-kvm.upstart)
    - d/qemu-system-x86.maintscript: transition off of
      /etc/init.d/qemu-system-x86 (can be dropped after Xenial)
    - Enable pie by default, on ubuntu/s390x. (Is the default since
      >=Xenial, no cloud archive backport <=Xenial to consider)
    - no pie for relocatable LD calls (fixed upstream in commit
      7ecf44a5)
    - CVEs: CVE-2016-5403, CVE-2016-6351, CVE-2016-6490 (now Upstream)
    - Revert fix for CVE-2016-5403, causes regression see USN-3047-2.
      (Improved fix included by upstream)
    - Enable GPU Passthru for ppc64le (is upstream in qemu 2.7)
    - Fixed wrong migration blocker when vhost is used (is upstream in
      qemu 2.8)
  * Added Changes:
    - d/rules, d/control-in: avoid people editing d/control by warning
      header and non writable permissions
    - fixed moving trusty machine type definition whic...

Full path tests were done along the qemu 2.8 merge that now passed.
I'm uploading the test logs - with those in place I can finally again migrate all combinations from T or T+UCA through X->Y->Z and back til X (note back working bur not supported).tar -czf.

These are the log of the full path tests using ppas for UCA and Xenial/Yakkety as well as the now migrated qemu of Zesty.

Since Zesty now is ready we have opened a window where migrations of a Trusty Type guest on Xenial can not be migrated to Zesty (expected). Yet the zesty fix was a prereq for the SRU itself.

Lets put the Xenial and Yakkety fixes into SRU queue now.

@corey.bryant - the same it true for the UCA-Liberty fix time gets close that this shall be released.

I uploaded the fixes to the unapproved queue, waiting for an SRU pass to come by now.

The plan is to run the full multi-arch test suite once more against proposed for verification then.
@corey-bryant: you might then run (part of) your matrix as well once in proposed right?

But now waiting for SRU review first.

Hello Mehdi, or anyone else affected,

Accepted qemu into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu10.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Mehdi, or anyone else affected,

Accepted qemu into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.6.1+dfsg-0ubuntu5.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

FYI - testing of the full chain T/X/Y/Z in proposed for verification is in progress

Full chain is working fine as expected.
But I'm still holding back to mark v-d too early as I hit an autopkgtest of open-iscsi on yakkety (only there).
I'm 98% sure it is unrelated, but I want to make sure first.

Locally in adt the test works just fine - I contacted a few people I thought I heard discussing about something very similar.
I think we are safe considering the update good, but I'll wait the weekend for replies on my inquiry on known-issues of that testcase.

Hi,
I contacted smoser who was hacking on open-iscsi recently due to related reasons.
I'll quote his mail for the record, but the TL;DR is - we should be fine ignoring the fail results of open-iscsi in yakkety.

Reasons:
- Test known to be unreliable
- works when reproduced in local ADT
- Xenial test is set to ignored failure as well

Actions:
- I think this should be marked ignored failure as well in Yakkety

Xenial: http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#qemu
Yakkety: http://people.canonical.com/~ubuntu-archive/proposed-migration/yakkety/update_excuses.html#qemu

Quote of Scotts detailed reply at http://paste.ubuntu.com/24032648/

With open-iscsi in Yakkety being ignorable and all other tests are good in autopkgtest, QA and Migration tests setting verification-done.

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.9

---------------
qemu (1:2.5+dfsg-5ubuntu10.9) xenial; urgency=medium

  * fix ambiguous machine trusty and utopic machine types (LP: #1641532)
    - d/p/ubuntu/define-ubuntu-machine-types.patch update type definitions
    - d/qemu-system-x86.NEWS to describe the issue

 -- Christian Ehrhardt <email address hidden> Mon, 30 Jan 2017 08:04:27 +0100

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:2.6.1+dfsg-0ubuntu5.3

---------------
qemu (1:2.6.1+dfsg-0ubuntu5.3) yakkety; urgency=medium

  * fix ambiguous machine trusty and utopic machine types (LP: #1641532)
    - d/p/ubuntu/define-ubuntu-machine-types.patch update type definitions
    - d/qemu-system-x86.NEWS to describe the issue

 -- Christian Ehrhardt <email address hidden> Mon, 30 Jan 2017 08:07:06 +0100

Uploaded to liberty-staging ppa for the cloud archive. That'll need to be promoted to liberty-proposed once it builds successfully.

Hello Mehdi, or anyone else affected,

Accepted qemu into liberty-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:liberty-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-liberty-needed to verification-liberty-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-liberty-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Verified that with the fix "on both ends" we can not properly migrate from Liberty onwards.
The Vivid default type won't migrate post Xenial but that is expected as we only kept Trusty.
I also spawned a Trusty type on UCA-L and it migrated through up to Zesty as it should.

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:2.3+dfsg-5ubuntu9.4~cloud4
---------------

 qemu (1:2.3+dfsg-5ubuntu9.4~cloud4) trusty-liberty; urgency=medium
 .
   * Fix ambiguous machine types (LP: #1641532)
     - make utopic a true 2.1 type (was floating with qemu version)
     - drop duplicate ubuntu alias from utopic (should only point to vivid)

@james-page, Any ETA on this package arriving in mitaka-updates ? It is already in mitaka-proposed. Some other fixes are waiting on this SRU to be completed for UCA. I'm afraid that some other SRU preempts, again, the qemu fix for LP: #1626972. Thank you!