[Ubuntu 18.04] P8 compat mode migration fails for Ubuntu 16.04.4 P8 Guest from Ubuntu 16.04.4 P8 Host -> Ubuntu 18.04 P9 Host

I tested P8->P8 but couldn't P8->P9 lacking the HW in the same environment.
Thanks for spotting that issue, taking a look at backportability of this ...

The simply said "its dependencies" seems to be a dependency hell.
  72fdd4de8e5 spapr: register dummy ICPs later
referred to fix patch:
  72194664c8a1 spapr: use spapr->vsmt to compute VCPU ids
is not in Qemu 2.11, nor is the one this fixes
  8904e5a75005 spapr: Adjust default VSMT value for better migration compatibility

Without testing it seems we would need:
- ? marks part of the series I'm unsure it is a hard requirement but it seems entangled enough vsmt/vcpuid to consider it as well to not break functionality :-/
- f#<num> marks which other change it fixes (implicit depends)
- r#<num> marks which other change depends on it

#1 ? r#2 1a5008fc spapr: harden code that depends on VSMT
#2 f#7 72fdd4de spapr: register dummy ICPs later
#3 ? f#4 b1a568c1 spapr: fix missing CPU core nodes in DT when running with TCG
#4 ? 5d0fb150 spapr: consolidate the VCPU id numbering logic in a single place
#5 ? 14bb4486 rename spapr_vcpu_id() to spapr_get_vcpu_id()
#6 ? 648edb64 spapr: move VCPU calculation to core machine code
#7 f#9 72194664 spapr: use spapr->vsmt to compute VCPU ids
#8 f#9 4ad64cbd spapr: set vsmt to MAX(8, smp_threads)
#9 f#2.11 8904e5a7 spapr: Adjust default VSMT value for better migration compatibility
#10 r#9 1f20f2e0 spapr: Allow some cases where we can't set VSMT mode in the kernel

#1 has too much noise and isn't a hard requirement, so I skipped it
There also was some noise for:
- max_vthreads being renamed
- on #7 for threads[0]
- on #6 for POWERPC_CPU
- on #6 for spapr_caps_pre_load
but that was ok.

Overall backport in packaged format would be:
- https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=416b00af3a8bbfc26094c4a3eba0bce09888af6c
- https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=0c93d6500d13e9eeefcf8797e0f05e0225bf9438

I made it part of the ppa that was open for bug 1762854 - so you can test both at once.
=> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3233

So it seems doable, but it is quite a huge change still.
9 patches with overall:
 hw/ppc/spapr.c | 160 ++++++++++++++++++++++++++++++------------------
 hw/ppc/spapr_cpu_core.c | 9 --
 include/hw/ppc/spapr.h | 3

OTOH it is ppc64 only so it is your (=IBM) call to make, do you want to use the backport or do you think there is a much smaller fix (changing 1-5 lines somewhere) for qemu 2.11 available to "fix just the issue"?
This is a tradeoff, if it becomes too complex then the backport is clearly preferred, but if there would be a way to just handle this more comprehensively without introducing all the new VSMT handling first let me know.

TL;DR I'd ask you to:
- evaluate if a (much) simpler fix would be available
- if not please review the backport to fit your needs
- also then verify the ppa provided so that we can go forward

Setting status to incomplete as I wait for feedback on that.

------- Comment From <email address hidden> 2018-04-13 10:57 EDT-------
(In reply to comment #6)
> TL;DR I'd ask you to:
> - evaluate if a (much) simpler fix would be available

Yeah, I believe we can come up with something slightly simpler.

The kernel for 18.04 is 4.15, hence KVM supports setting the VSMT, and we
can drop all the patches that deal with kvmppc_smt_threads() != spapr->vsmt.

This means the following patches can be dropped from the backport:

lp-1763468-1-spapr-Allow-some-cases-where-we-can-t-set-VSMT-mode-.patch
lp-1763468-4-spapr-use-spapr-vsmt-to-compute-VCPU-ids.patch

These ones are cleanup and can be dropped:

lp-1763468-5-spapr-move-VCPU-calculation-to-core-machine-code.patch
lp-1763468-6-spapr-rename-spapr_vcpu_id-to-spapr_get_vcpu_id.patch
lp-1763468-7-spapr-consolidate-the-VCPU-id-numbering-logic-in-a-s.patch

This one fixes a regression from the above cleanup and can be dropped:

lp-1763468-8-spapr-fix-missing-CPU-core-nodes-in-DT-when-running-.patch

But we still need to adjust the default VSMT value, so we want to keep these:

lp-1763468-2-spapr-Adjust-default-VSMT-value-for-better-migration.patch
lp-1763468-3-spapr-set-vsmt-to-MAX-8-smp_threads.patch

and we also need to register dummy ICPs later, to "fix just the issue". This requires
to adapt lp-1763468-9-spapr-register-dummy-ICPs-later.patch to the fact that
xics_max_server_number() doesn't take arguments. The following does the trick:

perl -i -p -e 's/xics_max_server_number\(spapr\)/xics_max_server_number()/' \
lp-1763468-9-spapr-register-dummy-ICPs-later.patch

So you end up with 3 patches and a nicer diffstat:

hw/ppc/spapr.c | 35 ++++++++++++++++++++---------------
1 file changed, 20 insertions(+), 15 deletions(-)

Is it enough for you to go forward ?

------- Comment From <email address hidden> 2018-04-16 15:39 EDT-------
FYI, I could talk with Michael Roth, maintainer of QEMU stable: QEMU 2.11.2 will ship the 3 patches mentioned in comment #7.

Thanks Greg for taking a look with me.
We unfortunately can't rely on kernel 4.15 due to the fact that the virt stack in 18.04 will be made available to Xenial via the Ubuntu Cloud Archive [1].

So this could run on hosts as much back as 4.4
With that said we would still need patches -1 and -4.

You said -5, -6 and -7 (plus -8 as fix) are only cleanup so those we might drop.
Due to the dependencies outlined before I kept them in the series, but I'm happy to follow your guidance on that.

And with -1, -2, -3, -4, - 9 there is not even noise on xics_max_server_number.

I made a respin of the build in the ppa [2] with that.
Please verify if that suits your needs for:
- this bug 1763468
- other current ppc bug 1762854

[1]: https://wiki.ubuntu.com/OpenStack/CloudArchive
[2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3233

------- Comment From <email address hidden> 2018-04-17 10:22 EDT-------
(In reply to comment #9)
>
> And with -1, -2, -3, -4, - 9 there is not even noise on
> xics_max_server_number.
>

Patch list was reviewed and tested internally. You can go ahead with this fix.

Thanks for the verify, pushed to Bionic-unapproved

Got P9 HW myself:
current qemu:
a) P8->P9 worked
b) P8->P9->P8 failed (silently)

proposed qemu for this bug:
a) P8->P9 worked
b) P8->P9->P8 worked

P9->P8 also working if one sets the right compat, like for example
Default on P9 guest:
cpu : POWER9 (architected), altivec supported
revision : 2.2 (pvr 004e 1202)
Set compat:
 <cpu mode='host-model' check='partial'>
   <model fallback='allow'>power8</model>
 </cpu>
Then is is in the guest:
cpu : POWER8 (architected), altivec supported
revision : 2.2 (pvr 004e 1202)
And this migrates P9->P8

+1 on confidence of the proposed upload

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7

---------------
qemu (1:2.11+dfsg-1ubuntu7) bionic; urgency=medium

  * d/p/ubuntu/lp-1762854-*: fix issue with SCSI-2 devices denying Protection
    information (LP: #1762854).
  * d/p/ubuntu/lp-1763468-*: fix VSMT handling to fix ppc64el P8/P9 migration
    (LP: #1763468).

 -- Christian Ehrhardt <email address hidden> Wed, 11 Apr 2018 07:46:18 +0200

------- Comment From <email address hidden> 2018-04-30 02:34 EDT-------
(In reply to comment #14)
> This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7
>
> ---------------
> qemu (1:2.11+dfsg-1ubuntu7) bionic; urgency=medium
>
> * d/p/ubuntu/lp-1762854-*: fix issue with SCSI-2 devices denying Protection
> information (LP: #1762854).
> * d/p/ubuntu/lp-1763468-*: fix VSMT handling to fix ppc64el P8/P9 migration
> (LP: #1763468).
>
> -- Christian Ehrhardt <email address hidden> Wed, 11 Apr 2018
> 07:46:18 +0200

I observe issue with loading cpu state in target now with qemu (1:2.11+dfsg-1ubuntu7),

# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [ 99 %]error: internal error: qemu unexpectedly closed the monitor: 2018-04-30T06:29:00.728297Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-04-30T06:29:00.735891Z qemu-system-ppc64: load of migration failed: Operation not permitted

System configuration:

Ubuntu 18.04 P9 Host:
Kernel - 4.15.0-20-generic
Qemu - 1:2.11+dfsg-1ubuntu7
SLOF - 20170724+dfsg-1ubuntu1
Libvirt - 4.0.0-1ubuntu8

Ubuntu 16.04.4 P8 Host:
Kernel - 4.13.0-39-generic
Qemu - 1:2.5+dfsg-5ubuntu10.25
SLOF - 20151103+dfsg-1ubuntu1.1
Libvirt - 1.3.1-1ubuntu10.21

Ubuntu 16.04.4 P8 Guest:
Kernel - 4.13.0-39-generic

-- Bala

Hi Bala,
interesting, but not 100% the same case IMHO.

The former fix and discussion was about 18.04 on P8 to 18.04 on P9 and it worked fine after the fixes that were pointed out were integrated.

I'd ask you to file a new bug (internally for you and your PPC dev's as well as with launchpad for us) to discuss potential needs for 16.04 on P8 to 18.04 on P9. Only then I think tracking and discussions will go well.

The old title defies my understanding of this case thou, never the less all changes so far were to 18.04.
And on confirmation it was 18.04 broken -> 18.04 working after the fix.
I'll set up a few systems to try if they are free atm.
But a new bug would still be appreciated.

------- Comment From <email address hidden> 2018-04-30 03:16 EDT-------
(In reply to comment #16)
> Hi Bala,
> interesting, but not 100% the same case IMHO.
>
> The former fix and discussion was about 18.04 on P8 to 18.04 on P9 and it
> worked fine after the fixes that were pointed out were integrated.

This Bug was originally raised for issue in migrating Ubuntu 16.04.4 P8 Guest on Ubuntu 16.04.4 P8 Host -> Ubuntu 18.04 P9 Host.

>
> I'd ask you to file a new bug (internally for you and your PPC dev's as well
> as with launchpad for us) to discuss potential needs for 16.04 on P8 to
> 18.04 on P9. Only then I think tracking and discussions will go well.

I think the potential reason is that, it is the basic usecase for a customer currently having Ubuntu 16.04.4 P8 Guest on Ubuntu 16.04.4 P8 Host, wants to move the guest to P9 Host or in cloud if the migration takes place in automated way after plugging in P9 servers in racks migration should be seamless.

-- Bala

------- Comment From <email address hidden> 2018-04-30 04:02 EDT-------
(In reply to comment #18)
> The old title defies my understanding of this case thou, never the less all
> changes so far were to 18.04.
> And on confirmation it was 18.04 broken -> 18.04 working after the fix.
> I'll set up a few systems to try if they are free atm.
> But a new bug would still be appreciated.

I second Bala's statements: the list of fixes was decided with 16.04.4 -> 18.04 migration in mind... I'm now working with Bala to investigate.

Ok, if that was what you had in mind when selecting patches fine.
Thanks for taking a look.

I'll for tracking eventually fork a bug number for the remaining issue on this then, but this can wait until it is discussed and pre-tested here.

I wanted to take a look at this again, but TBH at the moment all I've got was crashing my P9 system three times. I'll start a mail thread on it and keep this bug clean of it for now.

While resolving that I hope that Bala/Greg will find something about the issue here on their side.

------- Comment From <email address hidden> 2018-05-02 08:34 EDT-------
After working with Greg, we found that migration of Ubuntu 16.04.4 guest from Ubuntu 16.04.4 P8 Host to ubuntu 18.04 P9 host works but we need to explicitly boot the guest in source with compat settings even on P8 host, ie

<cpu mode='host-model'>
<model>power8</model>
</cpu>

As Greg stated, this is because Ubuntu 16.04.4 use older qemu (Qemu 2.5) that boots the guest in raw mode where as Ubuntu 18.04 use newer Qemu (Qemu 2.11) that boots the guest in architected mode, which means if the guest runs in raw mode, then migration is only supported between identical hosts and if it run in architected mode, the guest can be migrated to any host that also supports this architected mode.

In our case we try to migrate from raw mode to architected mode that's why we have to give explicit compat setting is source so that respective qemu command will be formed with compat setting in target for migration to start. This is limitation we have from past and Greg would be working on it to fix.

We can safely close this bug but with a proper documentation to make sure this limitation is known to the user.

But still backward migration is broken, migrating back Ubuntu 16.04.4 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 Host is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu is not compatible with older qemu. So this is a new issue with different root cause for which I would raise a separate bug.

Thanks Greg.

-- Bala

------- Comment From <email address hidden> 2018-05-23 02:33 EDT-------
Hello Canonical,
FYI we have now closed this bug by documenting the limitation as below. Thanks!

https://wiki.ubuntu.com/ppc64el/uKVM#Limitation_in_migrating_POWER8_guest_from_POWER8_host_to_POWER9_host

Thanks Iranna for the update, the wiki entry LGTM