eoan: ppc64el install on pseries-eoan VM fails to install

Seems like we hit an illegal instruction in libc. I used to be able to install previous releases in a VM, so maybe gcc is emitting code that the qemu can't emulate correctly.

Oct 11 18:42:46 localechooser: Generation complete.
Oct 11 18:42:46 kernel: [ 820.757132] debconf-set-sel[11139]: illegal instruction (4) at 7f8874cefe68 nip 7f8874cefe68 lr 7f8874cefe48 code 1 in libc-2.30.so[7f8874c80000+210000]
Oct 11 18:42:46 kernel: [ 820.757238] debconf-set-sel[11139]: code: 40820008 eb6ffffa 7f23cb78 4bffeaa5 2c030030 41820aa4 2c030035 38e00001
Oct 11 18:42:46 kernel: [ 820.757260] debconf-set-sel[11139]: code: 41820a98 810d8f98 75090080 418215bc <fc18048e> 7c080066 790807a0 e8bf0076
Oct 11 18:42:46 base-installer: /usr/lib/post-base-installer.d/05preseed: return: line 46: Illegal number: tasksel
Oct 11 18:42:46 base-installer: warning: /usr/lib/post-base-installer.d/05preseed returned error code 2
Oct 11 18:42:46 base-installer: warning: /usr/lib/post-base-installer.d/05tzsetup returned error code 10

I'm not sure what to think about this failure; what's clear is it's not 'ubiquity', because we're dealing with debian-installer, but I'm not sure what component. In fact, it looks more subtle than this, some sort of more obscure failure:

Oct 11 18:42:06 base-installer: 'Ubuntu-Server 19.10 _Eoan Ermine_ - Beta ppc64el (20191011)'
Oct 11 18:42:06 base-installer: Copying package lists...
Oct 11 18:42:08 base-installer: Illegal instruction
Oct 11 18:42:08 kernel: [ 782.792307] find[10970]: illegal instruction (4) at 77a45c98baa0 nip 77a45c98baa0 lr 77a45c98cf00 code 1 in libc-2.30.so[77a45c930000+210000]
Oct 11 18:42:08 kernel: [ 782.793461] find[10970]: code: fb81ffe0 fba1ffe8 fbc1fff0 fbe1fff8 f821ffa1 7c7d1b78 7c9f2378 812d8f98
Oct 11 18:42:08 kernel: [ 782.793531] find[10970]: code: 7cbc2b78 7cdb3378 75290080 41820344 <fc18048e> 7c1e0066 7bde07a0 2c3ffc02
Oct 11 18:42:08 base-installer: gpgv: Signature made Fri Oct 11 06:47:32 2019 UTC
Oct 11 18:42:08 base-installer:
Oct 11 18:42:08 base-installer: gpgv: using RSA key D94AA3F0EFE21092
Oct 11 18:42:08 base-installer:
Oct 11 18:42:08 base-installer: gpgv: Can't check signature: No public key
Oct 11 18:42:08 base-installer:
Oct 11 18:42:08 base-installer: Sub-process apt-key returned an error code (2)
Oct 11 18:42:08 base-installer: ^MReading Package Indexes... 0%^M
Oct 11 18:42:10 base-installer: ^MReading Package Indexes... Error!^M
Oct 11 18:42:10 base-installer:
Oct 11 18:42:10 base-installer: Writing new source list

There are various instances of 'Illegal instructions' in syslog and errors we really shouldn't be seeing.

What type of CPU configured to virtualize ppc64el? The exact -cpu option in use might be the culprit here.

I was using the pseries-eoan virt-manager machine type, not sure what that maps to as a CPU though.

Trying to help with the cpu type here - the default for the mentioned config will be:
/usr/bin/qemu-system-ppc64le -name guest=ubuntu19.10-ppc64le-test,debug-threads=on ...
  -machine pseries-eoan,accel=tcg,usb=off,dump-guest-core=off ...

Without an explicit -cpu option set.

The libvirt XML just has no <cpu> section as all.

The default qemu picks in that case in the past was:
PowerPC power8_v2.0 PVR 004d0200
PowerPC power8 (alias for power8_v2.0)

Checking the qemu code shows:
4363 static void spapr_machine_3_1_class_options(MachineClass *mc)
4370 mc->default_cpu_type = POWERPC_CPU_TYPE_NAME("power8_v2.0");

With the code we can map this
(qemu versions go into the machine type, so xenial would be 2.5, bionic 2.11, ...):
new: power9_v2.0
<3.1 power8_v2.0
<2.7 power7_v2.3

So in the setup here it should be power9_v2.0 while in any release before Eoan it would have been power8_v2.0.

I'd expect P9 should be P8 compatible, but maybe that helps in debugging this.
Downloading a ppc64 image for testing on my own ...

Step #1 - repro:
Repro the case as is - running ppc64el install in TCG on Eoan with all settings on default.
The system reports (proc/cpuinfo) POWER9 2.0 pvr 004e 1200 (just as expected).
And I have a bunch of illegal instructions hitting 4 binaries but essentially always being in libc2.30.so.

But I can run e.g. find in a shell without hitting that illegal instruction "again".
Something yet unknown is needed to hit these illegal instructions.
Maybe something special about the install environment that I haven't considered yet.

Step #2 - use old cpu type (p8 instead of p9)
By setting the pseries-disco machine type we know (see above) that we will get the formerly used "power8_v2.0" cpu type.
Running the Eoan iso install with that seems to work fine for me!

Step #3 - use old image (Disco instead of Eoan)
It is worth to check if this might be something "new" in Eoans glibc that triggers this or if more likely the switch of the emulated power8->power9 chip causes this. Therefore I ran a Disco ISO install in TCG with the defaulr (power9) cpu.
The install of that works fine as well.

The most likely theory that matches the above results would be that:
1. glibc in Eoan has some "if P9, then new code"
2. qemu TCG for ppc64el fails on that new code

TCG never was very important (it always is best effort and non production use), and users seem to have a workaround by configuring former machine types or explicitly a power8 cpu type.
=> Prio: low.

OTOH we could patch qemu in Eoan (needs to be done super fast thou) to default to P8 cpu on the pseries-eoan type. IMHO we need IBM to guide us on this.

Question-@IBM:
a) do you want us to switch the default emulated CPU to from power9_v2.0 to power8_v2.0 (If so we need the answer asap)?
b) are there known missing/bad instructions in P9-TCG that we should fix by backporting changes?

While not too important if it only affects running under qemu's emulation, I added a glibc task as someone maybe should take a look at this on bare metal to be sure that the p9 code in glibc isn't generally broken (there could be issues that just didn't surface yet).

And as assumed before, it clearly isn't a D-I issue - mark that task as invalid.

Need to investigate to find which P9 instructions is said to be invalid - then investigation in TCG code to ensure it is yet supported will be done.

No objection from me Thierry, the odd part is that e.g. find has failed (we see that in syslog) but when I later run find it works just fine. So the problem is how to exactly to trigger the issue for further debugging.

To get a better hold of the actual bad instruction I installed Ubuntu 19.10 in a guest using the 3.1 based machine type. Then after the full install (much more capable than the busybox of the installer) I switched the machine type and booted up with the new machine type that also implies the power9 cpu.

With that it still boots up, but I've seen a few illegal instructions.

On login I see /etc/update-motd.d/98-fsck-at-reboot, also apt install of gdb failed.
In the journal I have found 8 more.

I can't install anything, so it seems I need to switch back and forth between CPU types whenever I need anything on top ...
But now (full install) fortunately the crash collection in /var/crash worked so I have a few dumps to check.

With the older emulated CPU I then went into debugging these cases to check for a common pattern ...

All fails I had were on sysdeps/powerpc/fpu/get-rounding-mode.h:30 which is:

 27 static inline int
 28 get_rounding_mode (void)
 29 {
 30 return _FPU_GET_RC ();
 31 }

And that in turn is defined as:
 83 # ifdef _ARCH_PWR9
 84 # define _FPU_GET_RC() _FPU_GET_RC_ISA300()
 85 # elif defined __BUILTIN_CPU_SUPPORTS__
 86 # define _FPU_GET_RC()»»···»···»···»···»···»···\
 87 ({fpu_control_t __rc;»»···»···»···»···»···»···\
 88 __rc = __glibc_likely (__builtin_cpu_supports ("arch_3_00"))»···\
 89 ? _FPU_GET_RC_ISA300 ()»··»···»···»···»···»···\
 90 : _FPU_GETCW (__rc) & _FPU_MASK_RC;»··»···»···»···\
 91 __rc;»··»···»···»···»···»···»···»···\
 92 })
 93 # else
 94 # define _FPU_GET_RC()»»···»···»···»···»···\
 95 ({fpu_control_t __rc = _FPU_GETCW (__rc) & _FPU_MASK_RC;»·\
 96 __rc;»··»···»···»···»···»···»···\
 97 })
 98 # endif

In assembly layout that is on

  beq 0x...
> mffsl 0
  mfvsrd r8,vs0

That seems like this code:
 75 # define _FPU_GET_RC_ISA300()»··»···»···»···»···»···\
 76 ({union { double __d; unsigned long long __ll; } __u;»»···»···\
 77 __asm__ __volatile__(»··»···»···»···»···»···\
 78 ".machine push; .machine \"power9\"; mffsl %0; .machine pop" »\
 79 : "=f" (__u.__d));»···»···»···»···»···»···\
 80 (fpu_control_t) (__u.__ll & _FPU_MASK_RC);»·»···»···»···\
 81 })

=> https://git.qemu.org/?p=qemu.git;a=commit;h=31eb7dddacc242af244798854444c3497b515e9f maybe?

That isn't even in 4.1 but it might be related as it is about mffsl.

There might be much more needed than just that, so I'm waiting for a comment by IBM now that I identified the instruction that fails.

Biggest question to IBM probably is, are there p9 HW machines out there that lack mffsl?
Like a DD1.0 or a other PVR that is P9 but lacks the mffsl and would fail the same way the TCG does?

Attached a few crash files (which already had retraces ran) if someone needs them.

------- Comment From <email address hidden> 2019-10-15 10:04 EDT-------
From Paul:

I added TCG support for mffsl, mffscrn, mffscrni, mffsce to QEMU, and I think they will appear in QEMU 4.2, which is not yet released.

I found that the support for these instructions were missing because of my efforts to test work I did for GLibC to use those instructions in GLibC 2.30 and GLibC 2.31. It appears that eoan has GLibC 2.30 and has hit these instructions.

Everything I can find says that these instructions exist in DD1 and later versions of POWER9.

> a) do you want us to switch the default emulated CPU to from power9_v2.0 to power8_v2.0 (If so we need the answer asap)?

I'm not sure what the decision criteria is here, but I'll try to offer some information below.

> b) are there known missing/bad instructions in P9-TCG that we should fix by backporting changes?

Making some presumptions...you are looking for how to make QEMU work out of the box for a ppc64el TCG guest. I would hazard that there is some risk to backporting the instructions to QEMU 4.0, but I can't quantify that easily. I think it would be low, but I'm really not that familiar with the QEMU code base.

The other consideration is that if you switch the default, you've perhaps solved the problem in the default case, but POWER9 support would still be broken if somebody tries to use it.

It would seem like the most comprehensive solution is to backport.

BUT, I can't speak to whether there are other instructions missing between QEMU 4.0 and 4.2. :-/ Maybe Leonardo can better speak to a preferred path forward.

> I added TCG support for mffsl, mffscrn, mffscrni, mffsce to QEMU, and I
> think they will appear in QEMU 4.2, which is not yet released.

Yeah I have found these.

> I found that the support for these instructions were missing because of
> my efforts to test work I did for GLibC to use those instructions in
> GLibC 2.30 and GLibC 2.31. It appears that eoan has GLibC 2.30 and has
> hit these instructions.

Yep that is how we hit it.

> Everything I can find says that these instructions exist in DD1 and
> later versions of POWER9.

I'm glad to hear this, having it to work in TCG is nice but not important.
If there would have been HW failing on this glibc would have needed to
be changed.
Thanks for that info.

> > a) do you want us to switch the default emulated CPU to from
> power9_v2.0 to power8_v2.0 (If so we need the answer asap)?
>
> I'm not sure what the decision criteria is here, but I'll try to offer
> some information below.

We are too late anyway, since release is so close and TCG being not
too important won't trigger an ISO respin.

> > b) are there known missing/bad instructions in P9-TCG that we should
> fix by backporting changes?
>
> Making some presumptions...you are looking for how to make QEMU work out
> of the box for a ppc64el TCG guest. I would hazard that there is some
> risk to backporting the instructions to QEMU 4.0, but I can't quantify
> that easily. I think it would be low, but I'm really not that familiar
> with the QEMU code base.
>
> The other consideration is that if you switch the default, you've
> perhaps solved the problem in the default case, but POWER9 support would
> still be broken if somebody tries to use it.

You are right P9 would still be broken, and as I said above for
switching the types it also is too late now.

> It would seem like the most comprehensive solution is to backport.
>
> BUT, I can't speak to whether there are other instructions missing
> between QEMU 4.0 and 4.2. :-/ Maybe Leonardo can better speak to a
> preferred path forward.

Yeah if we can make the backport work without too much
crazyness/effert and potential regression risk then I'd do that.
Looking forward Ubuntu 20.04 will have qemu 4.2 so there it will be good.

@Leonardo - let me know what you think about backporting the mff*
instructions to qemu 4.0.
If it seems easy enough let's do it, otherwise les mark this won't fix
for Eoan and close it with 4.2 for the next Ubuntu release.

In order to check that this issue is limited to qemu emulation.
I've tested ppc64le KVM guests on a ppc64le Ubuntu Eoan (19.10) host on a POWER9 bare metal - PowerNV 9006-12P.
- Guests: Ubuntu Bionic (18.04); Ubuntu Disco (19.04); Ubuntu Eoan (19.10).
Also tested these guests on a ppc64le Ubuntu Eoan host on a POWER8 bare metal (host in SMT off mode) - PowerNV 8335-GTA.
I was able to SSH into all guests.

While not having the priority for an upload on their own. If easy to backport we should even consider older versions of qemu.

------- Comment From <email address hidden> 2019-10-29 14:42 EDT-------
> While not having the priority for an upload on their own. If easy to
> backport we should even consider older versions of qemu.

I'm not sure who "we" refers to (comment #20), and there is an unanswered question directed at Leonardo (comment #18).

For clarity, what are the next steps here, and who owns them?

@ <email address hidden> please let me summarize - you're comment numbers (that probably refer to bugzilla numbers) seem to be differently numbered that the ones here in the LP ticket (since #20 here is a attached crash file).

So based on this LP we have some partly unanswered questions in 10, 17 and 19.

In comment #26 the question about a backport is directed to Leonardo, since we think that the expertise on the P9 instructions is at the IBM Power team.

Right now we are waiting for a decision to whether doing a backport or not and in case yes, for a backport itself.
Therefore the project entry of this LP is currently set to Incomplete (what is typically done in case we are waiting for a response).

This bug was fixed in the package qemu - 1:4.2-1ubuntu1

---------------
qemu (1:4.2-1ubuntu1) focal; urgency=medium

  * Merge with Debian testing, Among many other things this fixes LP Bugs:
    LP: #1847806 - add mff* instructions to not break on ppc64 with newer glibc
    LP: #1812822 - avoid crashes on detaching vhost_net interfaces
    LP: #1852744 - Crypto Passthrough Interrupt Support
    LP: #1853316 - CCW IPL Support
    Remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.maintscript: clean old sysv and upstart scripts
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type (LP: 1304107 1621042)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types
      - d/qemu-system-x86.NEWS Info on fixed machine type definitions
        for host-phys-bits=true (LP: 1776189)
      - add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - provide pseries-bionic-2.11-sxxm type as convenience with all
        meltdown/spectre workarounds enabled by default. (LP: 1761372).
    - Enable nesting by default
      - d/p/ubuntu/expose-vmx_qemu64cpu.patch: expose nested kvm by default
        in qemu64 cpu type.
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - improved dependencies
      - Make qemu-system-common depend on qemu-block-extra
      - Make qemu-utils depend on qemu-block-extra
      - let qemu-utils recommend sharutils
    - s390x support
      - Create qemu-system-s390x package
      - Enable numa support for s390x
      - d/rules: build s390-ccw.img with upstream Makefile
      - d/rules: build s390-netboot.img with upstream Makefile
    - arch aware kvm wrappers
    - d/control: update VCS links
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/control-in: Disable capstone disassembler library support (universe)
    - d/control: disable bluetooth being deprecated
    - d/not-installed: ignore new interop docs and extra icons for now
    - d/not-installed: do not install elf2dmp until namespaced
    - d/qemu-utils.install: install new tools qemu-edid and qemu-keymap
    - d/control-in: promote qemu-efi/ovmf in Ubuntu (LP 1570617)
    - d/binfmt-update-in: fix binfmt being called in some containers
      (LP 1840956)
  - Dropped changes (in Debian)
    - qemu-guest-agent: freeze-hook fixes (LP: 1484990)
      - d/qemu-guest-agent.install: provide /etc/qemu/fsfree...

Done in Focal, but I'm not planning to backport emulation related changes where most of the people don't care and the regression risk we trigger might still be huge.

Marking bionic debian-installer as "invalid", as the qemu changes will not be pulled back to bionic.

Marking eoan glibc as "invalid" as the qemu changes will not be pulled back to eoan.

From comment #25, is the required glibc support in either glibc 2.30 and/or glibc 2.31?

------- Comment From <email address hidden> 2020-02-10 10:43 EDT-------
> From comment #25, is the required glibc support in either glibc 2.30 and/or
> glibc 2.31?

Can you clarify, required support for what? For provoking the issue with qemu? Some of the code which made use of the new instructions arrived in glibc 2.30. More arrived in 2.31.

The required support of these instructions in QEMU arrived in QEMU 4.2.

Yes, that was the background - we wanted to know if the new instructions were in glibc 2.30 and/or glibc 2.31 or not.
First to adjust the status of that ticket accordingly,
second to push another ticket (if needed) that discusses the upgrade to glibc 2.31 in focal.

Qemu 4.2 recently arrived in focal.

So in other words - and thx to your clarification - we are good on that for focal!

Default installer for Focal will be subiquity, not d-i.

Waiting on focal's adoption of glibc 2.31.

This was a glibc vs qemu-emulation issue.
Glibc versions in >=Eoan failed with the Qemu versions <=Focal.

In coordination with IBM everyone agreed that emulation isn't a primary use case and that backports would therefore not be SRUed as the risk to break something for more common use cases outweights the gain. Especially now that users can just upgrade to Focal if they need a qemu able to emulate the insructions (and the Ubuntu Cloud Archive backports that to Bionic).

Closing the remaining tasks.