Mantic s390x install fails: grep-status: *** stack smashing detected ***: terminated

The described behavior happens when installing a QEMU VM, the host system is Jammy running on an LPAR.

@fheimes it would be interesting to know if the same happens on a "bare" lpar (no qemu).

Some binaries under /snap/subiquity/.../usr/bin on s390x use the libc from within the snap (they work properly) whereas others use the libc from outside the snap (/lib/s390x-linux-gnu) and they fail.

root@ubuntu-server:/# ldd /snap/subiquity/current/usr/bin/zstd
 linux-vdso64.so.1 (0x000003ffe1ead000)
 libz.so.1 => /snap/core22/current/lib/s390x-linux-gnu/libz.so.1 (0x000003ffb6580000)
 liblzma.so.5 => /snap/core22/current/lib/s390x-linux-gnu/liblzma.so.5 (0x000003ffb6500000)
 liblz4.so.1 => /snap/core22/current/lib/s390x-linux-gnu/liblz4.so.1 (0x000003ffb6480000)
 libc.so.6 => /snap/core22/current/lib/s390x-linux-gnu/libc.so.6 (0x000003ffb6280000)
 /snap/core22/current/lib/ld64.so.1 => /lib/ld64.so.1 (0x000003ffb6700000)
root@ubuntu-server:/# /snap/subiquity/current/usr/bin/zstd
stdin is a console, aborting
root@ubuntu-server:/# ldd /snap/subiquity/current/usr/bin/distro-info
 linux-vdso64.so.1 (0x000003ffdd2c8000)
 libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ffa3b80000)
 /snap/core22/current/lib/ld64.so.1 => /lib/ld64.so.1 (0x000003ffa3f00000)
root@ubuntu-server:/# /snap/subiquity/current/usr/bin/distro-info
*** stack smashing detected ***: terminated
Aborted (core dumped)

This looks somewhat related to snapcraft patchelf. See binary patching at https://snapcraft.io/docs/linters-classic

Excerpt from build log from Sept 11 [1]:

Patch ELF file: 'usr/bin/grep-dctrl'
  Interpreter='/snap/core22/current/lib/ld64.so.1'
  Current rpath=[]
  Proposed rpath=['/snap/core22/current/lib/s390x-linux-gnu']
Patch ELF file: 'usr/bin/join-dctrl'

Excerpt from build log from Sept 15 [2]:

Patch ELF file: 'usr/bin/grep-dctrl'
  Interpreter='/snap/core22/current/lib/ld64.so.1'
Patch ELF file: 'usr/bin/join-dctrl'

[1] https://launchpadlibrarian.net/686744619/buildlog_snap_ubuntu_jammy_s390x_subiquity_BUILDING.txt.gz
[2] https://launchpadlibrarian.net/687486601/buildlog_snap_ubuntu_jammy_s390x_subiquity_BUILDING.txt.gz

May I see the snapcraft.yaml for this? Is grep-status properly patched?

It's here https://github.com/canonical/subiquity/blob/main/snapcraft.yaml

and it's clear the files have been patchelfed as they have the interpreter set to /snap/core22/... but for some reason the ones from the latest build don't have rpath set:

ubuntu@juju-fa3835-porterboxen-37:~$ readelf -l ./subiquity-5095/usr/bin/grep-status | grep interpreter
      [Requesting program interpreter: /snap/core22/current/lib/ld64.so.1]
ubuntu@juju-fa3835-porterboxen-37:~$ readelf -l ./subiquity-5108/usr/bin/grep-status | grep interpreter
      [Requesting program interpreter: /snap/core22/current/lib/ld64.so.1]
ubuntu@juju-fa3835-porterboxen-37:~$ readelf -d ./subiquity-5095/usr/bin/grep-status | grep RPATH
 0x000000000000000f (RPATH) Library rpath: [/snap/core22/current/lib/s390x-linux-gnu]
ubuntu@juju-fa3835-porterboxen-37:~$ readelf -d ./subiquity-5108/usr/bin/grep-status | grep RPATH
ubuntu@juju-fa3835-porterboxen-37:~$

I assume the cause for this is going to turn out to be that there was a glibc SRU released between the two snap builds:

Get:4 http://ftpmaster.internal/ubuntu jammy-updates/main s390x libc6-dev s390x 2.35-0ubuntu3.1 [1499 kB]
Get:4 http://ftpmaster.internal/ubuntu jammy-updates/main s390x libc6-dev s390x 2.35-0ubuntu3.3 [1501 kB]

I noticed that snapcraft tries a bunch of different ways to determine the dependencies of an elf object https://github.com/snapcore/snapcraft/blob/main/snapcraft/elf/_elf_file.py#L416-L439 and this bug in particular being fixed https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1995362 might influence which one succeeds.

Or possibly its https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/2007599 making something fail that used to succeed as that patch probably breaks using the new libc6.so with the old ld.so or something along those lines.

Hmm, but this (LP#2007599) is also included in Jammy and Lunar, and as of today I am only aware that this issue happens with a mantic install.

@fheimes The bug would likely occur as well on recent s390x builds for other series. Currently though, only the devel series (i.e., mantic) uses grep-status. I wouldn't be surprised if you ran into the same issue after doing a snap refresh latest/edge on a 22.04 ISO for instance.

Ok, that could be of course - if the issue came in _after_ 22.04.3 got released, since the .3 was significantly tested and didn't showed that issue - but I see.

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2036424

An acceptable workaround seems to be to build the snap with edge core22, which is what we'll do for beta.

Rebuilt snap on beta and stable/ubuntu-23.10 channels. It does appear to have patchelf'ed correctly, per the edge core22 suggestion.

> Hmm, but this (LP#2007599) is also included in Jammy and Lunar, and as of today I am only aware that this issue happens with a mantic install.

Yes but the issue with a missing path is that libraries are using ld.so from the core snap and libc6 from the host system. If the core snap and host system are using the same version of glibc this isn't going to cause any problems!

Just wanted to let you know that I did a few installation (z/VM and LPAR) based on the respin from the 20th (20092023) and it looks good - didn't bumped into this anymore.

@Frank - I think this problem is resolved with stable core22. The build logs look as expected. Would you mind testing the build found at subiquity channel beta/s390x-stable-core22?

@dbungert, sorry for the delay,
I just tried beta/s390x-stable-core22 (based on the image 2023-10-05, but of course updated the subiquity):
  subiquity 22.02.2+git1772.4a76b65e 5196
  snap refresh subiquity --channel=beta/s390x-stable-core22 --ignore-running
  subiquity 22.02.2+git1793.3841511f 5227 latest/beta/…
but I unfortunately still face the grep-status issue on an LPAR install:

 2023-10-08 20:11:02,365 DEBUG subiquitycore.utils:121 arun_command called: ['gr
ep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'in
stalled', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/stat
us']
 2023-10-08 20:11:02,481 DEBUG subiquitycore.utils:135 arun_command ['grep-statu
s', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed'
, '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status'] exi
ted with code -6
 2023-10-08 20:11:02,481 ERROR root:30 finish: subiquity/Install/install/curtin_
install: FAIL: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-imag
e', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names'
, '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.
 2023-10-08 20:11:02,481 DEBUG subiquity.common.errorreport:394 generating crash report
 2023-10-08 20:11:02,484 INFO subiquity.common.errorreport:415 saving crash report 'install failed crashed with CalledProcessError' to /var/crash/1696795862.481621742.install_fail.crash
 2023-10-08 20:11:02,484 ERROR root:30 finish: subiquity/Install/install: FAIL: Command '['grep-status', '--whole-pkg', '-FProvides', 'linux-image', '--and', '-FStatus', 'installed', '--show-field=Package', '--no-field-names', '/target/var/lib/dpkg/status']' died with <Signals.SIGABRT: 6>.
 2023-10-08 20:11:02,484 INFO root:30 start: subiquity/ErrorReporter/1696795862.481621742.install_fail/add_info:
...

These are all the snap versions that I see during the installation:
Name Version Rev Tracking Publisher Notes
core22 20230801 865 latest/stable canonical** base
lxd 5.18-db8c6f9 25850 latest/stable/… canonical** -
snapd 2.60.3 20099 latest/stable canonical** snapd
subiquity 22.02.2+git1793.3841511f 5227 latest/beta/… canonical** classic

I've also attached the logs and crash files.

Thanks Frank.

I'm marking the "Affects Mantic" as fix released, as we have a plan. The builds today continue to use edge channel core22, like before. Keeping the issue open in general to track a fix to move back to stable channel core22.