libseccomp 2.4.3-1ubuntu3.18.04.2 causes systemd to segfault on boot

Thanks for reporting this issue. I am not able to reproduce it myself - have you customised the syscall filtering in this profile at all?

To capture some more details that might help debug this issue, could you please run

apport-collect 1886115

in a terminal? This should automatically capture various details and upload them to this bug report.

> apport-collect 1886115
That doesn't seem to succeed (because the installed package is libseccomp2, but here listed as libseccomp since the former isn't known to launchpad).

I tried
$ apport-bug -u 1886115 libseccomp2
but that appears to attempt to create a new bug, rather then updating the existing one. See below for what
$ apport-bug --save /tmp/ab libseccomp2
generates.
--8<--
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.15
Architecture: amd64
CurrentDesktop: MATE
Date: Thu Jul 2 19:13:13 2020
Dependencies:
 gcc-8-base 8.4.0-1ubuntu1~18.04
 libc6 2.27-3ubuntu1
 libgcc1 1:8.4.0-1ubuntu1~18.04
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2019-03-30 (460 days ago)
InstallationMedia: Ubuntu 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
NonfreeKernelModules: wl
Package: libseccomp2 2.4.3-1ubuntu3.18.04.2
PackageArchitecture: amd64
ProcCpuinfoMinimal:
 processor : 3
 vendor_id : GenuineIntel
 cpu family : 6
 model : 42
 model name : Intel(R) Core(TM) i5-2415M CPU @ 2.30GHz
 stepping : 7
 microcode : 0x2f
 cpu MHz : 1396.555
 cache size : 3072 KB
 physical id : 0
 siblings : 4
 core id : 1
 cpu cores : 2
 apicid : 3
 initial apicid : 3
 fpu : yes
 fpu_exception : yes
 cpuid level : 13
 wp : yes
 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
 bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
 bogomips : 4589.43
 clflush size : 64
 cache_alignment : 64
 address sizes : 36 bits physical, 48 bits virtual
 power management:
ProcEnviron:
 LANGUAGE=en_US
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 5.3.0-62.56~18.04.1-generic 5.3.18
SourcePackage: libseccomp
Tags: bionic
Uname: Linux 5.3.0-62-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
_MarkForUpload: True
-->8--

If this is indeed related to the Gentoo bug, I cannot see anywhere in libseccomp where the environment is being modified. As such I suspect this is likely actually a bug in systemd where it is modifying the environment across the exec() and the libseccomp update has just caused it to actually manifest (due to changed memory layouts as a result of this new library version / size etc). So assigning this to systemd as well for now.

You can specify the package name using `-p` - so perhaps:

apport-collect -p systemd 1886115
apport-collect -p libseccomp2 1886115

Would do the trick?

> have you customised the syscall filtering in this profile at all?
Don't know what you mean by profile there. The OS runs on bare-metal here. The only related modification I made has been (now) the reported commenting-out of
--8<--
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io @reboot @swap
-->8--
in /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service which is (here) just a symbolic link to /lib/systemd/system/systemd-resolved.service.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I am confused - in the initial bug report you mention /etc/systemd/system/dbus-org.freedesktop.resolve1.service as the systemd unit but now you also mention /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service - can you confirm which one you have had to disable the SystemCallFilter?

Also there appear to be a bunch of systemd managed services which have failed to run - could you perhaps downgrade libseccomp back to the previous version and see if this resolves the issue? I think this would be a better solution *for now* than disabling the syscall filter (plus it will help confirm whether this is definitely a causal factor).

sudo apt install libseccomp=2.4.1-0ubuntu0.18.04.2

The other option to try and help debug this would be to try and capture a more complete stack trace of the crash - perhaps something like the following might be enough:

1. Ensure the likely offending version of libseccomp is installed (2.4.3-1ubuntu3.18.04.2) (if you downgraded it above)
2. Uncomment out the SystemCallFilter line from the systemd unit
3. Restart the systemd unit (do not reboot) (assuming it is the dbus one you first mentioned):
  sudo systemctl restart dbus-org.freedesktop.resolve1.service

Then with any luck if systemd does crash it at least not be during boot - and apport should catch the crash and create a crash dump. Then you should be able to report this (either as a separate bug or you could attach it to this bug via `apport-bug /var/crash/<crash-dump.crash>`

> can you confirm which one you have had to disable the SystemCallFilter?
  Both files are just symlinks to /lib/systemd/system/systemd-resolved.service.

> Also there appear to be a bunch of systemd managed services which have failed to run
  I moved snap related services out of the way recently as it isn't used here. systemd complains, but I didn't observe any other ill effects yet.

> could you perhaps downgrade libseccomp back to the previous version and see if this resolves the issue?
--8<--
$ sudo apt install libseccomp=2.4.1-0ubuntu0.18.04.2
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libseccomp
tho@deimos:~$ sudo apt install libseccomp2=2.4.1-0ubuntu0.18.04.2
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '2.4.1-0ubuntu0.18.04.2' for 'libseccomp2' was not found
-->8--

  I attempted to downgrade earlier, but couldn't readily find that original package. Let me have another look (I might have still the original install medium) ...

> try and capture a more complete stack trace of the crash
  Hmmh? The call stack is fairly shallow here (not untypical for non-recursive applications programmed in C), what do you expect is missing? I saved the original core file, but can produce a new one as you described above later.

2.3.1-2.1ubuntu4 is the version of libseccomp2 which is available, i.e.
$ sudo apt install libseccomp2=2.3.1-2.1ubuntu4
succeeds. That allowed me to uncomment, i.e. enable, the SystemCallFilter in /lib/systemd/system/systemd-resolved.service again and reboot successfully.

Ok, since I can't reproduce this locally, if you are interested / able to help with debugging it, could you please attach the core dump. Or if this contains potentially sensitive details, you could install the dbg versions of the packages and reproduce the crash and this would provide a more complete stack trace.

Hmmh, if I see this right, there's a systemd-dbg package available for Ubuntu 16.04, but not for 18.04.

FWIW, I attached the original core file as bug1886115.core .

Thanks - in Ubuntu releases 18.04 onwards debug symbols are provided via the separate -dbgsyms packages which require extra configuration - https://wiki.ubuntu.com/DebuggingProgramCrash

TL;DR:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
echo -e "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse\ndeb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt install ubuntu-dbgsym-keyring
sudo apt update
sudo apt install systemd-dbgsym
gdb /sbin/init bug1886115.core
(gdb) bt

Which gives the following more complete stack trace (sadly a bunch of stuff has been optimised out, but also since this is crashing in freep() it seems that memory corruption has already occurred prior to this point and we don't have any info as to where this happened):

#0 0x00007f16807ba187 in kill () at ../sysdeps/unix/syscall-template.S:78
#1 0x0000563b957223b7 in crash (sig=11) at ../src/core/main.c:196
#2 <signal handler called>
#3 __GI___libc_free (mem=0x4a60d140dfd9a5) at malloc.c:3103
#4 0x0000563b9577c22e in freep () at ../src/basic/alloc-util.h:61
#5 unit_fail_if_noncanonical (u=<optimised out>, where=<optimised out>) at ../src/core/unit.c:4774
#6 0x0000563b957672d6 in mount_enter_mounting (m=0x563b9782dba0) at ../src/core/mount.c:946
#7 mount_start.lto_priv.216 (u=0x563b9782dba0) at ../src/core/mount.c:1092
#8 0x0000563b9576ba22 in unit_start (u=0x563b9782dba0) at ../src/core/unit.c:1861
#9 job_perform_on_unit.lto_priv.424 (j=0x7ffe78153248) at ../src/core/job.c:553
#10 0x0000563b9574f51a in job_run_and_invalidate (j=<optimised out>) at ../src/core/job.c:618
#11 manager_dispatch_run_queue.lto_priv.429 (source=<optimised out>, userdata=<optimised out>, userdata=<optimised out>) at ../src/core/manager.c:1830
#12 0x00007f16803a509a in source_dispatch (s=s@entry=0x563b9780e8c0) at ../src/libsystemd/sd-event/sd-event.c:2341
#13 0x00007f16803a53ea in sd_event_dispatch (e=<optimised out>, e@entry=0x563b9780e620) at ../src/libsystemd/sd-event/sd-event.c:2663
#14 0x00007f16803a5579 in sd_event_run (e=<optimised out>, timeout=18446744073709551615) at ../src/libsystemd/sd-event/sd-event.c:2723
#15 0x0000563b9572a49d in manager_loop (m=0x563b97810d90) at ../src/core/manager.c:2541
#16 invoke_main_loop (m=0x563b97810d90, ret_reexecute=0x7ffe7815347a, ret_retval=<optimised out>, ret_shutdown_verb=<optimised out>, ret_fds=0x7ffe78153480, ret_switch_root_dir=0x7ffe781534a8, ret_switch_root_init=0x7ffe781534a0, ret_error_message=0x7ffe78153490)
    at ../src/core/main.c:1778
#17 0x0000563b9571560c in main (argc=<optimised out>, argv=<optimised out>) at ../src/core/main.c:2561

What is really needed is to try and catch the memory corruption as it happens - I am not sure if it is possible to run systemd via valgrind but this could be one option.

Note that 2.4.1-0ubuntu0.18.04.2 was previously in bionic and had been since May of 2019 (2.3.1-2.1ubuntu4 is what bionic was released with, but later updated to 2.4.1-0ubuntu0.18.04.2). 2.4.1-0ubuntu0.18.04.2 can be found here: https://launchpad.net/ubuntu/+source/libseccomp/2.4.1-0ubuntu0.18.04.2

This seems related:
* https://bugzilla.redhat.com/show_bug.cgi?id=1653068
* https://github.com/systemd/systemd/pull/11157

I can't say why the libseccomp update would change anything, though the redhat bug shows an AVC denial, so I wonder if you see anything related to systemd-resolved with 'journalctl | grep audit | grep systemd' at the time of the boot failure. If you see a seccomp denial, that might indicate a change in libseccomp to further investigate.

Regardless, systemd should not be crashing and https://github.com/systemd/systemd/pull/11157 should be backported IME.

> I wonder if you see anything related to systemd-resolved with 'journalctl | grep audit | grep systemd'
No, not since March 30th.

> at the time of the boot failure.
No log files were written at that time (no log entries made it to disk before the segfault).

Ok, normally I'd say it's very risky to include a bugfix for a bug that does not have a clear testcase, especially for a component such as systemd (high risk). That being said, looking at the actual change - fixing an uninitialized pointer - I feel much more confident.

Hello guenthert, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.42 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

with the proposed version of system, i've rebooted 100 times with no issue, so no obvious regression has been introduced.

  I didn't quite have Dan's patience, but FWIW my system successfully rebooted 25/25 times with the systemd package (237-3ubuntu10.42) from bionic-proposed in combination with libseccomp2:amd64 (2.4.3-1ubuntu3.18.04.3) installed and syscall filtering enabled in /lib/systemd/system/systemd-resolved.service.

All autopkgtests for the newly accepted systemd (237-3ubuntu10.42) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.0/5.0.0-58.62~18.04.1 (armhf)
linux-aws-edge/5.0.0-1019.21~18.04.1 (amd64)
linux-hwe-5.4/5.4.0-42.46~18.04.1 (arm64, armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (arm64)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
casync/2+61.20180112-1 (amd64)
linux-raspi2-5.3/5.3.0-1030.32~18.04.2 (armhf)
umockdev/0.11.1-1 (armhf)
netplan.io/0.99-0ubuntu3~18.04.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

This bug was fixed in the package systemd - 237-3ubuntu10.42

---------------
systemd (237-3ubuntu10.42) bionic; urgency=medium

  [ Dan Streetman ]
  * d/p/lp1860926/0001-networkd-Allow-to-retain-configs-even-if-carrier-is-.patch,
    d/p/lp1860926/0002-network-Change-IgnoreCarrierLoss-default-to-value-of.patch,
    d/p/lp1860926/0003-network-always-drop-configs-when-corresponding-netwo.patch:
    - Add IgnoreCarrierLoss and default to value of ConfigureWithoutCarrier
      (LP: #1860926)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9a12a31a62f1a50cd3a67a164ee34c546809815e
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3cc3870fde47982a4dda53f820e18065e5488e7e
  * d/e/rules-ubuntu/40-vm-hotadd.rules:
    - Hotadd only offline memory and CPUs
      (LP: #1876018)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ba305d7ad00e80bc1a03f93e6986eef7cbbb18fc
  * d/p/lp1881972-network-strdup-iif-and-oif-when-creating-RoutingPoli.patch:
    - Avoid double-free by strdup'ing iif/oif strings for new policy rules
      (LP: #1881972)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=874056f0d429aaa2cc872c3b35ec33cd3b740483
  * d/p/lp1886197-seccomp-more-comprehensive-protection-against-libsec.patch
    - Fix FTBFS on arm64 due to libseccomp changes (LP: #1886197)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c284a72ca2e3d87bfe1c20afb2fcfb379cda544f
  * d/p/lp1832754/0001-umount-Try-unmounting-even-if-remounting-read-only-f.patch,
    d/p/lp1832754/0002-umount-Don-t-bother-remounting-api-and-ro-filesystem.patch:
    - Try unmounting even if ro-remount fails, and don't bother remounting api/ro fs
      (LP: #1832754)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a518baa673aeaaf42000a3a01b7e03347652b216

  [ Alex Murray, Jamie Strandboge ]
  * d/p/lp1886115-pid1-fix-free-of-uninitialized-pointer-in-unit_fail_.patch:
    - Fix free of uninitialized pointer (LP: #1886115)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=491c76fd0f2fba0007a9b54d63a50f21add643c8

 -- Dan Streetman <email address hidden> Wed, 08 Jul 2020 14:59:14 -0400

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.