eztrace FTBFS with LTO

Bug #2016471 reported by Graham Inggs
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eztrace (Ubuntu)
Status tracked in Mantic
Mantic
Fix Released
Undecided
Unassigned
llvm-defaults (Ubuntu)
Status tracked in Mantic
Mantic
Invalid
Undecided
Unassigned
llvm-toolchain-15 (Ubuntu)
Status tracked in Mantic
Mantic
Fix Released
Undecided
Unassigned
lto-disabled-list (Ubuntu)
Status tracked in Mantic
Mantic
Fix Released
Undecided
Lukas Märdian

Bug Description

eztrace 2.0+repack-11 FTBFS due to mpich test failure when built with LTO enabled.

The following tests FAILED:
   3 - mpi_tests (Failed)
   5 - memory_tests (Failed)
  10 - openmp_tests (Failed)
  12 - pthread_tests (Failed)
  15 - posixio_tests (Failed)
Errors while running CTest
make[2]: *** [Makefile:74: test] Error 8
make[2]: Leaving directory '/<<PKGBUILDDIR>>/build-mpich'

The mpich tests pass after disaling LTO, however then the ompt tests still fail.

The following tests FAILED:
   5 - ompt_tests (Failed)
Errors while running CTest
make[2]: *** [Makefile:74: test] Error 8
make[2]: Leaving directory '/<<PKGBUILDDIR>>/build-ompt'

Lukas Märdian (slyon)
tags: added: rls-mm-incoming
Revision history for this message
Lukas Märdian (slyon) wrote :

As Graham stated, disabling LTO make the build/tests (mostly) pass, but fails for the "OMPT_ARCHS" (amd64 arm64 i386 ppc64el riscv64), as defined in d/rules.

This can be done via lto-disabled-list or d/rules:
export DEB_BUILD_MAINT_OPTIONS = hardening=+all optimize=-lto

I tried a few different things in eztrace to make the toolchain mimic the Debian build (which passes):
* Switch to clang-14:
  CC=clang-14 in d/rules and clang-14 & libomp-14-dev B-Ds in d/control
* Strip Ubuntu's -Bsymbolic-functions optimization from the linker flags
  export DEB_LDFLAGS_MAINT_STRIP = -Wl,-Bsymbolic-functions in d/rules

But didn't have any success with the above changes.

Then I noticed bug #1899199, which indicates the root-cause being in the libomp5 dependency instead of eztrace itself, i.e. llvm-toolchain-* (or libomp.so.5 specifically) needs to be build without the -Wl,-Bsymbolic-functions LDFLAGS optimization. This seems to be the case!

After downloading relevant binaries from Debian (https://packages.debian.org/source/sid/llvm-toolchain-15), I was able to build eztrace successfully (with just LTO disabled, but no other changes):

sbuild --extra-package ../libomp5-15_15.0.7-4_amd64.deb --extra-package ../libomp-15-dev_15.0.7-4_amd64.deb -dlunar .

So IMO this is something for the Foundations toolchain people to look into, fixing our llvm-toolchain (libomp) build.

Revision history for this message
François Trahay (trahay) wrote :

I'm the main developer of EZTrace. The OMPT problem does come from bug #1899199.

If the llvm-toolchain build is not fixed in Ubuntu, I can implement a workaround in EZTrace (as suggested in https://bugs.launchpad.net/ubuntu/+source/llvm-defaults/+bug/1899199/comments/11), but I would prefer to avoid such unnecessary hacks.

Lukas Märdian (slyon)
Changed in eztrace (Ubuntu):
assignee: nobody → Lukas Märdian (slyon)
status: New → In Progress
Changed in lto-disabled-list (Ubuntu):
assignee: nobody → Lukas Märdian (slyon)
status: New → In Progress
Revision history for this message
Lukas Märdian (slyon) wrote :

François,
If you could come up with a patch (fix/workaround) for EZTrace, that'd be much appreciated!
Applying the "-Wl,-Bsymbolic-functions" fix to llvm-toolchain will most probably lead to a bigger discussion, as this is an optimization Ubuntu enabled on purpose a long time ago, and disabling it in a core-package like the LLVM toolchain might be a high impact change.

See this email thread for the -Wl,-Bsymbolic-functions background:
https://lists.ubuntu.com/archives/ubuntu-devel/2018-June/040369.html

Lukas Märdian (slyon)
Changed in lto-disabled-list (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lto-disabled-list - 40

---------------
lto-disabled-list (40) devel; urgency=medium

  * Add clasp on s390x to avoid FTBFS (LP: #2017091)
  * Add eztrace to avoid FTBFS (LP: #2016471)

 -- Lukas Märdian <email address hidden> Thu, 20 Apr 2023 09:23:56 +0200

Changed in lto-disabled-list (Ubuntu):
status: Fix Committed → Fix Released
Steve Langasek (vorlon)
tags: removed: rls-mm-incoming
Steve Langasek (vorlon)
tags: added: fr-4344
tags: added: foundations-todo
Lukas Märdian (slyon)
Changed in eztrace (Ubuntu Mantic):
assignee: Lukas Märdian (slyon) → nobody
Steve Langasek (vorlon)
Changed in llvm-toolchain-15 (Ubuntu Mantic):
assignee: nobody → Steve Langasek (vorlon)
Revision history for this message
Steve Langasek (vorlon) wrote :

> Applying the "-Wl,-Bsymbolic-functions" fix to llvm-toolchain
> will most probably lead to a bigger discussion, as this is an
> optimization Ubuntu enabled on purpose a long time ago,
> and disabling it in a core-package like the LLVM toolchain
> might be a high impact change.

In fact, there are almost no reverse-dependencies of libomp in Ubuntu. I've uploaded a patch that disables -Bsymbolic-functions for libomp only, which should hopefully fix the eztrace build.

Changed in llvm-toolchain-15 (Ubuntu Mantic):
status: New → Fix Committed
Revision history for this message
Steve Langasek (vorlon) wrote :
Download full text (4.2 KiB)

Attempted to rebuild eztrace with the libomp built with -Wl,-Bsymbolic and it didn't help.

Correct version of libomp downloaded:

Get:108 http://ftpmaster.internal/ubuntu mantic-proposed/universe amd64 libomp5-15 amd64 1:15.0.7-4ubuntu2 [480 kB]

(https://launchpad.net/ubuntu/+source/eztrace/2.0+repack-12/+build/26015911)

expected linker options when building libomp:

[2100/2653] : && /<<PKGBUILDDIR>>/build-llvm/./bin/clang --target=x86_64-pc-linux-gnu -fPIC -fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -fdebug-prefix-map=/<<PKGBUILDDIR>>/build-llvm/runtimes/runtimes-bins=build-llvm/runtimes/runtimes-bins -fdebug-prefix-map=/<<PKGBUILDDIR>>/= -no-canonical-prefixes -ffile-prefix-map=/<<PKGBUILDDIR>>/build-llvm/runtimes/runtimes-bins=build-llvm/runtimes/runtimes-bins -ffile-prefix-map=/<<PKGBUILDDIR>>/= -no-canonical-prefixes -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-extra -Wno-pedantic -O3 -DNDEBUG -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--build-id -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=gold -Wl,--warn-shared-textrel -Wl,--as-needed -Wl,--version-script=/<<PKGBUILDDIR>>/openmp/runtime/src/exports_so.txt -static-libgcc -Wl,-z,noexecstack -Wl,-Bno-symbolic -shared -Wl,-soname,libomp.so.5 -o openmp/runtime/src/libomp.so.5 openmp/runtime/src/CMakeFiles/omp.dir/kmp_alloc.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_atomic.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_csupport.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_debug.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_itt.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_environment.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_error.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_global.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_i18n.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_io.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_runtime.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_settings.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_str.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_tasking.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_threadprivate.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_utility.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_barrier.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_wait_release.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_affinity.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_dispatch.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_lock.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_sched.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/z_Linux_util.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_gsupport.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/thirdparty/ittnotify/ittnotify_static.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_taskdeps.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_cancel.c...

Read more...

Steve Langasek (vorlon)
Changed in llvm-toolchain-15 (Ubuntu Mantic):
assignee: Steve Langasek (vorlon) → nobody
status: Fix Committed → Triaged
Revision history for this message
Steve Langasek (vorlon) wrote :

Looking more closely, there's no relocation for this symbol because the symbol isn't present at all.
$ objdump -T ./build-llvm/runtimes/runtimes-bins/openmp/runtime/src/libomp.so.5 | grep -i ompt
$

I'm not sure if this is a result of a deliberate change in the llvm implementation, but it certainly doesn't help to build with -Bno-symbolic if the symbol doesn't exist.

Revision history for this message
Graham Inggs (ginggs) wrote :

This appears to be fixed now in:

llvm-toolchain-15 (1:15.0.7-4ubuntu6) mantic; urgency=medium

  * Fix up -Wl,-Bsymbolic-functions disable
  * Cherry-pick upstream fix for cmake changes

Changed in llvm-toolchain-15 (Ubuntu Mantic):
status: Triaged → Fix Committed
Changed in llvm-defaults (Ubuntu Mantic):
status: New → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package llvm-toolchain-15 - 1:15.0.7-6

---------------
llvm-toolchain-15 (1:15.0.7-6) unstable; urgency=medium

  [ Sylvestre Ledru ]
  * Only pass -Bno-symbolic if it exists

  [ Gianfranco Costamagna ]
  * Revert LIBCXXABI_ARM_EHABI => _LIBCXXABI_ARM_EHABI change,
    FTBFS on arm*.

 -- Sylvestre Ledru <email address hidden> Mon, 26 Jun 2023 10:13:40 +0200

Changed in llvm-toolchain-15 (Ubuntu Mantic):
status: Fix Committed → Fix Released
Steve Langasek (vorlon)
Changed in eztrace (Ubuntu Mantic):
status: In Progress → Fix Released
Revision history for this message
François Trahay (trahay) wrote :

The problem was fixed in the llvm-toochain-15 package, but I also fixed it in the eztrace (upstream) repo.

EZTrace version 2.1 now uses `OMP_TOOL_LIBRARIES=$LD_PRELOAD` as a workaround, as suggested in
https://bugs.launchpad.net/ubuntu/+source/llvm-defaults/+bug/1899199/comments/11

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.