ld:i386 crashes with -static -fPIE -pie

malloc_printerr() on error detection "free(): invalid next size (fast)" ends up calling into:

backtrace.c:init()
dl-libc.c:do_dlopen()
malloc.c:calloc()
malloc.c:malloc_printerr()

The malloc error reporting should only report the first error, not attempt to recusively report all error (we knew it was corrupted at the outer most point, so any further work inside malloc is also likely to find corruption).

Full stack trace to follow.

The main problem is the process does not abort() and die, it hangs around in:

pthread_once.S:pthread_one()
backtrace.c:__backtrace()

I think due to recursive lock, this lock should trylock() on the 2nd time and abort() the process immediately. It does appear to deadlock itself.

(gdb) bt
#0 pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:95
#1 0x00007f8dfb540994 in __backtrace (array=<value optimized out>, size=64) at ../sysdeps/ia64/backtrace.c:85
#2 0x00007f8dfb4b280b in __libc_message (do_abort=2, fmt=0x7f8dfb599fc0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:178
#3 0x00007f8dfb4b8126 in malloc_printerr (action=3, str=0x7f8dfb5980f9 "malloc(): memory corruption", ptr=<value optimized out>) at malloc.c:6311
#4 0x00007f8dfb4bbba4 in _int_malloc (av=0x7f8dfb7d0e80, bytes=<value optimized out>) at malloc.c:4411
#5 0x00007f8dfb4bc5e6 in __libc_calloc (n=<value optimized out>, elem_size=<value optimized out>) at malloc.c:4075
#6 0x00007f8dfda14d1f in _dl_new_object (realname=0x247de20 "/lib64/libgcc_s.so.1", libname=0x7f8dfb596e3e "libgcc_s.so.1", type=2, loader=0x0, mode=-1879048191, nsid=0) at dl-object.c:77
#7 0x00007f8dfda111ae in _dl_map_object_from_fd (name=0x7f8dfb596e3e "libgcc_s.so.1", fd=6, fbp=0x7fffd2c1ace0, realname=0x247de20 "/lib64/libgcc_s.so.1", loader=0x0, l_type=2, mode=-1879048191, stack_endp=0x7fffd2c1b028, nsid=0)
    at dl-load.c:975
#8 0x00007f8dfda1236a in _dl_map_object (loader=0x0, name=0x7f8dfb596e3e "libgcc_s.so.1", type=2, trace_mode=0, mode=<value optimized out>, nsid=<value optimized out>) at dl-load.c:2274
#9 0x00007f8dfda1ca34 in dl_open_worker (a=0x7fffd2c1b250) at dl-open.c:227
#10 0x00007f8dfda181a6 in _dl_catch_error (objname=0x7fffd2c1b2a0, errstring=0x7fffd2c1b298, mallocedp=0x7fffd2c1b2af, operate=0x7f8dfda1c910 <dl_open_worker>, args=0x7fffd2c1b250) at dl-error.c:178
#11 0x00007f8dfda1c4ea in _dl_open (file=0x7f8dfb596e3e "libgcc_s.so.1", mode=-2147483647, caller_dlopen=0x0, nsid=-2, argc=8, argv=<value optimized out>, env=0x7fffd2c30020) at dl-open.c:569
#12 0x00007f8dfb568340 in do_dlopen (ptr=<value optimized out>) at dl-libc.c:86
#13 0x00007f8dfda181a6 in _dl_catch_error (objname=0x7fffd2c1b460, errstring=0x7fffd2c1b458, mallocedp=0x7fffd2c1b46f, operate=0x7f8dfb568300 <do_dlopen>, args=0x7fffd2c1b440) at dl-error.c:178
#14 0x00007f8dfb568497 in dlerror_run (name=<value optimized out>, mode=<value optimized out>) at dl-libc.c:47
#15 __libc_dlopen_mode (name=<value optimized out>, mode=<value optimized out>) at dl-libc.c:160
#16 0x00007f8dfb540895 in init () at ../sysdeps/ia64/backtrace.c:41
#17 0x00007f8dfb7e1b23 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104
#18 0x00007f8dfb540994 in __backtrace (array=<value optimized out>, size=64) at ../sysdeps/ia64/backtrace.c:85
#19 0x00007f8dfb4b280b in __libc_message (do_abort=2, fmt=0x7f8dfb599fc0 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:178
#20 0x00007f8dfb4b8126 in malloc_printerr (action=3, str=0x7f8dfb59a2b8 "free(): invalid next size (fast)", ptr=<value optimized out>) at malloc.c:6311
#21 0x00007f8dfb4bac53 in _int_free (av=0x7f8dfb7d0e80, p=0x24d52c0, have_lock=0) at malloc.c:4811

See also bug#956

*** Bug 956 has been marked as a duplicate of this bug. ***

This is going to be difficult to fix and invasive. I will do my best to explain why.

At the point of the failure we want to be able to print a backtrace. The only way to get a reliable backtrace is to use the unwinder provided by gcc via libgcc_s.so.1 (this may vary by machine). In order to get access to the unwinder we must dlopen that shared library. During the dlopen process we need to calloc enough structures to hookup the new shared library into the structures used by the dynamic linker.

One resolution to this problem is to ensure that malloc has a fall-back allocation scheme that is robust against failure and then during the malloc_printerr we flip an internal bit and switch to the temporary reserve allocations. We could also create a new internal API for using the temporary allocations and then dlopen could use that in the event that we are crashing and need to dlopen one last library (the unwinder on demand). That would prevent other threads from consuming the reserve allocations after malloc_printerr is entered by another thread.

This is a considerable amount of work and we aren't going to get to this issue until a core developer or someone with serious interest commits to fixing this. Therefore I'm moving this to SUSPENDED until we find the resources to fix the issue.

This issue should remain open and new issues submited about this bug should be marked as duplicates of this issue.

On Wed, Nov 13, 2013 at 03:57:02AM +0000, carlos at redhat dot com wrote:
> One resolution to this problem is to ensure that malloc has a fall-back
> allocation scheme that is robust against failure and then during the
> malloc_printerr we flip an internal bit and switch to the temporary reserve
> allocations. We could also create a new internal API for using the temporary
> allocations and then dlopen could use that in the event that we are crashing
> and need to dlopen one last library (the unwinder on demand). That would
> prevent other threads from consuming the reserve allocations after
> malloc_printerr is entered by another thread.
>
> This is a considerable amount of work and we aren't going to get to this issue
> until a core developer or someone with serious interest commits to fixing this.
> Therefore I'm moving this to SUSPENDED until we find the resources to fix the
> issue.
>
Why not reuse a singal-safe malloc for dlopen?

This fancy backtrace stuff is nice and all but... the process must die!

Can't the pthread_once use a non-blocking lock ?

Can the lock be a recursive type ?

Can pthread_trylock() used in this non-critial path ? if already locked, and

if possible to check if locked by our thread-id ?

then we immediately abort the process (causing execution of the process to die, like it should). No backtrace is emitted, great!

How do I stop this fancy backtrace stuff from working ? I want to setup an environment variable to turn it off as a workaround ?

How do I make this fancy backtrace stuff work, by preloading the dlopen() stuff it might need, during initialization of malloc() ? I want to setup an environment variable for that too.

There is no need to actually fix the bug, you are over thinking the issue. But this fancy stuff needs to be turned off or preloaded, before the process gets into an undefined state (due to memory bug).

Another idea, do not backtrace() every malloc() error, only the first one (the outer most one).

But right now the process deadlocks itself, on what looks to be a non-recursive mutex trying to do fancy backtrace on every malloc() problem found.

The process must die.

On Wed, Nov 13, 2013 at 01:00:06PM +0000, darryl.miles at darrylmiles dot org wrote:
> How do I stop this fancy backtrace stuff from working ? I want to setup an
> environment variable to turn it off as a workaround ?
>
> How do I make this fancy backtrace stuff work, by preloading the dlopen() stuff
> it might need, during initialization of malloc() ? I want to setup an
> environment variable for that too.
>

As a quick workaround you can add following code to your application/preload this.

#include <execinfo.h>
static void __attribute__ ((constructor))
init_backtrace()
{
   void *bt[10];
   backtrace (bt, 10);
}

Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the wrong thing to do, and libgcc_eh should be static-linked into libc. (The other big reason is the possibility of pthread_cancel aborting the program.) At one time in the distant past, it was necessary for there to only be one copy of this code (and its data) in the whole program; otherwise, exception propagation (or backtracing) across DSOs would not work reliably. But modern unwinding code uses dl_iterate_phdr and works fine even if multiple copies of the code are present in the program.

Fixing this error in the way I describe will greatly simplify glibc and improve its reliability.

(In reply to Rich Felker from comment #9)
> Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the
> wrong thing to do, and libgcc_eh should be static-linked into libc. (The
> other big reason is the possibility of pthread_cancel aborting the program.)
> At one time in the distant past, it was necessary for there to only be one
> copy of this code (and its data) in the whole program; otherwise, exception
> propagation (or backtracing) across DSOs would not work reliably. But modern
> unwinding code uses dl_iterate_phdr and works fine even if multiple copies
> of the code are present in the program.
>
> Fixing this error in the way I describe will greatly simplify glibc and
> improve its reliability.

That sounds like a good idea to me, I also agree that dlopening libgcc_s.so.1 always seemed like a terrible idea to me. We just need the resources to do the rewrite and fixup the linking to use libgcc_eh. I will leave this SUSPENDED until we find someone to clean this up.

On Wed, 13 Nov 2013, bugdal at aerifal dot cx wrote:

> Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the wrong
> thing to do, and libgcc_eh should be static-linked into libc. (The other big

Static-linking libgcc_eh into any glibc library is a bad idea because it
complicates bootstrapping: it means glibc built with an initial bootstrap
compiler (which was built without glibc headers available, implying full
EH functionality is not present in libgcc) is not identical to glibc built
with a compiler built using full shared glibc and headers. (It's *also* a
bad idea because new compilers can start using new DWARF unwind opcodes
that an old copy of the unwind code won't understand, causing problems
using new programs with old glibc.)

The answer for libpthread is for it to dlopen libgcc_s when loaded rather
than at pthread_cancel time (or to be made to depend (DT_NEEDED) on
libgcc_s in a way that doesn't require libgcc_s to be available when
libpthread is built). The answer for other cases is to disable the
backtracing by default as discussed in bug 12189 (possibly with an
environment variable, not available in setuid programs, that can reenable
it - in which case glibc would dlopen libgcc_s at startup).

On Wed, Nov 13, 2013 at 04:12:53PM +0000, joseph at codesourcery dot com wrote:
> http://sourceware.org/bugzilla/show_bug.cgi?id=16159
>
> --- Comment #11 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
> On Wed, 13 Nov 2013, bugdal at aerifal dot cx wrote:
>
> > Carlos, this is yet another reason why dlopen'ing libgcc_s is simply the wrong
> > thing to do, and libgcc_eh should be static-linked into libc. (The other big
>
> Static-linking libgcc_eh into any glibc library is a bad idea because it
> complicates bootstrapping: it means glibc built with an initial bootstrap
> compiler (which was built without glibc headers available, implying full
> EH functionality is not present in libgcc) is not identical to glibc built
> with a compiler built using full shared glibc and headers. (It's *also* a
> bad idea because new compilers can start using new DWARF unwind opcodes
> that an old copy of the unwind code won't understand, causing problems
> using new programs with old glibc.)
>
Why did you jump from dlopening to static linking? Dynamic linking would
work and if there is concern that user does not have one we could
provide a stub implementation and function to test if we deal with stub
or real one.

Joseph, the bootstrapping issue can presumably be fixed (and bootstrapping made easier) simply by providing a way to install headers without building glibc. This may even allow you to shave one or more steps off of the full bootstrap process.

As for the issue of new DWARF opcodes, if they prevent older unwind code from being able to interpret the unwind information at all (rather than just failing to take advantage of the new features) that seems like a fundamental design bug elsewhere that should be reported. I'm not clear whether or not that's really the case.

With that said, I find your alternate fix proposal acceptable. For the libpthread issue, I believe the DT_NEEDED could be generated at build time using a fake libgcc_s.so.1 in the glibc source tree. As for disabling backtrace by default, that's perfectly acceptable. Alternatively, glibc could always attempt to load libgcc_s.so.1 at startup and disable backtrace if it's not found.

Ondrej, is having glibc contain a DT_NEEDED entry for libgcc.so.1 really an option that's on the table? I think this would also interfere with bootstrapping issues Joseph and others may be concerned about, as well as hurting load-time performance.

On Wed, 13 Nov 2013, neleai at seznam dot cz wrote:

> Why did you jump from dlopening to static linking? Dynamic linking would
> work and if there is concern that user does not have one we could
> provide a stub implementation and function to test if we deal with stub
> or real one.

I don't think default dlopening libgcc_s from libc at startup is desirable
on performance grounds (most programs will never need it), whereas from
libpthread it's likely to be less significant.

On Wed, 13 Nov 2013, bugdal at aerifal dot cx wrote:

> Joseph, the bootstrapping issue can presumably be fixed (and bootstrapping made
> easier) simply by providing a way to install headers without building glibc.

There already is. But to install the correct set of headers (some
generated at build time) you first need an appropriately configured
compiler to configure glibc. That's the old three-compiler bootstrap
process: first build a basic compiler, then install headers with it and
crt*.o and build a dummy libc.so, then build a second compiler with shared
libgcc, then build glibc, then build a third compiler. I changed things
in glibc and GCC so that a two-compiler process suffices: the initial
compiler built without headers can build glibc and the result is identical
to what you get if you repeatedly alternate GCC and glibc builds.
(Ideally you'd have a one-compiler process, where the second compiler
build only builds/rebuilds GCC's runtime libraries where they depend on
system headers or shared glibc, not GCC itself.)

Joseph, do you have a benchmark to measure libgcc overhead?

I tried a following

cat "int main()
{
  return 42;
}" > x.c
gcc x.c -O3 -o nogcc
gcc x.c -O3 -lgcc -o withgcc
time for I in `seq 1 10000`; do ./nogcc; done
time for I in `seq 1 10000`; do ./withgcc; done

And I cannot distinguish these from noise. When I linked with -lpthread there was a noticable slowdown.

Ondrej, did you even check your results with readelf or ldd? -lgcc is a static library and is always linked, so of course it won't make any difference. You need to test with -lgcc_s (and double-check to make sure the dependency really got added).

BTW, I'm not sure how well your test will do measuring exec time versus other overhead. If you'd like, I have a test I can post that execs itself and measures the actual time from just before the execve syscall to the start of main.

On Thu, Nov 14, 2013 at 03:54:30PM +0000, bugdal at aerifal dot cx wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=16159
>
> --- Comment #18 from Rich Felker <bugdal at aerifal dot cx> ---
> Ondrej, did you even check your results with readelf or ldd? -lgcc is a static
> library and is always linked, so of course it won't make any difference. You
> need to test with -lgcc_s (and double-check to make sure the dependency really
> got added).
>
I asked for benchmark because of that, with a lgcc_s there is difference.

plain

real 0m3.039s
user 0m0.195s
sys 0m3.049s

with lgcc_s

real 0m3.141s
user 0m0.169s
sys 0m3.179s

with lpthread

real 0m3.282s
user 0m0.182s
sys 0m3.308s

> BTW, I'm not sure how well your test will do measuring exec time versus other
> overhead. If you'd like, I have a test I can post that execs itself and
> measures the actual time from just before the execve syscall to the start of
> main.
>
These also count as I wanted to show a relative performance impact. If
this is taken into extreme we could improve performance by staticaly linking lm and lpthread

Or using prelink.

On Thu, Nov 14, 2013 at 04:47:48PM +0000, neleai at seznam dot cz wrote:
> These also count as I wanted to show a relative performance impact. If

I agree this approach makes sense, but the relative performance impact
could change when the program (possibly linked with libgcc_s) is
invoked via posix_spawn or vfork+exec from a high-load server versus
as part of an inefficient shell script where the shell may have a lot
of additional syscall overhead on each command (this might also vary
between shells; dash or busybox ash might perform very differently
from bash). So while we may not care about the most extreme impact, I
think it's important to consider how large the relative overhead is
when the invocation conditions are a low-overhead, real-world
scenario.

> this is taken into extreme we could improve performance by staticaly linking lm
> and lpthread

Yes, of course -- actually, I would recommend merging all of the glibc
.so's into libc.so, but I understand that the current situation with
symbol versions greatly complicates this, and that there might be
other issues. It would certainly improve load-time performance and
memory overhead for small programs, though. But I think this is
outside the scope of this bug report. The interest in looking at
performance here is asking whether a proposed change would make
performance noticably worse (a regression), not how we can best
optimize startup performance.

(In reply to Darryl Miles from comment #6)
> How do I stop this fancy backtrace stuff from working ? I want to setup an
> environment variable to turn it off as a workaround ?

According to:
https://lists.gnu.org/archive/html/bug-gnulib/2013-11/msg00103.html
setting MALLOC_CHECK_=2 in the environment is sufficient to prevent the error message attempts; but that sounds like something you set at program start rather than something we can do via setenv() at the time of reporting the first error (because setenv uses malloc).

> There is no need to actually fix the bug, you are over thinking the issue.

Yes, there IS a need to fix something. The link above points to a case of a user that is unhappy that their ./configure failed because the conftest program hung after tickling a malloc corruption bug in regex. Configure should never hang (thankfully, configure tests are one case where the MALLOC_CHECK_=2 trick may be sufficient - someone probing for known glibc bugs doesn't care about a bactrace, only about successful exit status).