gcc-linaro uses an unreasonable amount of memory to compile qemu on armel

Nonreduced preprocessed source attached. Build with this command (natively on arm) to reproduce:

/usr/lib/gcc/arm-linux-gnueabi/4.5.2/cc1 -quiet gcc-oom.i -quiet -dumpbase translate.c -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -auxbase-strip translate.o -g -O2 -Werror -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wendif-labels -Wwrite-strings -Wmissing-prototypes -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fno-strict-aliasing -fstack-protector -o translate.s

$ gcc --version
gcc (Ubuntu/Linaro 4.5.2-2ubuntu1) 4.5.2
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$

The OOM comes while cc1 is writing the output file out to disk; strace shows a large number of writes interspersed with calls to mmap2() requesting 1MiB allocations for every ~680-748KiB output to the file. By the time of the OOM (with ulimit now raised to 6GiB), the output file has grown to 174MiB here; but that only accounts for something like 260MiB, nothing close to the ulimit. So I'm still not sure what's going on here.

Oh, and of course arm is 32-bit, so raising the ulimit above 4GiB will have no effect whatsoever. Heh. So perhaps this is why it works ok in the cross-compiler, since I'm cross-compiling from amd64.

Still, the cross-build required only 3250MiB, and snapshots of cc1 usage natively never show it exceeding 1.7GiB. Is there some in-kernel limit on the maximum per-process memory that we're hitting?

Is this behavior - allocating another 1MiB for every 700KiB written to disk - expected from gcc, or is this a buggy use of a buffer?

The specific translate.c that is at issue here is target-sparc, used to build eg sparc64-linux-user/translate.o

On Sun, Feb 20, 2011 at 05:26:57PM -0000, Peter Maydell wrote:
> The specific translate.c that is at issue here is target-sparc, used to
> build eg sparc64-linux-user/translate.o

Actually, I've seen it with multiple different translate.c; the one I
submitted preprocessed source for should be the x86 translate.c.

There's been a report on the qemu mailing list of this problem on a ppc32 host as well; again, -fno-var-tracking allowed the compile to succeed.

Richard will attempt to reproduce on trunk, backport the patch if any, else report upstream.

Marking as triaged as we have sufficient information to reproduce.

Two patches posted upstream:

    http://gcc.gnu.org/ml/gcc-patches/2011-03/msg00193.html
    http://gcc.gnu.org/ml/gcc-patches/2011-03/msg00255.html

The first has also been committed upstream. The second is pending review, so I sent a ping today. I'll submit a merge request once both changes have gone in.

The first patch only applies to 4.6. The second (and main) one applies to both 4.5 and 4.6.

Richard, should we backport this to gcc-linaro 4.6 as well?

I am out of the office until 17/04/2011.

Note: This is an automated response to your message "[Bug 714921] Re:
gcc-linaro uses an unreasonable amount of memory to compile qemu on
armel" sent on 15/4/2011 4:50:47.

This is the only notification you will receive while this person is away.

Michael Hope <email address hidden> writes:
> Richard, should we backport this to gcc-linaro 4.6 as well?

I'm still hoping to get this in upstream 4.6. But if that doesn't
work out, then yeah, we should make it linaro local.

Richard

the patch is applied in the natty gcc-4.5 package