autopkgtest fails in hirsute on arm64

seen with the gcc-10 branch 20201003, while originally building that for another architecture. The build succeeds on another 32bit arch (arm-linux-gnueabihf) and on 64bit archs, but fails on i686-linux-gnu.

The build is also configured with --enable-default-pie and other hardening options turned on by default, built with make -j2.

virtual memory exhausted: Operation not permitted
make[5]: *** [Makefile:1117: insn-extract.o] Error 1
make[5]: *** Waiting for unfinished jobs....
rm gcc.pod gfortran.pod cpp.pod gccbrig.pod gccgo.pod gdc.pod
make[5]: Leaving directory '/<<PKGBUILDDIR>>/build/gcc'
make[4]: *** [Makefile:4878: all-stage3-gcc] Error 2
make[4]: Leaving directory '/<<PKGBUILDDIR>>/build'
make[3]: *** [Makefile:27347: stage3-bubble] Error 2
make[3]: Leaving directory '/<<PKGBUILDDIR>>/build'
make[2]: *** [Makefile:27411: bootstrap] Error 2
make[2]: Leaving directory '/<<PKGBUILDDIR>>/build'

virtual memory exhausted: Operation not permitted

ISTR Jakub mentioned a similar issue?

Yes, but for me it was resolved with r11-3518-g37ffe56c01e4a9e80a3b3c4f5beb86d80a0663db. If the (huge) switch is optimized by switchconv, then DF doesn't run into the latent problematic case.
I'm not using --enable-default-pie though.

I see the same thing for arm-linux-gnueabihf on trunk, also on
insn-extract.o. It's specific to RTL checking for me too.

I found it does work if I bootstrap with --enable-checking=yes,extra,
flip ENABLE_RTL_CHECKING to 1 in auto-host.h, and then recompile
insn-extract.c. But it does take a large amount of VM:

phase parsing : 51.10 ( 27%) 43.64 ( 74%) 94.74 ( 38%) 1334M ( 73%)

This used to work “a while ago” but I don't know when it stopped.

The file has 9975 lines, 435KiB, but that's a bairn compared to some
of the stuff we kick out. Perhaps it's just the sheer number of
(nested) XEXP macro expansions?

8441545d4f2afb9e9342e0dac378eafd03f00462 now builds insn-extract.o without rtl checking unconditionally.