Random segfaults when using ld.so explicitly to start a program

Confirmed on a 32-bit laptop. The script works just fine on a 64-bit machine.

Replacing /bin/bash with /bin/dash does not show anymore errors. Sounds like an upstream bash bug, not a kernel bug

If it is not a kernel bug, howcome it only appears with a Ubuntu patched kernel? At the minimum, it should be bisected which kernel patch broke bash.

I have Karmic installed and with kernel 2.6.31-14 this is totally reproducible. But if I boot to 2.6.28-15, the problem disappears.

Riku you're right. I guess we need to bisect using some previously built karmic kernels from https://edge.launchpad.net/ubuntu/karmic/+source/linux

We only need to concentrate on the the i386 builds. And then download and install the corresponding linux-image-*-generic_*.deb.

Help with verifying which build broke bash is appreciated.

Re-adding linux (kernel) as a possible bug originator.

I have a laptop with Fedora 12. On that machine, I have a debian-lenny chroot.

Within the chroot, I can reproduce the same behavior.

Outside of it, I can't (because which is a binary here).

# uname -a
Linux boober.fraggle 2.6.31.5-115.fc12.i686.PAE #1 SMP Wed Nov 4 00:27:30 EST 2009 i686 i686 i386 GNU/Linux

So it might be worth to check upstream kernels first.

On Thu, Nov 12, 2009 at 7:00 PM, Sven Herzberg <email address hidden> wrote:
> I have a laptop with Fedora 12. On that machine, I have a debian-lenny
> chroot.
>
> Within the chroot, I can reproduce the same behavior.
>
> Outside of it, I can't (because which is a binary here).
>
> # uname -a
> Linux boober.fraggle 2.6.31.5-115.fc12.i686.PAE #1 SMP Wed Nov 4 00:27:30 EST 2009 i686 i686 i386 GNU/Linux
>
> So it might be worth to check upstream kernels first.

Sven, that is very interesting information. Somehow the original bug
report was implying that this was only seen in Ubuntu kernels. Clearly
that is not the case. I'll try to make some time to bisect Ubuntu
kernels to see where we started seeing it.

I made some tests with various kernel versions using the test script attached to the bug description.

broken:
 * 2.6.31-14.48
 * 2.6.30-10.12

works:
 * 2.6.30-1

works:
 * 2.6.30-2.3

broken the next one:
* 2.6.30-3.4

So it seems with this release came the change that broke it.
Changes:
http://launchpadlibrarian.net/26403381/linux_2.6.30-3.4_i386.changes

I took a look at the diff from -2.3 to -3.4. I guess this is related to changelog entry:

[ Kees Cook ]
* SAUCE: [x86] implement cs-limit nx-emulation for ia32
    - LP: #369978

Kees, do you know if the nx-emulation could cause this?

This could be possible, though I do find it rather unexpected since other shared-library loading works fine (i.e. a PIE executable is effectively a shared library, like ld.so). If the problem does not appear on 64-bit, then it likely will not appear on the 32-bit generic-pae kernel (assuming this is on a system with PAE and the NX bit). Some further background is here:
https://wiki.ubuntu.com/Security/Features#Non-Exec%20Memory

I will investigate this further, but it does sound like a problem with the partial nx emulation patch.

I should clarify a bit further: this is likely a bad interaction between the randomized mmap allocations and the NX emulation. If you disable ASLR, the problem may also vanish (though note that this disables ASLR for the entire system, which is not really recommended):
https://wiki.ubuntu.com/Security/Features#ASLR

echo 0 > /proc/sys/kernel/randomize_va_space

hrm, it seems randomize_va_space=0 is not disabling ASLR for ld.so and vdso, so this did not change the problem for me. (I am able to reproduce this test-case on an nx-emulated system.

Also note that Fedora 12's kernel carries the same (or nearly identical) nx-emulation patch, so the behavior would be exposed there too (which is what comment 7 seems to show).

So, I suspect the exec-range ASLR that is specific to the patch is buggy. I will continue poking at this.

@kees If the problem does not appear on 64-bit, then it likely will not appear on the 32-bit generic-pae kernel (assuming this is on a system with PAE and the NX bit).

Correct, this is reproducible on 32bit kernel but not on a 64bit kernel.

uname :
Linux 2.6.31-14-generic-pae #48-Ubuntu SMP Fri Oct 16 15:22:42 UTC 2009 i686 GNU/Linux

hardware :
HP xw4100 / P4 / 2Gb / 36Gb SCSI320

dmesg output :
[37809.238716] apt-get[3530]: segfault at 44a5affd ip 00b8f36d sp bfb468b0 error 4 in libapt-pkg-libc6.10-6.so.4.8.1[af6000+bd000]
[37837.966813] apt-get[3535]: segfault at 7f34000 ip 00533a8c sp bf880780 error 4 in libapt-pkg-libc6.10-6.so.4.8.1[4fe000+bd000]
[97403.929131] dbus-daemon[6559]: segfault at 94 ip 00991a1f sp bfbda620 error 4 in libc-2.10.1.so[924000+13e000]
[97563.168196] apt-check[6696]: segfault at e3a59e6c ip 0015c8a5 sp bfed3430 error 5 in libapt-pkg-libc6.10-6.so.4.8.1[127000+bd000]
[106136.781026] landscape-sysin[7040]: segfault at 4000 ip 0815eb5d sp bf929580 error 4 in python2.6[8048000+1d3000]
[106347.068574] apt-get[7262]: segfault at e4646e6c ip 005de8a5 sp bfbc6660 error 5 in libapt-pkg-libc6.10-6.so.4.8.1[5a9000+bd000]
[106647.362444] dpkg[7399]: segfault at 8014 ip 08053388 sp bffb29e8 error 6 in dpkg[8048000+64000]
[107687.746824] python[7759]: segfault at 8004 ip 0810602a sp bfb6d9e0 error 4 in python2.6[8048000+1d3000]
[111018.052060] apt-get[8036]: segfault at e6ced630 ip 00b3b9b8 sp bfa8201c error 5 in libc-2.10.1.so[ac9000+13e000]
[111054.822837] apt-get[8061]: segfault at c8924014 ip 00c0be70 sp bf8f8770 error 5 in libapt-pkg-libc6.10-6.so.4.8.1[bc1000+bd000]
[111351.124310] dpkg[8152]: segfault at 796c6564 ip 00182f43 sp bf90515c error 4 in libc-2.10.1.so[110000+13e000]
[111376.933690] apt-get[8165]: segfault at b9b13908 ip 00bb5ad3 sp bfe9e690 error 4 in libapt-pkg-libc6.10-6.so.4.8.1[b6c000+bd000]

Frsh install, running only samba 3.4.0.
How to correct this issue please ? I have to install several other tools ...

Best regards.

Just tried to "echo 0 > /proc/sys/kernel/randomize_va_space" as root and apt-get install : same issue (segfault).

The problem appears on 2.6.31-15-generic-pae, too.

$ fgrep pae /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
 $ uname -a
Linux future 2.6.31-15-generic-pae #50-Ubuntu SMP Tue Nov 10 16:12:10 UTC 2009 i686 GNU/Linux

to Pekka:

If you have seen the above discussion then there was this link about the nx-emulation:
https://wiki.ubuntu.com/Security/Features#Non-Exec%20Memory

If you read it you will see that the generic-pae kernel is also using the nx-emulation if your processor does not support "nx".
As far as I see from your /proc/cpuinfo you don't have. So if this bug is really badly affect your work then you either use some older kernel (earlier then 2.6.30-3.4 because with that one was started the nx-emulation enabled) or use the 64bit kernel (if your processor is 64bit capable).

Correct. To avoid this issue, you must either run 64bit, or on a system
that has hardware support for the "NX" bit (check your BIOS, it is
frequently disabled).

@Jean you appear to be seeing a different problem; probably bad RAM. This problem is specific to DSO executables (built PIE or loaded via separate ld.so).

@Riku right, 64bit will not show it. 32bit PAE will only not show it if you have NX hardware (i.e. the CS-limit protections are not enabled in this case and are not used on 64bit at all).

Apparently only an issue with linux, not bash

This is still the case for 2.6.31-17-generic-pae (and non -pae). If there is no fix comming it would rock if someone could tell us how to rebuild the current kernel with the problematic patch dropped (in step-by-step fashion, as I usualy don't rebuild my kernel)?

I need to test further, but after doing the
echo 0 > /proc/sys/kernel/randomize_va_space
I don't get the out-of-memory errors in sb2 so far.

My machine is a T60 thinkpad and I found no way to enable NX. Sadly I don't even seem to be able to enter the bios.

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 14
model name : Genuine Intel(R) CPU T2400 @ 1.83GHz
stepping : 8
cpu MHz : 1000.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe constant_tsc arch_perfmon bts pni monitor vmx est tm2 xtpr pdcm
bogomips : 3657.84
clflush size : 64
power management:

(same fore 2nd core)

On Thu, Jan 14, 2010 at 07:53:51AM -0000, Stefan Kost wrote:
> This is still the case for 2.6.31-17-generic-pae (and non -pae). If
> there is no fix comming it would rock if someone could tell us how to

Diagnosing the why this is happening is proving difficult. No progress
has been made on it yet, unfortunately.

> rebuild the current kernel with the problematic patch dropped (in step-
> by-step fashion, as I usualy don't rebuild my kernel)?

Documentation on kernel rebuilds is here:
https://help.ubuntu.com/community/Kernel/Compile

The patches are these:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-karmic.git;a=commitdiff;h=1dc452d16c9329ce7b22e4552c367328e8a8c607
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-karmic.git;a=commitdiff;h=8bf095f8b62c5fdfe55a8c95be775ea62be7bc10
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-karmic.git;a=commitdiff;h=eb56f42a4bc9c671a19dcf8fd36eb3f341de7dfa

> My machine is a T60 thinkpad and I found no way to enable NX. Sadly I
> don't even seem to be able to enter the bios.

If other T60 documentation can be believed, press the "ThinkVantage"
button, press that during boot and then "Access BIOS" or "Start setup
utility".

> cpu family : 6
> model : 14
> model name : Genuine Intel(R) CPU T2400 @ 1.83GHz

This processor should have NX capabilities:
http://ark.intel.com/Product.aspx?id=27235 "Execute Disable Bit Yes"

Actually, setting /proc/sys/kernel/randomize_va_space to 0 makes this worse because it forces the brk (heap) area to immediately follow the text area, and the NX emulation patch performs randomization of exec regions no matter what randomize_va_space is. It seems the solution here is to fix up the brk starting position to be outside the area used for exec regions.

Here is a minimal test-case for brk crashing into other segments. This is nearly identical to the upstream issue seen with text ASLR, so the nx-emu allocation methods need to be adjusted to match what was done to upstream text ASLR to avoid brk collisions (with upstream, the text area is always at the highest segment address):
http://lkml.org/lkml/2007/8/14/325

This fixes the problems for me and tests out against normal system utilization, the minimal test case, and the original bash-loaded test-case, with randomize_va_space set to 0 or 2 ("1" still triggers the problem, but the combination of large PIE brk allocations and randomized-without-brk is not a sane combination).

https://lists.ubuntu.com/archives/kernel-team/2010-January/008314.html

I was able to compile the latest public karmic 32 bit generic kernel with the patch. When I have used that kernel I was not able to see anymore the reported problem. I would say that the fix was working for me.

Can someone tell me when could this patch appear on an average user desktop?
I would be interested about both Lucid and Karmic?
Can someone tell me also where can I check that which kernel version will include this patch as a release candidate?

Thank you in advance.

This has been committed to the Lucid tree now, and will be available there in 2.6.32-N.16:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-lucid.git;a=commitdiff;h=4ac4eee5ec83a0d9a41f7200c3879b41abd738c8

For Karmic, it will go into the next stable release update, so probably in a few weeks.

> For Karmic, it will go into the next stable release update, so probably in a few weeks.

What about Jaunty? I'm the original observer of this problem (Visi just took care of reporting it here) and unfortunately the information in the report about this problem not occurring with Jaunty / 2.6.28-15-generic is not correct because that's precisely the one I first observed this with. The latest 2.6.28-17-generic is affected as well.

(I couldn't figure out whether I should/could add Jaunty to the Affects-list and how to do that -- quite astonishingly Bug tracker user's manual at https://help.launchpad.net/ doesn't talk about reporting bugs!)

This bug was fixed in the package linux - 2.6.32-12.16

---------------
linux (2.6.32-12.16) lucid; urgency=low

  [ Andy Whitcroft ]

  * Revert "SAUCE: acpi battery -- delay first lookup of the battery until
    first use"
  * SAUCE: acpi battery -- move first lookup asynchronous
    - LP: #507211
  * [Config] update configs to cleanup generic configs
  * [Config] disable CONFIG_X86_CPU_DEBUG for amd64
  * [Config] enable USER_NS
    - LP: #480739, #509808

  [ Heiko Carstens ]

  * (pre-stable) driver-core: fix devtmpfs crash on s390
    - LP: #512370

  [ John Johansen ]

  * [Config] for server and virtual flavours make CONFIG_SCSI_SYM53C8XX_2=y
    - LP: #494565
  * [Config] VIRTIO=y for server/virtual flavours
    - LP: #494565

  [ Kay Sievers ]

  * (pre-stable) Driver-Core: devtmpfs - set root directory mode to 0755
    - LP: #512370

  [ Kees Cook ]

  * SAUCE: x86: brk away from exec rand area
    - LP: #452175

  [ Leann Ogasawara ]

  * [Upstream] e1000: enhance frame fragment detection
    - CVE-2009-4536
  * [Upstream] e1000e: enhance frame fragment detection
    - CVE-2009-4538

  [ Sebastian Kapfer ]

  * (pre-stable) Input: ALPS - add interleaved protocol support (Dell E6x00
    series)
    - LP: #296610

  [ Upstream Kernel Changes ]

  * inotify: do not reuse watch descriptors
    - LP: #485556
  * inotify: only warn once for inotify problems
  * revert "drivers/video/s3c-fb.c: fix clock setting for Samsung SoC
    Framebuffer"
  * memcg: ensure list is empty at rmdir
  * drm/i915: remove loop in Ironlake interrupt handler
  * block: Fix incorrect reporting of partition alignment
  * x86, mce: Thermal monitoring depends on APIC being enabled
  * futexes: Remove rw parameter from get_futex_key()
  * page allocator: update NR_FREE_PAGES only when necessary
  * x86, apic: use physical mode for IBM summit platforms
  * edac: i5000_edac critical fix panic out of bounds
  * x86: SGI UV: Fix mapping of MMIO registers
  * mfd: WM835x GPIO direction register is not locked
  * mfd: Correct WM835x ISINK ramp time defines
  * ALSA: hda - Fix missing capture mixer for ALC861/660 codecs
  * V4L/DVB (13868): gspca - sn9c20x: Fix test of unsigned.
  * reiserfs: truncate blocks not used by a write
  * HID: add device IDs for new model of Apple Wireless Keyboard
  * PCI/cardbus: Add a fixup hook and fix powerpc
  * Input: pmouse - move Sentelic probe down the list
  * asus-laptop: add Lenovo SL hotkey support
  * sched: Fix cpu_clock() in NMIs, on !CONFIG_HAVE_UNSTABLE_SCHED_CLOCK
  * sparc64: Fix NMI programming when perf events are active.
  * sparc64: Fix Niagara2 perf event handling.
  * i2c: Do not use device name after device_unregister
  * i2c/pca: Don't use *_interruptible
  * serial/8250_pnp: add a new Fujitsu Wacom Tablet PC device
  * sched: Fix task priority bug
  * vfs: Fix vmtruncate() regression
  * Linux 2.6.32.5
  * x86, msr/cpuid: Register enough minors for the MSR and CPUID drivers
  * V4L/DVB (13900): gspca - sunplus: Fix bridge exchanges.
  * Staging: asus_oled: fix oops in 2.6.32.2
  * Staging: hv: fix smp problems in the hyperv core code
  * tty: fix race in tty_fasync
  * ecryptfs: use after free
  * ecryptfs: initi...

@Hannu: I have not been able to reproduce this error on Jaunty (and Karmic was the first to carry the NX-emu patch which caused the problem). Can you please open a new bug for Jaunty if you're still seeing problems? Thanks!

I can confirm that the -proposed (-20) kernel fixes the testcase in qa-regression-testing (test-kernel-aslr-collisions.py)

This bug was fixed in the package linux - 2.6.31-20.57

---------------
linux (2.6.31-20.57) karmic-proposed; urgency=low

  [ Kees Cook ]

  * SAUCE: x86: brk away from exec rand area
    - LP: #452175

  [ Keng-Yu Lin ]

  * SAUCE: (drop after 2.6.31) input: Add hotkey quirk for Samsung
    Q210/P210 laptops
    - LP: #385683

  [ Pascal Bach ]

  * SAUCE: (drop after 2.6.31) input: Add hotkey quirk for Fujitsu Amilo
    1848+u
    - LP: #458503

  [ Stefan Bader ]

  * Revert "[Upstream] (drop after 2.6.31) acerhdf: fix fan control for
    AOA150 model"
    - LP: #494633
  * [Config] Disable CONFIG_USB_DEVICEFS
    - LP: #488274

  [ Upstream Kernel Changes ]

  * Revert "ext4: Don't update superblock write time when filesystem is
    read-only"
    - LP: #496816
  * Revert "ide: try to use PIO Mode 0 during probe if possible"
    - LP: #503430
  * Revert "isdn: isdn_ppp: Use SKB list facilities instead of home-grown
    implementation."
    - LP: #503430
  * nilfs2: fix kernel oops in error case of nilfs_ioctl_move_blocks
    - LP: #494633
  * cifs: don't use CIFSGetSrvInodeNumber in is_path_accessible
    - LP: #494633
  * cifs: clean up handling when server doesn't consistently support inode
    numbers
    - LP: #494633
  * cifs: clear server inode number flag while autodisabling
    - LP: #494633
  * CIFS: fix oops in cifs_lookup during net boot
    - LP: #494633
  * CIFS: Duplicate data on appending to some Samba servers
    - LP: #494633
  * rtl8187: Fix kernel oops when device is removed when LEDS enabled
    - LP: #494633
  * md: don't clear endpoint for resync when resync is interrupted.
    - LP: #494633
  * md/raid5: make sure curr_sync_completes is uptodate when reshape starts
    - LP: #494633
  * md/raid1/raid10: add a cond_resched
    - LP: #494633
  * ALSA: usb-audio: fix combine_word problem
    - LP: #494633
  * ALSA: hda - Dell Studio 1557 hd-audio quirk
    - LP: #494633
  * ALSA: AACI: fix AC97 multiple-open bug
    - LP: #494633
  * ALSA: AACI: fix recording bug
    - LP: #494633
  * jffs2: Fix memory corruption in jffs2_read_inode_range()
    - LP: #494633
  * sound: rawmidi: disable active-sensing-on-close by default
    - LP: #494633
  * sound: rawmidi: fix checking of O_APPEND when opening MIDI device
    - LP: #494633
  * sound: rawmidi: fix double init when opening MIDI device with O_APPEND
    - LP: #494633
  * sound: rawmidi: fix MIDI device O_APPEND error handling
    - LP: #494633
  * highmem: Fix race in debug_kmap_atomic() which could cause warn_count
    to underflow
    - LP: #494633
  * highmem: Fix debug_kmap_atomic() to also handle KM_IRQ_PTE, KM_NMI, and
    KM_NMI_PTE
    - LP: #494633
  * powerpc: Fix DEBUG_HIGHMEM build break from d4515646699
    - LP: #494633
  * kmap: fix build errors with DEBUG_HIGHMEM enabled
    - LP: #494633
  * V4L/DVB (13169): bttv: Fix potential out-of-order field processing
    - LP: #494633
  * V4L/DVB (13170): bttv: Fix reversed polarity error when switching video
    standard
    - LP: #494633
  * V4L/DVB (13109): tda18271: fix signedness issue in
    tda18271_rf_tracking_filters_init
    - LP: #494633
  * V4L/DVB (13107): tda18271: fix overflow in FM radio frequency...

I'm still seeing this problem (under scratchbox, trying to run the Maemo SDK) with 2.6.31-20.58 (and with 2.6.31-22.60). Was this reverted?

I haven't gotten the example proof of concept code to crash, but I cannot run af-sb-init.sh, which is required to start the Maemo SDK gui under scratchbox.

$ af-sb-init.sh start
/usr/bin/af-sb-init.sh: xrealloc: ../bash/general.c:762: cannot reallocate 262144 bytes (0 bytes allocated)

Reproducible for me on Debian wheezy/sid with custom kernel "3.3.6-pf". `echo 0 > /proc/sys/kernel/randomize_va_space` makes `/lib/ld-linux.so.2 /bin/bash` stop crashing with `cannot allocate 3 bytes (8192 bytes allocated)`. There is "nx" in /proc/cpuinfo flags.