Xorg segfaults during LiveCD installation using preseed file

Here are the backtraces from the systems where I experienced this crash, I realize they aren't 100% identical. Our installation procedure is very homogeneous so I think these might all be related, as they all occured using the same installation procedure.

124 - Vostro 3400
[ 115.517] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 115.517] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 115.518] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 115.518] Backtrace:
[ 115.518] 0: X (xorg_backtrace+0x3b) [0x80e94cb]
[ 115.518] 1: X (0x8048000+0x5dcd8) [0x80a5cd8] [ 115.518] 2: (vdso) (__kernel_rt_sigreturn+0x0) [0xd2040c]
[ 115.518] 3: /usr/lib/xorg/modules/drivers/intel_drv.so (0x1c8000+0xf2ea) [0x1d72ea]
[ 115.518] 4: /usr/lib/xorg/modules/drivers/intel_drv.so (0x1c8000+0x2662e) [0x1ee62e]
[ 115.518] 5: /usr/lib/xorg/modules/drivers/intel_drv.so (0x1c8000+0x26b4c) [0x1eeb4c]
[ 115.518] 6: X (0x8048000+0xe175e) [0x812975e]
[ 115.518] 7: X (0x8048000+0x23dd5) [0x806bdd5]
[ 115.518] 8: X (0x8048000+0x278d7) [0x806f8d7]
[ 115.518] 9: X (0x8048000+0x1a84c) [0x806284c]
[ 115.518] 10: /lib/libc.so.6 (__libc_start_main+0xe6) [0x277ce6]
[ 115.519] 11: X (0x8048000+0x1a441) [0x8062441]
[ 115.519] Segmentation fault at address 0x10
[ 115.519] Caught signal 11 (Segmentation fault). Server aborting
[ 115.519] Please consult the The X.Org Foundation support ^I at http://wiki.x.org for help.
[ 115.519] Please also check the log file at "/var/log/Xorg.0.log" for additional information.
[ 115.519] ubuntu kernel:
[ 115.608394] Xorg[5398]: segfault at 0 ip 08138a8e sp bfa81100 error 6 in Xorg[8048000+1ab000]

125 - Aspire One D255
[ 194.542] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.544] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.545] (WW) intel(0): intel_uxa_prepare _access: bo map failed: Cannot allocate memory
[ 194.545] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.546] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.547] (WW) intel(0): intel_uxa_prepare _access: bo map failed: Cannot allocate memory
[ 194.547] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.548] (WW) intel(0): i ntel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.548] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.549] (WW) intel(0): intel_uxa_prepare_access: bo map failed: Cannot allocate memory
[ 194.550] Backtrace:
[ 194.550] 0: X (xorg_backtrace+0x3b) [0x80e94cb]
[ 194.551] 1: X (0x8048000+0x5dcd8) [0x80a5cd8]
[ 194.551] 2: (vdso) (__kernel_rt_sigreturn+0x0) [0xe0b40c]
[ 194.551] 3: X (0x8048000+0x22a80) [0x806aa80]
[ 194.551] 4: X (0x8048000+0x278d7) [0x806f8d7]
[ 194.551] 5: X (0x8048000+0x1a84c) [0x806284c]
[ 194.551] 6: /lib/libc.so.6 (__libc_start_main+0xe6) [0x2efce6]
[ 194.551] 7: X (0x8048000+0x1a441) [0x806244...

Hi Daniel, for the crash we really gotta have a full backtrace, either via apport automatically or by hand via gdb.

Some directions are available at http://wiki.ubuntu.com/X/Backtracing

The crash (this one and the ones experienced previously) are due to programming errors in how X handles out of memory situations. So, it's definitely worth fixing (and I look forward to getting your full backtrace), but it doesn't solve the underlying issue which is why are the text boxes running out of memory? And is it system memory or graphics memory that's running out?

Unfortunately, these logs don't shed light on this question, but I think you could analyze it a bit, by watching memory counts while the installation proceeds. You'll have to figure out what the best way in your infrastructure to do this, but I might run top, cat /proc/meminfo, xrestop, cat /proc/slabinfo, etc. and watch for a number that either diminishes towards 0 or goes up unboundedly until the X crash occurs.

If you can modify the installer image, please get a crashdump from X by adding '-core' to it's cmdline options (wherever those are set).

Upstream isn't totally convinced this is a driver bug, but that the X client (=installer) is leaking memory. Please get a dump of 'cat /sys/kernel/debug/dri/0/i915_gem_objects' to see that it's the driver and/or client leaking memory, and not something below the stack.

Fwiw, we've been getting these out-of-memory-leading-to-X-crash bugs for about a month. The first reported bug I can recall was Mario's bug #705078 dated 2011-01-19.

I see ubiquity 2.5.6 was uploaded on 2011-01-17. If it is an X client leaking and we think the installer is a suspect, then the ubiquity changes in that version would be worth reviewing. In addition to the ubiquity changes (which sound innocuous from the changelog entries), it also incuded updates to grub-installer 1.57ubuntu2, netcfg 1.57ubuntu3, partman-auto 93ubuntu3, partman-target 71ubuntu1, user-setup 1.28ubuntu12, so I guess any of those could be candidates.

Is there a way we can pinpoint what X client is gobbling the memory?

This crash is now happening for us pretty reliably even without our test configuration, i.e. with a "pristine" image. Bug 720235 details one of the "look ma, no hands" crashes we had, and I just had another one with the latest (today) desktop image.

post-crash, here's the i915_gem_objects file asked for by Timo:
917 objects, 64212992 bytes
287 [287] objects, 58945536 [58945536] bytes in gtt
  0 [0] active objects, 0 [0] bytes
  6 [6] pinned objects, 4370432 [4370432] bytes
  281 [281] inactive objects, 54575104 [54575104] bytes
  0 [0] freed objects, 0 [0] bytes
6 pinned mappable objects, 4370432 bytes
33 fault mappable objects, 9695232 bytes
536866816 [268435456] gtt total

here's /proc/meminfo and free (at the end)

MemTotal: 1918868 kB
MemFree: 1122560 kB
Buffers: 93568 kB
Cached: 461948 kB
SwapCached: 0 kB
Active: 248412 kB
Inactive: 466556 kB
Active(anon): 163432 kB
Inactive(anon): 89776 kB
Active(file): 84980 kB
Inactive(file): 376780 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 1044544 kB
HighFree: 409540 kB
LowTotal: 874324 kB
LowFree: 713020 kB
SwapTotal: 1951740 kB
SwapFree: 1951740 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 159432 kB
Mapped: 54344 kB
Shmem: 93760 kB
Slab: 44560 kB
SReclaimable: 30888 kB
SUnreclaim: 13672 kB
KernelStack: 2240 kB
PageTables: 4552 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2911172 kB
Committed_AS: 1162748 kB
VmallocTotal: 122880 kB
VmallocUsed: 32816 kB
VmallocChunk: 81464 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 16376 kB
DirectMap4M: 892928 kB

             total used free shared buffers cached
Mem: 1918868 796308 1122560 0 93568 461952
-/+ buffers/cache: 240788 1678080
Swap: 1951740 0 1951740

Now what seems odd here is that this system has a swap file (picked up /dev/sda5 from a previous installation attempt), so it would seem to me that, if a process were gobbling up memory, the system would start thrashing before X crashed. In my experience this doesn't happen. But admittedly I have no idea how X handles memory for applications so I might be way off the mark here.

Since the problem is reproducible on the pristine image, I'll see what I can do to monitor the list of processes for runaway memory consumption, and report back here as soon as I have something.

Thanks all for your help!

This should be reproducable on pretty much any atom netbook with a 945 from what I can see, just boot a natty daily livecd and let it sit on the welcome screen for a few minutes. You can watch /var/log/Xorg.0.log and the bo map errors will start up before it eventually crashes. I think I have narrowed it down to this change in ubiquity:

http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/natty/ubiquity/natty/revision/380#src/panel/panel.c

reverting that change with the attached patch fixes the problem here but it could use more testing.

I have reproduced this on a Lenovo Thinkpad X201 (bug 720238) and Dell Vostro 3400 (bug 721220) - I just re-checked the Dell just for kicks and it's sitting here, all crashed, in front of me.

Is there a way for me to test a build with Robert's patch applied? I can test it across all our certification systems and let you know how it behaves.

Not sure how I associated with 2.5.6 with 1/17. But 2.5.9 (rev 380) on 1/14 is the better match.

I've prepared a ubiquity package with Cyrus Lien change from rev 380 reverted. Robert's testing indicates this seems to eliminate the problem. I seem to be unable to reproduce the original bug on my 945 mini after multiple tries, so can't confirm the fix.

https://launchpad.net/~bryce/+archive/durian

But I'm hoping Daniel can use my package to confirm reverting the patch fixes the error.

If it does make the issue go away, I'm not sure it can be classified as a true fix, since it'll revert the fix for bug #693300 but at least that's just a cosmetic issue that I guess the Ubiquity guys can figure out a better fix for.

I don't know how to test this with a livecd image, but Daniel you guys are the experts at testing images so presumably you have A Way. Let me know when you've verified it fixes it and we can proceed from there.

The easiest way to prove it's something in the panel leaking would be to kill the panel as soon as the session starts (or remove the binary in an early command temporarily).

And actually glancing at the code, perhaps the leak is a little bit further up. I don't see any calls to unreference the memory of the pixmap allocation after setting it after each expose event. Here's another patch that might help address this.

I'm building an image for testing of Bryce's fix. In the meanwhile, I did some quick testing using Mario's suggestions.

Like Robert did, I just let the Thinkpad X201 sit there while monitoring processes and Xorg.0.log. Sure enough, the panel process starts using up the CPU, going up to about 80%, and eventually exiting with status 1 (I attached gdb to it, it didn't segfault or anything). At around that time, the bo map errors start appearing in the log.

I then started the install process, but quickly switched to a terminal and killed the panel process. This time the installation went on without a hitch.

So if the panel is around, it eventually dies and usually takes Xorg with it. If the panel is killed the system can install and/or remain sitting there without problems.

I was able to replicate this behavior on the Dell Vostro 3400, if left as-is the panel eventually crashes, if the panel is killed then it works OK.

I'll update as soon as I have some more results.

I built an image based on latest Natty daily and incorporating Bryce's fixed ubiquity. It looks promising, already the panel is no longer misbehaving and it doesn't consume CPU like it used to. I was able to complete the installation process on the X201 where the failure was so easily reproduced previously.

Also, on the Dell, whereas the "pristine" image just crashed eventually, the image with the fix is able to go further into the installation, until I hit an unrelated Ubiquity bug (I'll report on that one separately). There are no panel-related crashes on this one.

So it's looking good, my next step is to plug the image in our testing infrastructure and see how most systems behave. I'll report back as soon as I have something else.

Hi All,

I used the Natty daily with Bryce's fix for ubiquity on our test machines, I'm glad to report that none of them dropped to LiveCD environment or rebooted. The vast majority completed the installation process correctly, the few that failed did so for unrelated reasons. This comprises about 70 laptops of different makes and hardware specifications.

I think, then, that Bryce is on the right track with his fix, seeing as to how it solves the issue, though at the cost of reintroducing the problem originally fixed by the offending line (bug # 693300).

I also created a CD image with Mario's patch from comment #12 (just that one, excluding Bryce's patch) and although I was only able to test it on a virtual machine, I see the same odd behavior from the panel, i.e. it gobbles up 60-70% of the CPU. If you need me to, I can test this on actual hardware next week.

Thanks!

Hi Daniel,

So this is my take on the issue based on your results :

1) the function call commented out by Bryces patch fixes on_expose from
being called continuously. That function creates a new expose event making
on_expose call again.

2) the above causes the high cpu regardless of memory leak

3) the patch is provided fixes the memory leak for the pixmap that isn't
cleared.

So combined the two patches seem correct, but the original problem with
necessity to refresh the panel remains.
On Feb 25, 2011 8:05 PM, "Daniel Manrique" <email address hidden>
wrote:
> Hi All,
>
> I used the Natty daily with Bryce's fix for ubiquity on our test
> machines, I'm glad to report that none of them dropped to LiveCD
> environment or rebooted. The vast majority completed the installation
> process correctly, the few that failed did so for unrelated reasons.
> This comprises about 70 laptops of different makes and hardware
> specifications.
>
> I think, then, that Bryce is on the right track with his fix, seeing as
> to how it solves the issue, though at the cost of reintroducing the
> problem originally fixed by the offending line (bug # 693300).
>
> I also created a CD image with Mario's patch from comment #12 (just that
> one, excluding Bryce's patch) and although I was only able to test it on
> a virtual machine, I see the same odd behavior from the panel, i.e. it
> gobbles up 60-70% of the CPU. If you need me to, I can test this on
> actual hardware next week.
>
> Thanks!
>
> --
> You received this bug notification because you are a member of Ubuntu
> Installer Team, which is subscribed to ubiquity in ubuntu.
> https://bugs.launchpad.net/bugs/714829
>
> Title:
> Xorg segfaults during LiveCD installation using preseed file

Spoke with cjwatson on IRC, he'll take it from here; he's about to roll a new ubiquity upload with this.

I'm merging these two patches and have reopened bug 693300. Thanks all.

This bug was fixed in the package ubiquity - 2.5.19

---------------
ubiquity (2.5.19) natty; urgency=low

  [ Colin Watson ]
  * Automatic update of included source packages: base-installer
    1.116ubuntu1, console-setup 1.57ubuntu8, flash-kernel 2.28ubuntu15,
    partman-auto 93ubuntu8, partman-target 71ubuntu2.

  [ Bryce Harrington ]
  * Revert "Queue a redraw of the panel after setting the background". This
    change is implicated in a memory leak leading to OOM conditions and
    eventual crash of Xserver. (Possible fix for LP: #714829, would reopen
    693300)

  [ Mario Limonciello ]
  * Fix reference leak in panel set_background function.
 -- Colin Watson <email address hidden> Mon, 28 Feb 2011 19:54:45 +0000

[Fixed in ubiquity, nothing really to do on the X end.]