libc6 crash while running 'xm'

I am running into this issue as well on a 12.04 domU with a AMD Opteron 6272 based box.

[14553.237586] apport[18522] trap invalid opcode ip:7f1d632625fc sp:7ffff8701390 error:0 in libm-2.15.so[7f1d63220000+f9000]

Debugging info

(gdb) exec-file python-dbg
(gdb) run
Starting program: /usr/bin/python-dbg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 2.7.3 (default, Apr 20 2012, 22:01:19)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print test

Program received signal SIGILL, Illegal instruction.
__ieee754_exp_fma4 (x=-0.5) at ../sysdeps/ieee754/dbl-64/e_exp.c:63
63 ../sysdeps/ieee754/dbl-64/e_exp.c: No such file or directory.
(gdb) bt
#0 __ieee754_exp_fma4 (x=-0.5) at ../sysdeps/ieee754/dbl-64/e_exp.c:63
#1 0x000000000058509f in ?? ()
#2 0x00000000009bde80 in ?? ()
#3 0x0000000100c52a10 in ?? ()
#4 0x0000000000417380 in ?? ()
#5 0x00000000009d2970 in ?? ()
#6 0x0000000000000000 in ?? ()
(gdb) info address __ieee754_exp_fma4
Symbol "__ieee754_exp_fma4" is a function at address 0x7ffff6cc35f0.
(gdb) disassemble 0x7ffff6cc35f0,+30
Dump of assembler code from 0x7ffff6cc35f0 to 0x7ffff6cc360e:
   0x00007ffff6cc35f0 <__ieee754_exp_fma4+0>: push %rbp
   0x00007ffff6cc35f1 <__ieee754_exp_fma4+1>: mov %rsp,%rbp
   0x00007ffff6cc35f4 <__ieee754_exp_fma4+4>: and $0xffffffffffffffe0,%rsp
   0x00007ffff6cc35f8 <__ieee754_exp_fma4+8>: add $0x10,%rsp
=> 0x00007ffff6cc35fc <__ieee754_exp_fma4+12>: vmovsd %xmm0,-0x20(%rsp)
   0x00007ffff6cc3602 <__ieee754_exp_fma4+18>: mov -0x20(%rsp),%rax
   0x00007ffff6cc3607 <__ieee754_exp_fma4+23>: mov %rax,%rcx
   0x00007ffff6cc360a <__ieee754_exp_fma4+26>: shr $0x20,%rcx

vmovsd is an avx instruction, but avx isn't supported according to cpu flags.

# grep flags /proc/cpuinfo | head -1
flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm rep_good nopl aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch xop fma4 perfctr_core arat cpb

perhaps eglibc is assuming if fma4 is supported, avx must be too.

thanks
jim

Status changed to 'Confirmed' because the bug affects multiple users.

Hi

Some additional information that leads me to believe this is a xen issue.

gcc 4.6.3 code has the following

#define OPTION_MASK_ISA_FMA4_SET \
  (OPTION_MASK_ISA_FMA4 | OPTION_MASK_ISA_SSE4A_SET \
   | OPTION_MASK_ISA_AVX_SET)

So when you use -mfma4 option it also enables avx instruction set.

I believe xen likely should be filtering out fma4 support if its doing the same for avx.

thanks
jim

Hi

It appears this issue can be worked around by adding the following options to xen's command line in grub.

cpuid_mask_ext_ecx=0xFFFEFFF cpuid_mask_ecx=0xEFFFFFF

This masks both FMA4 and AVX from cpuid for dom0 and all domU's

Also note that xen has a cpuid option for domU configs that allows to you mask these as well, but it doesn't seem to work for userspace making direct cpuid calls. The kernel will see that fma4 isn't there, but glibc still see's it as available and you get crashes.

jim

Hello,

cpuid_mask_ext_ecx=0xFFFEFFF cpuid_mask_ecx=0xEFFFFFF didn't work with my setup, but I found a similiar workaround:

Reducing the instruction set to an older K10 opteron architecture helps: cpuid_mask_cpu=fam_10_rev_c

Hi

In briefly talking with the xen folks, this is a glibc bug. I filed the following bug report with them.

http://sourceware.org/bugzilla/show_bug.cgi?id=14059

jim

HI

The attached patch against 2.15-0ubuntu10 resolves the issue for me.

It does this by disabling FMA4 if AVX is unavailable.

thanks
jim

The attachment "fma4-depends-avx.diff" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

Solved by Redhat: http://sources.redhat.com/git/gitweb.cgi?p=glibc.git;a=commitdiff;h=1a0994f5356214e8af8a1c1cc33fbf74a7ac8993 & http://sourceware.org/bugzilla/show_bug.cgi?id=14059#c7 can Ubuntu measure up to them, please?

Same on AMD Opteron(tm) Processor 4226 with ubuntu 12.04

libc6: 2.15-0ubuntu10
linux-image-generic: 3.2.0.24.26
xen-hypervisor-4.1-amd64: 4.1.2-2ubuntu2

cpuid_mask_ext_ecx=0xFFFEFFF cpuid_mask_ecx=0xEFFFFFF didnt work.

cpuid_mask_cpu=fam_10_rev_c works for me

Same on AMD FX(tm)-8120 Eight-Core Processor Ubuntu 12.04 amd 64 - fresh install

-libc6: 2.15-0ubuntu10
-xen-hypervisor-4.1-amd64: 4.1.2-2ubuntu2
-linux-image-3.2.0-24-generic: 3.2.0-24.39

cpuid_mask_ext_ecx=0xFFFEFFF cpuid_mask_ecx=0xEFFFFFF didnt work.

cpuid_mask_cpu=fam_10_rev_c didnt work either.

same syslog: xm[1407] trap invalid opcode ip:7fd7362da5fc sp:7fffaee56150 error:0 in libm-2.15.so[7fd736298000+f9000]

Attached is a debdiff for the patch proposed in comment #7 by jwestfall: disabling FMA4 if AVX support is unavailable.

A package containing this patch has been pushed to my lpbugs PPA at: https://launchpad.net/~kmdm/+archive/lpbugs/

IMPORTANT NOTE: This is *UNTESTED* since due to commercial constraints we had to re-image our affected server hardware to a 32-bit installation and so no longer have the ability to perform adequate testing.

I installed the patch from Kenny Millington's PPA.
It's now working as intended.

I'm fixing this in quantal right now, and will backport to precise, with upstream's "more correct" fix. I'd love testers, once I get this all done here, as I have no AMD hardware to test on, only Intel. (The upstream fix was widely tested, however, so I have a fair amount of trust in it)

Note that this may also need fixing in lucid, but that backport might cause a bit of pain. Looking into it now.

I am also affected by this bug (dual Opteron 12 core 6238), but when starting /etc/init.d/xcp-xapi
Fresh install of Ubuntu 12.04

About to try the cpuid_mask_ext_ecx fix.
Adam: I'll test the backport if you need?

# /etc/init.d/xcp-xapi start
Error: Connection refused (calling connect )

syslog:
Jun 19 14:17:08 host1 xapi: [ info|host1|0 thread_zero||watchdog] (Re)starting xapi...
Jun 19 14:17:08 host1 kernel: [ 8209.795675] xapi[32494] trap invalid opcode ip:7f8fe3cbae11 sp:7fff711645c0 error:0 in libm-2.15.so[7f8fe3c78000+f9000]
Jun 19 14:17:09 host1 kernel: [ 8210.027119] apport[32496] trap invalid opcode ip:7ff62e2625fc sp:7fffe5a49970 error:0 in libm-2.15.so[7ff62e220000+f9000]
Jun 19 14:17:09 host1 kernel: [ 8210.027231] Process 32496(apport) has RLIMIT_CORE set to 1
Jun 19 14:17:09 host1 kernel: [ 8210.027245] Aborting core
Jun 19 14:17:09 host1 xcp-fe: 32494 (/usr/sbin/xapi -nowatchdog -writereadyfile /var/run/xapi_startup.cookie -writ...) exitted with signal -5
Jun 19 14:17:09 host1 xapi: [ info|host1|0 thread_zero||watchdog] received signal -5
Jun 19 14:17:09 host1 xapi: [ info|host1|0 thread_zero||watchdog] xapi watchdog exiting.
Jun 19 14:17:09 host1 xapi: [ info|host1|0 thread_zero||watchdog] Fatal: xapi died with signal -5: not restarting (watchdog never restarts on this signal)

This bug also affects the ubuntu install on our dual AMD 6238 cpus while running XenServer 6.0.201 and also Xen Cloud Platform 1.5

The installer shows:
Warning: Failure trying to run: chroot /target dpkg --force-depends --install /var/cache/apt/archives/debconf_1.5.42ubuntu1_all.deb

In the log console while the installer is running (ctrl-alt-f4) I see the following (which leads me to believe it's the above bug):

kernel: [ 2051.425647] pycompile[20389] trap invalid opcode ip:7f84167cae11 sp:7fffc8ca98e0 error:0 in libm-2.15.so[7f8416788000+f9000]
deboostrap: Illegal instruction
debootstrap: dpkg: error processing debconf (--install):

The above issue was also seen in bug 982430

Unfortunately this is still affecting me, and I can't get Ubuntu 12.04 LTS server installed on fully patched XenServer 6.0.2
I tried to get around it by installing Ubuntu 10.04.4 LTS and upgrading to 10.10->11.04->11.10 (all fine) and then the upgrade from 11.10->12.04 failed with the Python upgrade crashing aborting the install with more than half of the packages not upgraded, and keeps crashing each time I try to apt-get dist-upgrade:

Setting up python2.7-minimal (2.7.3-0ubuntu3) ...
Illegal instruction (core dumped)
dpkg: error processing python2.7-minimal (--configure):
 subprocess installed post-installation script returned error exit status 132
Errors were encountered while processing:
 python2.7-minimal

syslog shows:

Jul 10 10:28:50 newserver kernel: [36302.562366] apport[51047] trap invalid opcode ip:7fe6a8b6ae11 sp:7fff9372dd70 error:0 in libm-2.15.so[7fe6a8b28000+f9000]
Jul 10 10:28:50 newserver kernel: [36302.562455] Process 51047(apport) has RLIMIT_CORE set to 1
Jul 10 10:28:50 newserver kernel: [36302.562466] Aborting core
Jul 10 10:28:50 newserver kernel: [36303.299834] apt-check[51134] trap invalid opcode ip:7fa90f94ae11 sp:7fffabddde60 error:0 in libm-2.15.so[7fa90f908000+f9000]
Jul 10 10:28:50 newserver kernel: [36303.376493] apport[51135] trap invalid opcode ip:7fd43d72ae11 sp:7fff2b985dd0 error:0 in libm-2.15.so[7fd43d6e8000+f9000]
Jul 10 10:28:50 newserver kernel: [36303.376581] Process 51135(apport) has RLIMIT_CORE set to 1
Jul 10 10:28:50 newserver kernel: [36303.376593] Aborting core
Jul 10 10:29:08 newserver kernel: [36321.163589] python2.7[51660] trap invalid opcode ip:7f2d5d63ae11 sp:7fff61e1bf90 error:0 in libm-2.15.so[7f2d5d5f8000+f9000]
Jul 10 10:29:08 newserver kernel: [36321.240592] apport[51661] trap invalid opcode ip:7f7fbb46ae11 sp:7fff10f78d60 error:0 in libm-2.15.so[7f7fbb428000+f9000]
Jul 10 10:29:08 newserver kernel: [36321.240679] Process 51661(apport) has RLIMIT_CORE set to 1
Jul 10 10:29:08 newserver kernel: [36321.240690] Aborting core

Similar problem on my AMD FX-8120 and Ubuntu 12.10 x64
https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1029175

When I install Ubuntu 12.04 as Guest System on a Xen Server I get this dmesg-errors all the time:

Jul 25 05:05:25 xxxx kernel: [129839.757278] landscape-sysin[27526]
trap invalid opcode ip:7f3c3e11a5fc sp:7fff1a08fc50 error:0 in
libm-2.15.so[7f3c3e0d8000+f9000]
Jul 25 05:05:25 xxxx kernel: [129839.797121] apport[27527] trap invalid
opcode ip:7f434d23a5fc sp:7fff3ef75670 error:0 in
libm-2.15.so[7f434d1f8000+f9000]

And when I type a non-existing command into the shell I get a core-dumped-error:

root@xxxx:~# asdf
Illegal instruction (core dumped)

For me this is a severe problem (not "medium")... I will have to remove Ubuntu 12.04 from the supported vserver-distributions until this problem is solved!

This bug was fixed in the package eglibc - 2.15-0ubuntu16

---------------
eglibc (2.15-0ubuntu16) quantal; urgency=low

  * Backport fix from 2.16 to fix htons() conversion errors on non-x86
    architectures, by correctly casting to uint16_t (LP: #1016349)
  * Restore missing AT_EMPTY_PATH definition in fnctl.h (LP: #1010069)
  * Backport FMA4/AVX detection from glibc 2.16 (LP: #956051, #979003)
  * Backport fixups to AVX-using code to match the detection backport.
 -- Adam Conrad <email address hidden> Thu, 09 Aug 2012 15:15:53 -0600

Can I ask that the Precise fix gets put into 12.04.1 LTS as at the moment the 12.04 LTS installer will not install on XenServer, this would make a big difference rather than having to try and install 11.10 and then upgrade (if the correct package will be used in the upgrade).
This is quite an urgent issue for Xen/XenServer/XCP users meaning that the LTS edition can't be used on a very large range of servers and virtual machines either by installing it directly or by upgrading.
Thanks very much.
Regards,
Tom Worley

I'd like to second Tom Worley's comments, this bug is a pain.

Me, too! A lot of new servers are waiting since april for being tested without this bug!
It is really to consider to look for alternative server distributions :(

Thanks for the support guys, but the deadline for the freeze on 12.04.1 LTS has been and gone.
Adam: Would it be at least possible to push the update out now even though it won't make it into the installation media, so that we can at least upgrade or use the net-install image?
Thanks,
Tom.

Bug was fixed almost 2 weeks ago, my FX-8120 works fine.

Is there anywhere we can get the updated deb for glibc from?
This is a show-stopper for us.

Thanks,
TS.

There's an upload in the queue for precise-proposed for this, it's just pending review. As soon as it makes it to precise-updates, I'll be sure to re-spin debian-installer so that at least the netboot images are sane (and it will make it to CD media for 12.04.2)

Hello Adrian, or anyone else affected,

Accepted eglibc into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/eglibc/2.15-0ubuntu10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I just enabled the proposed repository and installed eglibc-2.15-0ubuntu10.1 but it didn't change anything.

I still get the same errors as before when entering a non-existing command in the shell.

https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/956051/comments/20

I just did a fresh install of ubuntu 12.04 and afterwards installed the proposed eglibc-source package.

But as I see, this package installed nearly everything in /usr/src/eglibc ... do I have to do something more to get it installed?

Alexander, the package you'd want to be upgrading to the precise-proposed version would be libc6.

Thank you Adam! Now it looks like everything is working :-)

I'm afraid it still isn't a bulletproof fix.

The dom0 boots up fine, but while debootstrapping a precise machine I encountered a core dump with unknown exception during debconf installation.

I doubled checked and booted without xen and the debootstrapping went just fine.

debootstrap seems to use the current non fixed libc files which could explain the above behaviour.

Lets see what happens when the fixed eglibc will be published.

I can confirm that after adding proposed to a 11.10 install, then apt-get update and do-release-upgrade'ing it to 12.04 the install now works and so far so good!
Hopefully it'll get into main and the netinstall soon!
Thanks for the work Adam.

4 days running on several servers with no issues at all, all oneirc -> precise guest OS upgrades using the proposed repository, on XenServer 6.0.2 with AMD cpus (dual Opteron 12 core 6238) on the hosts.

Nearly a month on multiple XenServer VMs running 12.04.1 and no problems at all.
Can I presume the update hasn't made it into today's libc6 security updates?
Thanks.

Unfortunately not, worse still it appears the fixed version (2.15-0ubuntu10.1) appears to have been removed from precise proposed... I fortunately have the debs in my cache still but I no longer can get the libc6-dev debs to install build-essential.

This really needs to be fixed as a matter of urgency.

They were removed due to being superseded by a security update. Redoing the
update (+ the security fix) is in progress.

Thanks for that Scott, how long will it likely be before this is merged back into precise main repos?

Thanks Scott, do you have any idea on a rough ETA (and also when we could expect this new version to make it into the main repositories?)

I am told "today" for -proposed. It's generally a week or so from -proposed
to the main repositories.

Any update on this? I'm completely blocked on this at the moment.

Hello Adrian, or anyone else affected,

Accepted eglibc into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/eglibc/2.15-0ubuntu10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Tested across several machines, both Xen Dom0's where this specific error was occurring and other machines which had the same error produced during some dpkg operations are now fixed.

I am affected by this bug too (rented 1und1 Virtual Server) and I can confirm that the following steps solve the problem (taken from https://wiki.ubuntu.com/Testing/EnableProposed)

1.) add precise-proposed to /etc/apt/sources.list as described

 deb http://archive.ubuntu.com/ubuntu/ precise-proposed restricted main multiverse universe

2.) create /etc/apt/preferences.d/proposed as described in https://wiki.ubuntu.com/Testing/EnableProposed

3.) sudo apt-get install libc6/precise-proposed libc-bin/precise-proposed

Likewise, I can confirm this latest package fixes the issue for me on various XenServer DomU instances running Precise 12.04.1 LTS 64 bit with Hosts/Dom0 on dual AMD 6238 cpus.

I needed a step 2.5 to update my 1and1 Virtual Server. Refer to https://wiki.ubuntu.com/Testing/EnableProposed for steps 1 and 2. Steps 2.5 and 3 are commands that can be copied and pasted.

1.) add precise-proposed to /etc/apt/sources.list as described in https://wiki.ubuntu.com/Testing/EnableProposed

2.) create /etc/apt/preferences.d/proposed as described in https://wiki.ubuntu.com/Testing/EnableProposed

2.5) apt-get update

3.) apt-get install libc6/precise-proposed libc-bin/precise-proposed

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package eglibc - 2.15-0ubuntu10.3

---------------
eglibc (2.15-0ubuntu10.3) precise; urgency=low

  * Backport fixes for dbl-64 and ldbl-128 issues (LP: #1000498)
  * Backport another FMA support patch from glibc master branch.

eglibc (2.15-0ubuntu10.2) precise-security; urgency=low

  * SECURITY UPDATE: stack buffer overflow in vfprintf handling
    (LP: #1031301)
    - debian/patches/any/CVE-2012-3406.patch: switch to malloc when
      array grows too large to handle via alloca extension
    - CVE-2012-3406
  * SECURITY UPDATE: stdlib strtod integer/buffer overflows
    - debian/patches/any/CVE-2012-3480.patch: rearrange calculations
      and modify types to void integer overflows
    - CVE-2012-3480

eglibc (2.15-0ubuntu10.1) precise; urgency=low

  * Backport fix from 2.16 to fix htons() conversion errors on non-x86
    architectures, by correctly casting to uint16_t (LP: #1016349)
  * Restore missing AT_EMPTY_PATH definition in fnctl.h (LP: #1010069)
  * Backport FMA4/AVX detection from glibc 2.16 (LP: #956051, #979003)
  * Backport fixups to AVX-using code to match the detection backport.
  * Backport fix from 2.16 for sscanf/realloc deadlock (LP: #1028038)
  * Backport for bogus FPE on underflow for exp(double) (LP: #1007457)
 -- Adam Conrad <email address hidden> Wed, 03 Oct 2012 15:58:02 -0600

Is there any way this can get pushed to precise instead of precise-updates? See https://bugs.launchpad.net/ubuntu/+bug/1007439 - it's now impossible to use xen-tools (e.g. xen-create-image) to make a 12.04 guest because this bug is hit almost instantly during the debootstrap install process (which doesn't comprehend -updates).

Although that bug is related to installing a precise guest under XenServer, you can observe this behavior by trying to install a precise guest on a precise domU with a CPU that is affected by this bug.

Justin, here is a workaround for xen-create-image:
https://bugs.launchpad.net/ubuntu/+bug/1007439/comments/15