REGRESSION: glibc 2.7-9ubuntu1 NSS module broken due to toolchain changes

Confirmed. This is EXTREME bad juju. Nothing works. I, too, am afraid to shut down. This is libc6 2.7-9ubuntu1 on Hardy i386. The system went into the crapper during an apt-get update. The previous version of libc was 2.7-5ubuntu2.

i confirm this bug, had to boot into live cd and dpkg -x oldlibc /pathtomounted borked system to boot back up

any more details on the exact procedure of that workaround? That's hilarious, but I'm cowering.

This was really scary. On a reboot I couldn't even get a shell in recovery mode.

"malloc: unknown:0: assertion botched
free: called with unallocated block argument
last command: sudo mount
Aborting... init: tty1 main process ended, respawning"

Anyways, people have talked about solutions here. http://ubuntuforums.org/showthread.php?t=722886

Oh noes. I'm hit by this too. I don't want to reboot...

I was able to recover my system without a reboot by installing the packages, still in the archive, of version 2.7-5ubuntu2. For whatever reason, su is able to run while sudo is not. Obviously this will only help you if you have a root password.

If you don't have a root password, a solution that I used to downgrade the package which involves a reboot follows:

- If you have upgraded, but didn't reboot yet, get the older version from your favorite mirror and save it to somewhere you know. An example link is http://us.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6_2.7-5ubuntu2_i386.deb
- If you can't launch a browser for whatever reason, use alt+f2, type this: uxterm -e /bin/dash and use links to download.
- reboot.
- on the grub, select your favorite kernel, hit the 'e' key, select the line starting with 'kernel', change the 'ro' word on the line to 'rw' and add 'init=/bin/dash' to the end.
- wait for 10 seconds, nothing will be displayed when the system is booted into a root shell. try 'ls' to see whether it's done.
- cd /to/the/path/you/downloaded/the/package and use dpkg to downgrade: 'dpkg -i libc6_2.7-5ubuntu2_i386.deb'
- type 'sync' to make sure that everything gets written for sure
- hit ctrl+alt+del to reboot. you should be set now.

-HTH

yep, i just spent hours figuring this out. in the end, i just did a clean install, then fixed my /etc/apt/preferences to say:

Package: libc6
Pin: version 2.7-5ubuntu2
Pin-Priority: 1001

And it's going to stay this way until libc6 is fixed :x

Manually copying the contents of the older .deb files, as described in the forum thread, works perfectly.
Flawless victory.

Someone said su worked, not here, not even bash can be started... one bricked workstation :)

$ bash

malloc: ../bash/subst.c:3472: assertion botched
free: called with unallocated block argument
Aborting...complete: usage: complete [-abcdefgjksuv] [-pr] [-o option] [-A action] [-G globpat] [-W wordlist] [-P prefix] [-S suffix] [-X filterpat] [-F function] [-C command] [name ...]

malloc: ../bash/subst.c:3472: assertion botched
free: called with unallocated block argument
Aborting...
malloc: ../bash/subst.c:3472: assertion botched
free: called with unallocated block argument
Aborting...Aborted (core dumped)

ok, all u do is boot your live cd and open a terminal, wget http://us.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6_2.7-5ubuntu2_i386.deb
then mount your ubuntu drive whichever that is, it may be here:/media/disk-1 or something or another, open a terminal and then dpkg -x libc6_2.7-5ubuntu2_i386.deb /media/drivewhereubuntu is already installed then reboot

Here's how I did:

* Boot from the Live CD/DVD
* Depending on the Live CD you are using (Gnome is better, cause you'll only need to double click the appropriate icon on your desktop), you'll have to manually mount your root partition or not.
If you need to mount it (so, if you are using KDE), the best way is:

sudo su
mkdir /myroot
mount /dev/sda1 /myroot (where /dev/sda1 should be replaced in case)

* Then, download the old glibc:

chroot /myroot wget http://us.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6_2.7-5ubuntu2_i386.deb

* And run dpkg in order to downgrade it:

chroot /myroot dpkg -x libc6_2.7-5ubuntu2_i386.deb

* After, you can always remove the package from your root dir.

Thanks Lillipuziano! Your solution works perfect ;) I got my system back in few minutes ;)

Kyle M Weller, Lillipuziano, thanks a lot you for your solutions. Works great for me.

@jyio: What if you don't have a /etc/apt/preferences?

Just create one in kate, copy your lines into that new preferences and save it to /etc/apt and ?

Or how else?

@Fred: that's what I just did, and "apt-cache policy libc6" now shows a higher priority for the currently installed version

This bug was fixed in the package glibc - 2.7-9ubuntu2

---------------
glibc (2.7-9ubuntu2) hardy; urgency=low

  * Clear out LDFLAGS when building; glibc isn't happy building with
    -Wl,-Bsymbolic-functions. LP: #201673

 -- Steve Langasek <email address hidden> Thu, 13 Mar 2008 08:34:28 +0000

How long does it take to new packages to appear in repos? I'm using the main server with Synaptic, but I still can't see it listed.

Just give it a couple of hours. No pressure ;)

It will take a little while for this to become available, and it will probably require manual handholding on the buildds. All the necessary people are awake and aware of the problem, and we'll ensure that a response is available as quickly as we can.

Thanks guys! We all love Ubuntu. :-)

I had an open terminal when this happen and just have done:

$ wget http://us.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6_2.7-5ubuntu2_i386.deb

and after:

$ su -c "dpkg -i libc6_2.7-5ubuntu2_i386.deb"

And it works fine now.. :-) scared update :(

People are concerned about 'sudo' and MALLOC_CHECK_ variable. Anybody could run anything exporting it with "=1", but suid and sgid binaries don't work with it.

From `info malloc`:

There is one problem with `MALLOC_CHECK_': in SUID or SGID binaries it could possibly be exploited since diverging from the normal programs behavior it now writes something to the standard error descriptor. Therefore the use of MALLOC_CHECK_' is disabled by default for SUID and SGID binaries. It can be enabled again by the system administrator by adding a file `/etc/suid-debug' (the content is not important it could be empty).

It means I could 'touch /etc/suid-debug'. FYI: this bug caught me too; I'm an idiot, because my root has no password. lol

Lillipuziano, Kubuntu mounts volumes automatically. One click in Dolphin, D3lphin or (either)Konqi on the disk in the Storage Media area.
Appearing on the desktop without an event (insertion of some media) doesn't work yet (at least in KDE3, I haven't spent much time with KDE4 yet).

Lillipuziano,
I could not get your solution to work.

----
- root@ubuntu:/home/ubuntu# chroot /myroot dpkg -x libc6_2.7-5ubuntu2_i386.deb
- dpkg-deb: --extract needs a target directory.
- Perhaps you should be using dpkg --install ?
----

Fixed packages are now available on archive.ubuntu.com.

Me too....

I reinstall the system, and waiting for stable update

Fred, it's:

dpkg -X package /target/directory/

(notice the capital X)

Okay, got my system fixed now.
Using the LiveCD and;
cp --parents `dpkg -L libc6-i686 | xargs` /path/to/mounted/disk/
cp --parents `dpkg -L libc6 | xargs` /path/to/mounted/disk/

--

Shouldn't important and system-critical updates like this have been put into 'hardy-proposed' (Pre-release updates) for a few days before it got put into 'hardy-updates' (Recommended updates) ?

Hi

On Thu, Mar 13, 2008 at 02:02:54PM -0000, Fred wrote:
> Lillipuziano,
> I could not get your solution to work.
>
> ----
> - root@ubuntu:/home/ubuntu# chroot /myroot dpkg -x libc6_2.7-5ubuntu2_i386.deb
> - dpkg-deb: --extract needs a target directory.
> - Perhaps you should be using dpkg --install ?

Yes, -i worked for me.

Jan

--
   .~.
   /V\ Jan Groenewald
  /( )\ www.aims.ac.za
  ^^-^^

We don't do that for development releases, no; hardy-updates only becomes useful after release. As is prominently mentioned in a number of places, you should not be using pre-release versions of Ubuntu unless you are comfortable dealing with occasional (even severe) breakage.

OK, stuff happens. I recovered my laptop and will recover my server that is being configured for production. While I did not expect bugs that severe, I was warned and recovery was not that difficult. No biggie.

Here's three different conclusions:

1) R o o t P a s s w o r d

What is apparent to me, is that many system recovery features do require having a valid unix root password.

I think that it is highly precarious to leave users unable to "just screw that fancy stuff and logon as root" as they would be unable to perform various system recovery functions if that is so. (example, /home directory lost)

After any Ubuntu install, my FIRST command is always

  sudo passwd root

I think that a suitable installation action would be to "set root password to the first user password". (my root password is not the same)

2) M a i l i n g L i s t

I would personally like to subscribe to a "hardy-alerts" mailing list and get notified of such things as they occur. If such a list exists, I would like to subscribe to it, if not, I would like to see it. Not for minor bugs, but for showstopper bugs.

3) R e g r e s s i o n T e s t i n g

I write regression tests for anything that I program at work. I think that in the case of Ubuntu, it is possible to come up with an automated test that involves, at the very least, a successful multiuser boot, user able to login, ssh working, and the "system update drivetrain" working as well. A minimum system config that does not leave its users stranded. Such a test would be even more useful for production stage. If anyone needs help here, I could dedicate some time to it.

Guys,

I guess I did a miracle. lol

WARNING: if you want try this procedure, read with attention the whole thing before try. It shouldn't harm your system, but need to be careful to avoid screw it up and need redo everything again. This nano-howto is newbie-friendly and released under GPLv2. lmfao

Let me explain exactly what I did... As you can see above, I told my root doesn't have password and suid/sgid binaries and doesn't work with MALLOC_CHECK_="1". But I'm running Xubuntu on a amd64 box, which means I need some stuff functional on 64 and 32 bits (thanks Adobe for beeing lazy with flash plugin). Ehr, so I have ia32libs and also dchroot here. You *need* at least a xterm/console/whatever already running and a web browser would help a lot. Now the fancy workaround:

* run 'dhcroot -d' (the new environment doesn't matter, but I'm assuming you already used it once)

* run 'mkdir newmountpoint' (or any name, just need to be unique)

* run 'sudo mount /dev/sda1 newmountpoint' (check before what's the right partition)

* now 'cd newmountpoint/etc' (check if the content is the same at regular /etc with no chroot)

* create an empty file with 'sudo touch suid-debug' (and check if this file is under /etc, same as above)

* go back with 'cd' and run 'umount /dev/sda1' (might check the right partition above)

* close the chroot running 'exit' (JUST ONCE, BE CAREFUL)

Now you are again on buggy console. Download the new libc6_2.7-9ubuntu2_amd64.deb from main repository, as explained by Marco Rodrigues (using wget) or some browser. If your system is 32 bits download the right i386 package. Now you'll do the easy part:

* run 'export MALLOC_CHECK_=1'

* and run 'sudo dpkg -i libc6_2.7-9ubuntu2_amd64.deb'

Whew! If you did all steps right, you'll see something like below (it's Portuguese, i forgot to use LANG=C to get English text, anyway you dont need it too):

malloc: using debugging hooks
*** glibc detected *** sudo: free(): invalid pointer: 0x00007f88ac9e66c8 ***
*** glibc detected *** sudo: free(): invalid pointer: 0x00007f88ac9e6710 ***
(Lendo banco de dados ... 169973 arquivos e diretórios atualmente instalados.)
Preparando para substituir libc6 2.7-9ubuntu1 (usando libc6_2.7-9ubuntu2_amd64.deb) ...
Descompactando substituto libc6 ...
Instalando libc6 (2.7-9ubuntu2) ...

Hope that helps. (:

Thanks Bremm for your hint!!

Sudo and su -c weren't working for me until I did "export MALLOC_CHECK_=1"
After that, I have downgraded the package and everything is working again now :D

O maaaaan, you just saved me from rebooting... that is sweet as!!! (I didn't know about '-c' and it doesn't work without it!!)
:)
thanks!!!!!

>Marco Rodrigues wrote 8 hours ago: (permalink)
>
>I had an open terminal when this happen and just have done:
>
>$ wget http://us.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6_2.7-5ubuntu2_i386.deb
>
>and after:
>
>$ su -c "dpkg -i libc6_2.7-5ubuntu2_i386.deb"
>
>And it works fine now.. :-) scared update :(

following the procedure without a liveCD in the description I get an error after the 'mount -o remount,rw /root'

mount: Can't find /root in /proc/mounts

do i have any other options besides using a live cd?

Just to check your mirrors:
run 'sudo apt-get update'
run 'sudo apt-get -s upgrade | grep libc6'

If you get the following (note the 2.7-9ubuntu2) you should be good to go

  libavcodec1d libavutil1d libc6 libcairo2 libcupsimage2 libcupsys2 libdb4.6
Inst libc6 [2.7-5ubuntu2] (2.7-9ubuntu2 Ubuntu:8.04/hardy)
Conf libc6 (2.7-9ubuntu2 Ubuntu:8.04/hardy)

If you have 2.7-9ubuntu1 do not upgrade just yet :)

I've contacted Elias by private e-mail to try to diagnose his problem. If it turns out to be general I'll update the instructions here.

I just reinstalled my hardy laptop because of this.
I tried a lot of stuff without luck.
Since I had no Internet access to check this page, which could had saved me the reinstalation.

Tough luck...

Not 100% sure this is related. I'm running Hardy Alpha 64-bit, not because I am experienced, but because it has the drivers necessary for my wi-fi that Gutsy lacked.

Very successful until recent upgrade which failed to complete. The system is still booting and working fine but it impossible to do any installs/updates/upgrades.

Adept Updater simply reports
There was an error commiting changes. Possibly there was a problem downloading some packages or the commit would break packages.

Synaptic finds two broken packages: libc6-dev and libc6-i386, claims to successfully fix the dependency problems, but clearly doesn't!

libc6 is marked for upgrade from 2.7-5ubuntu2 to 2.7-9ubuntu2 but the upgrade fails and cannot be "unmarked".

Report from applying the upgrade is:
E: /var/cache/apt/archives/libc6_2.7-9ubuntu2_amd64.deb: subprocess pre-installation script returned error exit status 1

sudo apt-get -f upgrade gives (install gives similar):

Preparing to replace libc6 2.7-5ubuntu2 (using .../libc6_2.7-9ubuntu2_amd64.deb) ...
debconf: DbDriver "config": could not open /var/cache/debconf/config.dat
dpkg: error processing /var/cache/apt/archives/libc6_2.7-9ubuntu2_amd64.deb (--unpack):
 subprocess pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.7-9ubuntu2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Any help appreciated, I do realise that Alpha versions carry risks, so will live with it if I have to.

Peter: this is not directly related to this bug. It looks like the directory /var/cache/debconf is entirely missing, perhaps due to some filesystem corruption. Feel free to contact me by private mail (<email address hidden>) and I'll try to help you out more.

Colin:

Why dont you run updates for important base packages through a virtualbox installation of Ubuntu first. If it breaks you can just go back to save point X and fix it. Nobody will ever know ;)

The process could even be automated.
Like:
Only send the update if the vbox installtion reboots and shows X11 (gdm, kdm .. ) || login ( for a server ).

I think people do test/run LTS releases even in development. And this is not what i would call a confidence building exercise ;)
And we exspect bugs and breakage .. but not easily avoidable severe breakage :-P

But what do i know .. i just wanted to give my .02€

We are investigating what happened here (in terms of processes) and will make testing improvements as a result. I don't want to go into details prematurely.

Of course, resource-hungry testing measures are always a trade-off between development efficiency and safety. Ubuntu has derived great benefit in the past from being able to push uploads through quickly (so that many iterations can be run over the course of a working day), and I'm cautious about imposing mandatory delays that would slow that process down too badly. There are clearly much more lightweight measures that would achieve significant safety improvements, and we're likely to look at those first.

We will probably also ramp up our warnings about the development release not being suitable for production use. The object of the development release, whether leading up to an LTS or not, is not to build confidence; it is to produce a high-quality end result - and I do believe that the root cause of this change (LDFLAGS=-Wl,-Bsymbolic-functions) will provide significant benefits elsewhere. Yes, it is a shame that it blew up here, but on the bright side this is far enough before release that we have time for intensive testing to ensure that nothing else important is going to blow up in the same way.

It's very common for the reaction to failures to be to impose a much greater testing load on developers and/or development processes. You can increase testing to an unbounded extent, though, and clearly you hit diminishing returns at some point; even at that point, in our imperfect world, you'll still get failures. Thus we need to consider trade-offs and efficiency rather than simply saying "more testing please"; rest assured that that consideration will happen.

I do not want to discuss testing further on this bug report. Please take further discussion to development mailing lists.

It's a development build. Don't like breakage, then don't use it, it's as simple as that. People using Hardy are ( or at least should be! ) aware that such things can occur, so they should deal with it accordingly, period.

Hi,

I had a problem after upgrading from 7.10 to 8.04. Got the message: "Waiting for root file system". Below my way to solve/work around this issue.

I did boot the system with an older kernel, you can choose from GRUB.
After logging in I installed "live-initramfs" using the package manager.
It seems that it has reconfigured initramfs. Rebooting again and the
systems boots well.

Jan-Marc: live-initramfs is not appropriate, though it may have happened to fix this by accident. A more correct fix is just to run 'update-initramfs -u' as root, as my (updated) instructions in the description of this bug indicate.

"first reported on 2008-03-13'

Must have something to do with the 13th... Just did today's upgrades (April 13th) and the linux-restricted 386 kernel drops to initramfs busybox shell. Booting into -generic works fine.

Oddly enough, installing live-initramfs in the working (generic) kernel did not resolve the problem. But then removing (sudo apt-get remove live-initramfs) did, as the removal regenerated new boot initrd.img's for all the installed kernels.

$ sudo apt-get remove live-initramfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  user-setup busybox
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  live-initramfs
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 373kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 150602 files and directories currently installed.)
Removing live-initramfs ...
update-initramfs: Generating /boot/initrd.img-2.6.24-16-generic
update-initramfs: Generating /boot/initrd.img-2.6.24-16-386
update-initramfs: Generating /boot/initrd.img-2.6.24-15-generic
update-initramfs: Generating /boot/initrd.img-2.6.24-15-386
update-initramfs: Generating /boot/initrd.img-2.6.24-14-generic
update-initramfs: Generating /boot/initrd.img-2.6.24-14-386