libdbus-1-3 upgrade does not respawn init, resulting in unclean shutdown

Ubuntu on EC2 project is not the right place to file these any more. Instead, please use 'ubuntu-bug <package>' on an instance, or tag with 'ec2-images' and 'uec-images'

I'm unable to reproduce this bug.

The ami given (ami-30669459 ubuntu-images-testing/ubuntu-natty-daily-amd64-server-20110321) is no longer public.
I've tried running the latest daily :
  us-east-1 ami-d8e21eb1 canonical ubuntu-natty-daily-i386-server-20110418

then doing:
 $ sudo apt-get update
 $ sudo apt-get install --reinstall libdbus-1-3
 $ sudo shutdown -h now

I can see the '/ is busy' message in the console out, but the instance *does* go to terminated.

I suspect that the fallout of this is that the filesystem is not unmounted and thus not clean, right?

>I suspect that the fallout of this is that the filesystem is not unmounted and thus not clean, right?

Correct. This results in orphaned inodes on reboot. Touching
/var/run/init.upggraded is a possible solution. This will cause init
to respawn on shutdown.

You have reproduced the bug. The problem isn't that the system doesn't
shut down. The problem is an unclean filesystem.

Thanks for starting to take a look.

On 4/18/11, Scott Moser <email address hidden> wrote:
> I'm unable to reproduce this bug.
>
> The ami given (ami-30669459
> ubuntu-images-testing/ubuntu-natty-daily-amd64-server-20110321) is no longer
> public.
> I've tried running the latest daily :
> us-east-1 ami-d8e21eb1 canonical ubuntu-natty-daily-i386-server-20110418
>
> then doing:
> $ sudo apt-get update
> $ sudo apt-get install --reinstall libdbus-1-3
> $ sudo shutdown -h now
>
> I can see the '/ is busy' message in the console out, but the instance
> *does* go to terminated.
>
> I suspect that the fallout of this is that the filesystem is not
> unmounted and thus not clean, right?
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/740390
>
> Title:
> libdbus-1-3 upgrade does not respawn init, resulting in unclean
> shutdown
>
> Status in Official Ubuntu Images on EC2:
> Invalid
> Status in “dbus” package in Ubuntu:
> New
>
> Bug description:
> Binary package hint: dbus
>
> Steps to reproduce:
> 1. Start 20110321 daily build on Amazon EC2.
> ami-30669459
> ubuntu-images-testing/ubuntu-natty-daily-amd64-server-20110321
> 2. SSH.
> 3. sudo apt-get update
> 4. To simulate an upgrade, sudo apt-get --reinstall install libdbus-1-3
> libdbus-1-3_1.4.6-1ubuntu6 is reinstalled.
> 5. sudo lsof yields (snipped)
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
> NAME
> init 1 root DEL REG 202,1 134262
> /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
> upstart-u 201 root DEL REG 202,1 134262
> /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
> upstart-s 369 root DEL REG 202,1 134262
> /lib/x86_64-linux-gnu/libdbus-1.so.3.5.4
> 6. sudo shutdown -h now
> Error:
> mount: / is busy
>
> Possible solution: touch /var/run/init.upgraded in package postinst to
> force init to respawn/close open files at shutdown. This is checked in
> /etc/rc[06].d/S60umountroot. See bug 672177 for a similar situation
> with libc6.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu-on-ec2/+bug/740390/+subscribe
>

I just tested this on a clean natty VM with all updates applied, and I do indeed see an orphaned inode left over on /

We really need to trigger /var/run/init.upgraded on any of upstart's dependent libraries. Rather than making all the libraries do this, a trigger on /lib seems to make more sense where it can do an 'ldd' on /sbin/init and if the libraries changed fall into it, touch /var/run/init.upgraded. So, I'll open a task against upstart to do that. In the interim, we need to have libdbus-1-3 touch /var/run/init.upgraded as well.

Well, hindsight being 20-20, I should have seen this as more serious. Bug 1058517 is now open, it is really a dupe of this. we should collapse them one way or another.

Clint: I don't think there's any point in doing a complex trigger arrangement when (a) only a tiny number of libraries are involved, and (b) stateful-reexec will solve this anyway.

FWIW, libnih1/libnih-dbus1 have this problem too.

Added tasks for libnih.

This bug was fixed in the package dbus - 1.6.4-1ubuntu3

---------------
dbus (1.6.4-1ubuntu3) quantal-proposed; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * Fix unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:14:40 -0400

If stateful re-exec does indeed handle this then agreed, no need for any triggers.

This bug was fixed in the package dbus - 1.4.18-1ubuntu1.3

---------------
dbus (1.4.18-1ubuntu1.3) precise-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 06:12:39 -0400

This bug was fixed in the package dbus - 1.2.16-2ubuntu4.7

---------------
dbus (1.2.16-2ubuntu4.7) lucid-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:05:52 -0400

This bug was fixed in the package dbus - 1.4.6-1ubuntu6.4

---------------
dbus (1.4.6-1ubuntu6.4) natty-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:03:55 -0400

This bug was fixed in the package dbus - 1.4.14-1ubuntu1.3

---------------
dbus (1.4.14-1ubuntu1.3) oneiric-security; urgency=low

  * REGRESSION FIX: some applications launched with the activation helper
    may need DBUS_STARTER_ADDRESS. (LP: #1058343)
    - debian/patches/CVE-2012-3524-regression-fix.patch: hardcode the
      starter address to the default system bus address.
  * REGRESSION FIX: unclean shutdown after dbus upgrade (LP: #740390)
    - debian/libdbus-1-3.postinst: trigger an upstart re-exec before
      shutdown or reboot so that it can safely unmount the root
      filesystem.
 -- Marc Deslauriers <email address hidden> Wed, 03 Oct 2012 07:02:41 -0400

The dbus upgrade has been applied, and this basic unique desktop still show on shutdown: mount: / is busy (each time it is shutdown). This is on QQ i386 with pata/sata hdds and bios.

This bug was fixed in the package libnih - 1.0.3-4ubuntu11

---------------
libnih (1.0.3-4ubuntu11) quantal; urgency=low

  * Addition of debian/libnih-dbus1.postinst and
    debian/libnih1.postinst to force Upstart re-exec on shutdown
    to avoid unmounting disks uncleanly (LP: #740390).
 -- James Hunt <email address hidden> Wed, 03 Oct 2012 16:49:40 +0100

@dino99: Thanks for spotting this - I too saw this in testing. I've isolated the problem - see bug 1061639.

Thanks James, confirmation done.

I've created bug 1062202 to handle the lbnih SRU for precise.

Still get uncleaned shutdown process:

with the last shutdown, i've seen:
- modemmanager reloading its dozen drivers
- mount / is busy
- and last with a [drm] nouveau complaint, switching to fbcon (or so)

with the next reboot:

oem@dub:~$ dmesg | grep orphan
[ 8.812542] EXT4-fs (sdb5): orphan cleanup on readonly fs
[ 8.819511] EXT4-fs (sdb5): ext4_orphan_cleanup: deleting unreferenced inode 400809
[ 8.819581] EXT4-fs (sdb5): 1 orphan inode deleted

Hello Alan, or anyone else affected,

Accepted libnih into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libnih/1.0.3-4ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The fix for this bug has been awaiting testing feedback in the -proposed repository for precise for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

I verified this following James' and Marc's instructions in bug 1062202.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package libnih - 1.0.3-4ubuntu9.1

---------------
libnih (1.0.3-4ubuntu9.1) precise-proposed; urgency=low

  * Addition of debian/libnih-dbus1.postinst and
    debian/libnih1.postinst to force Upstart re-exec on shutdown
    to avoid unmounting disks uncleanly (LP: #740390, LP: #1062202).
 -- James Hunt <email address hidden> Fri, 05 Oct 2012 14:14:10 +0100