[Xenial/0.90] Systemd dependencies issues when used in "Shutdown mode"

Bug #1618900 reported by Guillaume Penin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
unattended-upgrades (Ubuntu)
Fix Released
Medium
Brian Murray
Xenial
Fix Released
Medium
Brian Murray
Yakkety
Fix Released
Medium
Brian Murray

Bug Description

[Impact]
There are some race conditions in unattended-upgrades when it is used in shutdown mode. It is possible that the network has already been brought down or for a local file system, like /var/, to be unmounted before the unattended-upgrades script is run. This then causes the updates not to be installed thereby making systems less secure - if security updates are enabled.

[Test Case]
Because this is a race condition, I haven't actually been able to recreate the bug. However, the fix is straightforward and testing for regressions should be sufficient.

1) On a freshly installed system.
2) Modify /etc/apt/apt.conf.d/50unattended-upgrades so that
Unattended-Upgrade::InstallOnShutdown "true"; is not commented out.
3) Reboot and watch updates being installed
4) Verify updates were installed by reviewing /var/log/unattended-upgrades/unattended-upgrades.log
5) Uncomment -updates in 50unattended-upgrades

Repeats steps 3 and 4 again. There isn't any need to run unattended-upgrades in *not* shutdown mode because we have only modified the systemd service file.

[Regression Potential]
There is little chance of a regression, but because of the change (and this feature now working better) its possible the shutdown process will take longer and surprise users.

Original Description
--------------------

Using unattended-upgrades 0.90 in "Shutdown mode" on Ubuntu Xenial, we encounter the following systemd dependencies issues :
- The network is often down when unattended-upgrades is running, so packages can not be downloaded (can be mitigated by using APT::Periodic::Download-Upgradeable-Packages "1";) :
    => ERROR An error occurred: 'Could not resolve host: ****.fr'
    => ERROR The URI 'https://****.fr:33000/ubuntu-security/pool/main/libi/libidn/libidn11_1.32-3ubuntu1.1_amd64.deb' failed to download, aborting
- Important mountpoints like /boot are unmounted before unattended-upgrades is running, so newer kernels can not be installed properly (ramdisk and grub configuration can not be generated)

Mathew Hodson (mhodson)
tags: added: xenial
Revision history for this message
Brian Murray (brian-murray) wrote :

The service possibly needs a relationship with the network service.

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStartSec=900

[Install]
WantedBy=shutdown.target

Changed in unattended-upgrades (Ubuntu):
importance: Undecided → Medium
tags: added: yakkety
Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
Revision history for this message
Martin Pitt (pitti) wrote :

If a unit needs to get stopped before the network gets stopped, you shold add "After=network.target". (See man systemd.special).

Revision history for this message
Martin Pitt (pitti) wrote :

Oh, actually not -- that would be for a service which gets *stopped* during shutdown (i. e. the usual way). If you want to *start* on shutdown, you need to be Before=network.target.

Revision history for this message
Brian Murray (brian-murray) wrote :

Before=local-fs.target would be needed for the mountpoints like /boot.

tags: added: rls-z-incoming
Changed in unattended-upgrades (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Brian Murray (brian-murray) wrote :

I've created an upstream pull request with these fixes.

https://github.com/mvo5/unattended-upgrades/pull/38

Changed in unattended-upgrades (Ubuntu):
assignee: nobody → Brian Murray (brian-murray)
Changed in unattended-upgrades (Ubuntu):
milestone: none → ubuntu-17.02
Revision history for this message
Brian Murray (brian-murray) wrote :

This should be fixed in Zesty now:

unattended-upgrades (0.93.1) unstable; urgency=medium

  [ Brian Murray ]
  * Create logfile_dpkg if it doesn't exist
  * Make sure the statedir exists too
  * ensure network and local-fs are available in shutdown
...
-- Michael Vogt <email address hidden> Sun, 11 Dec 2016 11:31:26 +0100

Changed in unattended-upgrades (Ubuntu):
status: Triaged → Fix Released
Changed in unattended-upgrades (Ubuntu Xenial):
status: New → Triaged
Changed in unattended-upgrades (Ubuntu Yakkety):
status: New → Triaged
Changed in unattended-upgrades (Ubuntu Xenial):
importance: Undecided → Medium
Changed in unattended-upgrades (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in unattended-upgrades (Ubuntu Xenial):
milestone: none → ubuntu-16.04.2
Changed in unattended-upgrades (Ubuntu Xenial):
assignee: nobody → Brian Murray (brian-murray)
Changed in unattended-upgrades (Ubuntu Yakkety):
milestone: none → yakkety-updates
Revision history for this message
Brian Murray (brian-murray) wrote :

I want to write a test case for this SRU, where did you see the error messages that appear in the bug description? Were they in /var/log/unattended-upgrades/unattended-upgrades.log?

Revision history for this message
Brian Murray (brian-murray) wrote :

Sorry, I read the code and see that it is producing those messages.

unattended-upgrade: print(_("An error occurred: '%s'") % item.error_text)
unattended-upgrade: logging.error(_("An error occurred: '%s'"), item.error_text)
Binary file unattended-upgrade.pyc matches

description: updated
Changed in unattended-upgrades (Ubuntu Xenial):
status: Triaged → In Progress
Changed in unattended-upgrades (Ubuntu Yakkety):
status: Triaged → In Progress
assignee: nobody → Brian Murray (brian-murray)
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Guillaume, or anyone else affected,

Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.92ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Yakkety):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in unattended-upgrades (Ubuntu Xenial):
status: In Progress → Fix Committed
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Guillaume, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.90ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Brian Murray (brian-murray) wrote :

verification-done for yakkety:

2016-12-20 10:41:41,532 INFO Starting unattended upgrades script
2016-12-20 10:41:41,532 INFO Allowed origins are: ['o=Ubuntu,a=yakkety', 'o=Ubuntu,a=yakkety-security']
2016-12-20 10:43:35,014 INFO Packages that will be upgraded: apport apport-gtk apt apt-transport-https apt-utils bind9-host curl dbus dbus-user-session dbus-x11 dnsutils firefox firefox-locale-en ghostscript ghostscript-x gstreamer1.0-plugins-good gstreamer1.0-pulseaudio imagemagick imagemagick-6.q16 imagemagick-common libapt-inst2.0 libapt-pkg5.0 libbind9-140 libcurl3 libcurl3-gnutls libdbus-1-3 libdns-export162 libdns162 libgd3 libgs9 libgs9-common libgstreamer-plugins-good1.0-0 libisc-export160 libisc160 libisccc140 libisccfg140 liblwres141 libmagickcore-6.q16-2 libmagickcore-6.q16-2-extra libmagickwand-6.q16-2 liboxideqt-qmlplugin liboxideqtcore0 liboxideqtquick0 libsmbclient libwbclient0 linux-generic linux-headers-generic linux-image-generic linux-libc-dev oxideqt-codecs python3-apport python3-cryptography python3-problem-report samba-libs tar thunderbird thunderbird-gnome-support thunderbird-locale-en thunderbird-locale-en-us tzdata vim vim-common vim-runtime vim-tiny
2016-12-20 10:43:35,014 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2016-12-20 10:45:35,240 INFO All upgrades installed
2016-12-20 10:47:15,408 INFO Initial blacklisted packages:
2016-12-20 10:47:15,409 INFO Initial whitelisted packages:
2016-12-20 10:47:15,409 INFO Starting unattended upgrades script
2016-12-20 10:47:15,409 INFO Allowed origins are: ['o=Ubuntu,a=yakkety', 'o=Ubuntu,a=yakkety-security', 'o=Ubuntu,a=yakkety-updates']
2016-12-20 10:48:57,310 INFO Packages that will be upgraded: cups-browsed cups-filters cups-filters-core-drivers deja-dup file-roller gnome-software gnome-software-common ifupdown im-config libc-bin libc-dev-bin libc6 libc6-dbg libc6-dev libcupsfilters1 libfcitx-config4 libfcitx-gclient0 libfcitx-utils0 libfontembed1 libgail-common libgail18 libgspell-1-1 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libmetacity0 libnautilus-extension1a libnss-resolve libpam-systemd libpulse-mainloop-glib0 libpulse0 libpulsedsp libsystemd0 libudev1 light-themes locales metacity metacity-common multiarch-support nautilus nautilus-data pulseaudio pulseaudio-module-bluetooth pulseaudio-utils python3-distupgrade qml-module-ubuntu-web qtdeclarative5-ubuntu-web-plugin snapd suru-icon-theme systemd systemd-sysv ubuntu-artwork ubuntu-mobile-icons ubuntu-mono ubuntu-release-upgrader-core ubuntu-release-upgrader-gtk ubuntu-software udev webapp-container webbrowser-app xmir xserver-common xserver-xorg-core
2016-12-20 10:48:57,310 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2016-12-20 10:50:02,211 INFO All upgrades installed

bdmurray@clean-yakkety-amd64:~$ apt-cache policy unattended-upgrades
unattended-upgrades:
  Installed: 0.92ubuntu1.2
  Candidate: 0.92ubuntu1.2
  Version table:
 *** 0.92ubuntu1.2 100
        100 /var/lib/dpkg/status

tags: added: verification-done-yakkety
Revision history for this message
Brian Murray (brian-murray) wrote :
Download full text (4.2 KiB)

2016-12-20 11:01:59,325 INFO Starting unattended upgrades script
2016-12-20 11:01:59,326 INFO Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security']
2016-12-20 11:04:43,551 INFO Packages that will be upgraded: apport apport-gtk apt apt-transport-https apt-utils bind9-host cpp-5 curl dbus dbus-x11 dnsutils firefox firefox-locale-en g++-5 gcc-5 gcc-5-base ghostscript ghostscript-x gir1.2-gst-plugins-base-1.0 gstreamer1.0-alsa gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good gstreamer1.0-pulseaudio gstreamer1.0-x imagemagick imagemagick-6.q16 imagemagick-common libapt-inst2.0 libapt-pkg5.0 libasan2 libatomic1 libbind9-140 libcc1-0 libcilkrts5 libcurl3 libcurl3-gnutls libdbus-1-3 libdns-export162 libdns162 libgcc-5-dev libgd3 libgomp1 libgs9 libgs9-common libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 libisc-export160 libisc160 libisccc140 libisccfg140 libitm1 liblsan0 liblwres141 libmagickcore-6.q16-2 libmagickcore-6.q16-2-extra libmagickwand-6.q16-2 libmpx0 liboxideqt-qmlplugin liboxideqtcore0 liboxideqtquick0 libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3.5 libpython3.5-minimal libpython3.5-stdlib libquadmath0 libsmbclient libstdc++-5-dev libstdc++6 libtracker-sparql-1.0-0 libtsan0 libubsan0 libwbclient0 linux-firmware linux-generic linux-headers-generic linux-image-generic linux-libc-dev oxideqt-codecs python2.7 python2.7-minimal python3-apport python3-cryptography python3-problem-report python3.5 python3.5-minimal samba-libs tar thunderbird thunderbird-gnome-support thunderbird-locale-en thunderbird-locale-en-us tzdata vim vim-common vim-runtime vim-tiny
2016-12-20 11:04:43,552 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2016-12-20 11:06:59,150 INFO All upgrades installed
2016-12-20 11:27:12,868 INFO Starting unattended upgrades script
2016-12-20 11:27:12,868 INFO Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security', 'o=Ubuntu,a=xenial-updates']
2016-12-20 11:29:45,097 INFO Packages that will be upgraded: accountsservice apparmor base-files bind9-host deja-dup distro-info-data dnsutils gir1.2-dbusmenu-glib-0.4 gnome-calculator gnome-software gnome-software-common ifupdown im-config init init-system-helpers initramfs-tools initramfs-tools-bin initramfs-tools-core isc-dhcp-client isc-dhcp-common kbd language-pack-en language-pack-gnome-en libaccountsservice0 libapparmor-perl libapparmor1 libappstream-glib8 libbind9-140 libc-bin libc-dev-bin libc6 libc6-dbg libc6-dev libdbusmenu-glib4 libdbusmenu-gtk3-4 libdbusmenu-gtk4 libdns-export162 libdns162 libfcitx-config4 libfcitx-gclient0 libfcitx-utils0 libisc-export160 libisc160 libisccc140 libisccfg140 liblightdm-gobject-1-0 liblwres141 libmetacity-private3a libnm-glib-vpn1 libnm-glib4 libnm-util2 libnm0 libpam-systemd libprocps4 libpulse-mainloop-glib0 libpulse0 libpulsedsp libqt5core5a libqt5dbus5 libqt5gui5 libqt5network5 libqt5opengl5 libqt5printsupport5 libqt5sql5 libqt5sql5-sqlite libqt5test5 libqt5widgets5 libqt5xml5 libsystemd0 libudev1 libwhoopsie0 light-themes lightdm linux-firmware locales metacity metacity-common multiarch-support network-manage...

Read more...

tags: added: verification-done-xenial
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 0.90ubuntu0.3

---------------
unattended-upgrades (0.90ubuntu0.3) xenial-proposed; urgency=medium

  * Ensure that network and local filesystems are available when running
    unattended-upgrades during shutdown mode. (LP: #1618900)

 -- Brian Murray <email address hidden> Wed, 14 Dec 2016 13:38:32 -0800

Changed in unattended-upgrades (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 0.92ubuntu1.2

---------------
unattended-upgrades (0.92ubuntu1.2) yakkety; urgency=medium

  * Ensure that network and local filesystems are available when running
    unattended-upgrades during shutdown mode. (LP: #1618900)

 -- Brian Murray <email address hidden> Wed, 14 Dec 2016 13:31:31 -0800

Changed in unattended-upgrades (Ubuntu Yakkety):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.