Fixing fsfreeze-hook can break unattended upgrades

Fix might be in qemu (add even more special cases) or in unattended upgrades (to properly handle or at least not die). Added a bug task for that.

Status changed to 'Confirmed' because the bug affects multiple users.

I already found this in the dup-ed bug and will fix it in u-u.
The fix is almost done, I'm adding a few extra tests.

I made this the master bug because it has nicer description, thanks! :-)

I'm subscribed here in case there is anything I'm needed, but the qemu task is actually invalid as the change won't be there.
Further I assigned it to rbalint to reflect that he is working on this in u-u.

This bug was fixed in the package unattended-upgrades - 1.10ubuntu3

---------------
unattended-upgrades (1.10ubuntu3) disco; urgency=medium

  * Detect changes to moved conffiles (LP: #1823872)
  * Add tests for checking conffile moves.
    Build depend on and use equivs to generate new test packages

 -- Balint Reczey <email address hidden> Thu, 11 Apr 2019 22:48:00 +0200

Thanks Rbalint for fixing that.
Since a triggering change in qemu is in at least >=Bionic and was reported to even affect continuous unattended upgrades there would you considering SRUing your changes as needed?
I added tasks for all releases, but set >=Bionic to high to to that being an issue in the field IMHO.

Yes, I'm SRUing that, but I also extend the fix first with https://github.com/mvo5/unattended-upgrades/pull/187 to not hold qemu needlessly back.

Hello Christian, or anyone else affected,

Accepted unattended-upgrades into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.10ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

An upload of unattended-upgrades to cosmic-proposed has been rejected from the upload queue for the following reason: "Some bugs are missing valuable SRU information and changelog links to an unrelated, old bug #182480. Please fix an re-upload.".

Hello Christian, or anyone else affected,

Accepted unattended-upgrades into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.5ubuntu3.18.10.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Christian, or anyone else affected,

Accepted unattended-upgrades into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Christian, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I only have a testcase for Bionic, I hope rbalint has one for the other releases.

root@b:~# apt list --upgradable
Listing... Done
qemu-guest-agent/bionic-updates,bionic-security 1:2.11+dfsg-1ubuntu7.12 amd64 [upgradable from: 1:2.11+dfsg-1ubuntu7]
N: There is 1 additional version. Please use the '-a' switch to see it
root@b:~#
root@b:~#
root@b:~# sudo unattended-upgrade -d
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
Using (^linux-image-[0-9]+\.[0-9\.]+-.*|^linux-headers-[0-9]+\.[0-9\.]+-.*|^linux-image-extra-[0-9]+\.[0-9\.]+-.*|^linux-modules-[0-9]+\.[0-9\.]+-.*|^linux-modules-extra-[0-9]+\.[0-9\.]+-.*|^linux-signed-image-[0-9]+\.[0-9\.]+-.*|^kfreebsd-image-[0-9]+\.[0-9\.]+-.*|^kfreebsd-headers-[0-9]+\.[0-9\.]+-.*|^gnumach-image-[0-9]+\.[0-9\.]+-.*|^.*-modules-[0-9]+\.[0-9\.]+-.*|^.*-kernel-[0-9]+\.[0-9\.]+-.*|^linux-backports-modules-.*-[0-9]+\.[0-9\.]+-.*|^linux-modules-.*-[0-9]+\.[0-9\.]+-.*|^linux-tools-[0-9]+\.[0-9\.]+-.*|^linux-cloud-tools-[0-9]+\.[0-9\.]+-.*) regexp to find kernel packages
Using (^linux-image-4\.15\.0\-48\-generic$|^linux-headers-4\.15\.0\-48\-generic$|^linux-image-extra-4\.15\.0\-48\-generic$|^linux-modules-4\.15\.0\-48\-generic$|^linux-modules-extra-4\.15\.0\-48\-generic$|^linux-signed-image-4\.15\.0\-48\-generic$|^kfreebsd-image-4\.15\.0\-48\-generic$|^kfreebsd-headers-4\.15\.0\-48\-generic$|^gnumach-image-4\.15\.0\-48\-generic$|^.*-modules-4\.15\.0\-48\-generic$|^.*-kernel-4\.15\.0\-48\-generic$|^linux-backports-modules-.*-4\.15\.0\-48\-generic$|^linux-modules-.*-4\.15\.0\-48\-generic$|^linux-tools-4\.15\.0\-48\-generic$|^linux-cloud-tools-4\.15\.0\-48\-generic$) regexp to find running kernel packages
Checking: qemu-guest-agent ([<Origin component:'universe' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>, <Origin component:'universe' archive:'bionic-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
pkgs that look like they should be upgraded: qemu-guest-agent
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 qemu-guest-agent amd64 1:2.11+dfsg-1ubuntu7.12 [232 kB]
Fetched 232 kB in 0s (0 B/s)
fetch.run() result: 0
<apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 0 IsTrusted: 1 FileSize: 232116 DestFile:'/var/cache/apt/archives/qemu-guest-agent_1%3a2.11+dfsg-1ubuntu7.12_amd64.deb' DescURI: 'http://archive.ubuntu.com/ubuntu/pool/universe/q/qemu/qemu-guest-agent_2.11+dfsg-1ubuntu7.12_amd64.deb' ID:1 ErrorText: ''>
check_conffile_prompt(/var/cache/apt/archives/qemu-guest-agent_1%3a2.11+dfsg-1ubuntu7.12_amd64.deb)
found pkg: qemu-guest-agent
conffile line: /etc/init.d/qemu-guest-agent f61a64ac1e48993023018fd1cff85191
current md5: f61a64ac1e48993023018fd1cff85191
conffile line: /etc/qemu/fsfreeze-hook/fsfreeze-hook 15f6ff42cbc5550a07ee21c2a471d...

@paelzer: From LP: #1820291 it looks like there is a ready test package for Cosmic and that could be used to test Disco as well. ;-)
I'll set one up for Xenial using equivs.

Status changed to 'Confirmed' because the bug affects multiple users.

I can indeed trigger it in cosmic with an upgrade from 1:2.12+dfsg-3ubuntu8 -> 1:2.12+dfsg-3ubuntu8.6

But forcing the same (by enabling cosmics sources) in Disco does not trigger the bug (1.10ubuntu5).
You will need an equiv trick to verify Disco as well I think.

Here he logs of the cosmic verify:

# trigger bug before upgrade
/etc/qemu/fsfreeze-hook
found conffile /etc/qemu/fsfreeze-hook in new pkg but on dpkg status
An error occurred: [Errno 21] Is a directory: '/etc/qemu/fsfreeze-hook'
Traceback (most recent call last):
  File "/usr/bin/unattended-upgrade", line 1669, in main
    install_start_time)
  File "/usr/bin/unattended-upgrade", line 1875, in run
    if conffile_prompt(item.destfile):
  File "/usr/bin/unattended-upgrade", line 1015, in conffile_prompt
    with open(prefix + conf_file, 'rb') as fp:
IsADirectoryError: [Errno 21] Is a directory: '/etc/qemu/fsfreeze-hook'
Extracting content from /var/log/unattended-upgrades/unattended-upgrades-dpkg.log since 2019-05-03 05:45:41
Traceback (most recent call last):
  File "/usr/bin/unattended-upgrade", line 2138, in <module>
    sys.exit(main(options))
  File "/usr/bin/unattended-upgrade", line 1669, in main
    install_start_time)
  File "/usr/bin/unattended-upgrade", line 1875, in run
    if conffile_prompt(item.destfile):
  File "/usr/bin/unattended-upgrade", line 1015, in conffile_prompt
    with open(prefix + conf_file, 'rb') as fp:
IsADirectoryError: [Errno 21] Is a directory: '/etc/qemu/fsfreeze-hook'

# upgrade to proposed
apt install unattended-upgrades
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  bsd-mailx default-mta | mail-transport-agent needrestart
The following packages will be upgraded:
  unattended-upgrades
1 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Need to get 41.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu cosmic-proposed/main amd64 unattended-upgrades all 1.5ubuntu3.18.10.4 [41.8 kB]
Fetched 41.8 kB in 0s (535 kB/s)
Preconfiguring packages ...
(Reading database ... 46512 files and directories currently installed.)
Preparing to unpack .../unattended-upgrades_1.5ubuntu3.18.10.4_all.deb ...
Unpacking unattended-upgrades (1.5ubuntu3.18.10.4) over (1.5ubuntu3.18.10.3) ...
Processing triggers for systemd (239-7ubuntu10.13) ...
Setting up unattended-upgrades (1.5ubuntu3.18.10.4) ...
Processing triggers for man-db (2.8.4-2) ...

# retry with fix
root@c:~# sudo unattended-upgrade -d
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
[...]
check_conffile_prompt(/var/cache/apt/archives/qemu-guest-agent_1%3a2.12+dfsg-3ubuntu8.6_amd64.deb)
found pkg: qemu-guest-agent
conffile line: /etc/init.d/qemu-guest-agent f61a64ac1e48993023018fd1cff85191
current md5: f61a64ac1e48993023018fd1cff85191
conffile line: /etc/qemu/fsfreeze-hook/fsfreeze-hook 15f6ff42cbc5550a07ee21c2a471d905
current md5: 15f6ff42cbc5550a07ee21c2a471d905
blacklist: []
whitelist: []
Packages that will be upgraded: qemu-guest-agent
Writing dpkg log to /var/log/unattended-upgrades/unattended-up...

With some more trickery I created this situation in Disco:

root@d:~# apt-cache policy qemu-guest-agent unattended-upgrades
qemu-guest-agent:
  Installed: 1:2.12+dfsg-3ubuntu8
  Candidate: 1:2.12+dfsg-3ubuntu8.6
unattended-upgrades:
  Installed: 1.10ubuntu5
  Candidate: 1.10ubuntu5

But in that case despite
root@d:~# apt list --upgradable
Listing... Done
qemu-guest-agent/disco 1:3.1+dfsg-2ubuntu3 amd64 [upgradable from: 1:2.12+dfsg-3ubuntu8]

But unattended upgrades does not like to upgrade it:
root@d:~# sudo unattended-upgrade -d
Running on the development release
Initial blacklist :
Initial whitelist:
Starting unattended upgrades script
[...]
No packages found that can be upgraded unattended and no pending auto-removals

Not sure what exactly you had in mind to test in Disco, but this seems not to be the path.
Can you set up your equivs setup there?

Verified 1.10ubuntu5.1 on Disco.

The crash was already fixed, but u-u did not recognize the moved conffile as the same:

root@dd-lp-1823872:~# dpkg -l qemu-guest-agent unattended-upgrades | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-===================-============-===========================================
ii qemu-guest-agent 1:3.1+dfsg-2ubuntu2 amd64 Guest-side qemu-system agent
ii unattended-upgrades 1.10ubuntu5 all automatic installation of security upgrades
root@dd-lp-1823872:~# unattended-upgrade --dry-run --verbose --debug
...
fetch.run() result: 0
<apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 1 IsTrusted: 1 FileSize: 202440 DestFile:'/var/cache/apt/archives/qemu-guest-agent_1%3a3.1+dfsg-2ubuntu3_amd64.deb' DescURI: 'http://archive.ubuntu.com/ubuntu/pool/universe/q/qemu/qemu-guest-agent_3.1+dfsg-2ubuntu3_amd64.deb' ID:0 ErrorText: ''>
check_conffile_prompt(/var/cache/apt/archives/qemu-guest-agent_1%3a3.1+dfsg-2ubuntu3_amd64.deb)
found pkg: qemu-guest-agent
conffile line: /etc/init.d/qemu-guest-agent f61a64ac1e48993023018fd1cff85191
current md5: f61a64ac1e48993023018fd1cff85191
conffile line: /etc/qemu/fsfreeze-hook/fsfreeze-hook 15f6ff42cbc5550a07ee21c2a471d905
/etc/qemu/fsfreeze-hook/fsfreeze-hook not in package conffiles /etc/init.d/qemu-guest-agent
/etc/qemu/fsfreeze-hook
found conffile /etc/qemu/fsfreeze-hook in new pkg but on dpkg status
blacklist: []
...

Fixed u-u recognize them being the same:
root@dd-lp-1823872:~# dpkg -l qemu-guest-agent unattended-upgrades | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-===================-============-===========================================
ii qemu-guest-agent 1:3.1+dfsg-2ubuntu2 amd64 Guest-side qemu-system agent
ii unattended-upgrades 1.10ubuntu5.1 all automatic installation of security upgrades
root@dd-lp-1823872:~# unattended-upgrade --dry-run --verbose --debug
...
fetch.run() result: 0
<apt_pkg.AcquireItem object:Status: 2 Complete: 1 Local: 1 IsTrusted: 1 FileSize: 202440 DestFile:'/var/cache/apt/archives/qemu-guest-agent_1%3a3.1+dfsg-2ubuntu3_amd64.deb' DescURI: 'http://archive.ubuntu.com/ubuntu/pool/universe/q/qemu/qemu-guest-agent_3.1+dfsg-2ubuntu3_amd64.deb' ID:0 ErrorText: ''>
check_conffile_prompt(/var/cache/apt/archives/qemu-guest-agent_1%3a3.1+dfsg-2ubuntu3_amd64.deb)
found pkg: qemu-guest-agent
conffile line: /etc/init.d/qemu-guest-agent f61a64ac1e48993023018fd1cff85191
current md5: f61a64ac1e48993023018fd1cff85191
conffile line: /etc/qemu/fsfreeze-hook/fsfreeze-hook 15f6ff42cbc5550a07ee21c2a471d905
current...

Verified 1.1ubuntu1.18.04.7~16.04.3 on Xenial:

root@x-lp-1823872:~# cat acpid.equivs
Source: acpid
Section: misc
Priority: optional
Standards-Version: 3.9.2

Package: acpid
File: /etc/acpi/events/powerbtn/powerbtn
 # /etc/acpi/events/powerbtn
 # This is called when the user presses the power button and calls
 # /etc/acpi/powerbtn.sh for further processing.
 .
 # Optionally you can specify the placeholder %e. It will pass
 # through the whole kernel event message to the program you've
 # specified.
 .
 # We need to react on "button power.*" and "button/power.*" because
 # of kernel changes.
 .
 event=button[ /]power
 action=/etc/acpi/powerbtn.sh
root@x-lp-1823872:~# equivs-build acpid.equivs
...
root@x-lp-1823872:~# dpkg -i acpid_1.0_all.deb
dpkg: warning: downgrading acpid from 1:2.0.26-1ubuntu2 to 1.0
(Reading database ... 32148 files and directories currently installed.)
Preparing to unpack acpid_1.0_all.deb ...
Unpacking acpid (1.0) over (1:2.0.26-1ubuntu2) ...
Setting up acpid (1.0) ...
Processing triggers for man-db (2.7.5-1) ...
root@x-lp-1823872:~# dpkg -l unattended-upgrades | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-==========================-============-===========================================
ii unattended-upgrades 1.1ubuntu1.18.04.7~16.04.2 all automatic installation of security upgrades
root@x-lp-1823872:~# unattended-upgrade --dry-run
Traceback (most recent call last):
  File "/usr/bin/unattended-upgrade", line 1998, in <module>
    sys.exit(main(options))
  File "/usr/bin/unattended-upgrade", line 1714, in main
    if conffile_prompt(item.destfile):
  File "/usr/bin/unattended-upgrade", line 929, in conffile_prompt
    with open(prefix + conf_file, 'rb') as fp:
IsADirectoryError: [Errno 21] Is a directory: '/etc/acpi/events/powerbtn'
root@x-lp-1823872:~# apt install -y -qq unattended-upgrades
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
Suggested packages:
  bsd-mailx default-mta | mail-transport-agent needrestart
The following packages will be upgraded:
  unattended-upgrades
1 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
Need to get 41.4 kB of archives.
After this operation, 8,192 B of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 32136 files and directories currently installed.)
Preparing to unpack .../unattended-upgrades_1.1ubuntu1.18.04.7~16.04.3_all.deb ...
Unpacking unattended-upgrades (1.1ubuntu1.18.04.7~16.04.3) over (1.1ubuntu1.18.04.7~16.04.2) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
Processing triggers for systemd (229-4ubuntu21.21) ...
Setting up unattended-upgrades (1.1ubuntu1.18.04.7~16.04.3) ...
root@x-lp-1823872:~# unattended-upgrade --dry-run --verbose
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed...

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package unattended-upgrades - 1.5ubuntu3.18.10.4

---------------
unattended-upgrades (1.5ubuntu3.18.10.4) cosmic; urgency=medium

  * Detect changes to moved conffiles (LP: #1823872)
  * Add tests for checking conffile moves.
    Build depend on and use equivs to generate new test packages
  * Detect unchanged moved conffiles.
    When a package moves a conffile properly without any change no conffile
    prompt needs to be shown thus the package can be upgraded unattended.
    (LP: #1823872)
    - Split() conffile data to set of names only once
    - Don't parse dpkg conffile db when there are no conffiles in the package
  * Skip sending email when no package had to be installed, upgraded or removed
    (LP: #1821103) (Closes: #924554)
  * Make sure autoremovals don't start with a dirty cache and remove other
    packages (LP: #1824341)
  * Continue applying minimal sets when one set can't be marked for upgrade.
    Thanks to Anderson Luiz Alves for the patch, it needed minor modifications
    (LP: #1824341)
  * Stop raising NoAllowedOriginError when marking packages to upgrade/install
    fails (LP: #1824876)
  * Adjust only transitive dependencies in the fallback when a package from an
    allowed origin can't be marked to install/upgrade.
    This is a much lighter approach than marking every upgradable package
    because the full fallback was triggered on packages held back as well,
    using an excessive amount of CPU time.
    Also it crashed with packages not having any version in allowed origins.
    (LP: #1824804, #1824949)
    - Follow all kinds of transitive dependencies when adjusting dependencies
    - Don't crash collecting transitive dependencies when package has no
      candidate (LP: #1825886)
  * Skip trying to upgrade held packages in call_adjusted() (LP: #1824804)
  * Use mark_install_adjusted() in rewind_cache()
    The original cache had packages marked with adjustments thus rewinding
    should also do adjustments to reach the same state.
    Also not using mark_install_adjusted() crashes when apt raises error on
    held packages. (LP: #1826157)
    - test_rewind: Update test to check if adjustend rewinding took place

 -- Balint Reczey <email address hidden> Fri, 26 Apr 2019 18:55:30 +0200

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.11

---------------
unattended-upgrades (1.1ubuntu1.18.04.11) bionic; urgency=medium

  * Detect changes to moved conffiles (LP: #1823872)
    - Add tests for checking conffile moves.
      Build depend on and use equivs to generate new test packages
    - Split() conffile data to set of names only once
    - Don't parse dpkg conffile db when there are no conffiles in the package
  * Detect unchanged moved conffiles.
    When a package moves a conffile properly without any change no conffile
    prompt needs to be shown thus the package can be upgraded unattended.
    (LP: #1823872)
  * Skip sending email when no package had to be installed, upgraded or removed
    (LP: #1821103) (Closes: #924554)
  * Make sure autoremovals don't start with a dirty cache and remove other
    packages (LP: #1824341)
  * Continue applying minimal sets when one set can't be marked for upgrade.
    Thanks to Anderson Luiz Alves for the patch, it needed minor modifications
    (LP: #1824341)
  * Stop raising NoAllowedOriginError when marking packages to upgrade/install
    fails (LP: #1824876)
  * Adjust only transitive dependencies in the fallback when a package from an
    allowed origin can't be marked to install/upgrade.
    This is a much lighter approach than marking every upgradable package
    because the full fallback was triggered on packages held back as well,
    using an excessive amount of CPU time.
    Also it crashed with packages not having any version in allowed origins.
    (LP: #1824804, #1824949)
  * Skip trying to upgrade held packages in call_adjusted() (LP: #1824804)
  * Follow all kinds of transitive dependencies when adjusting dependencies
  * Don't crash collecting transitive dependencies when package has no candidate
    (LP: #1825886)
  * Use mark_install_adjusted() in rewind_cache()
    The original cache had packages marked with adjustments thus rewinding
    should also do adjustments to reach the same state.
    Also not using mark_install_adjusted() crashes when apt raises error on
    held packages. (LP: #1826157)
    - test_rewind: Update test to check if adjustend rewinding took place

 -- Balint Reczey <email address hidden> Mon, 29 Apr 2019 12:13:14 +0200

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.3

---------------
unattended-upgrades (1.1ubuntu1.18.04.7~16.04.3) xenial; urgency=medium

  * Detect changes to moved conffiles (LP: #1823872)
    - Add tests for checking conffile moves.
      Build depend on and use equivs to generate new test packages
    - Split() conffile data to set of names only once
    - Don't parse dpkg conffile db when there are no conffiles in the package
  * Detect unchanged moved conffiles.
    When a package moves a conffile properly without any change no conffile
    prompt needs to be shown thus the package can be upgraded unattended.
    (LP: #1823872)
  * Skip sending email when no package had to be installed, upgraded or removed
    (LP: #1821103) (Closes: #924554)
  * Make sure autoremovals don't start with a dirty cache and remove other
    packages (LP: #1824341)
  * Continue applying minimal sets when one set can't be marked for upgrade.
    Thanks to Anderson Luiz Alves for the patch, it needed minor modifications
    (LP: #1824341)
  * Stop raising NoAllowedOriginError when marking packages to upgrade/install
    fails (LP: #1824876)
  * Adjust only transitive dependencies in the fallback when a package from an
    allowed origin can't be marked to install/upgrade.
    This is a much lighter approach than marking every upgradable package
    because the full fallback was triggered on packages held back as well,
    using an excessive amount of CPU time.
    Also it crashed with packages not having any version in allowed origins.
    (LP: #1824804, #1824949)
  * Skip trying to upgrade held packages in call_adjusted() (LP: #1824804)
  * Follow all kinds of transitive dependencies when adjusting dependencies
  * Don't crash collecting transitive dependencies when package has no candidate
    (LP: #1825886)
  * Use mark_install_adjusted() in rewind_cache()
    The original cache had packages marked with adjustments thus rewinding
    should also do adjustments to reach the same state.
    Also not using mark_install_adjusted() crashes when apt raises error on
    held packages. (LP: #1826157)
    - test_rewind: Update test to check if adjustend rewinding took place
  * do_auto_remove() is successful unless a commit() operation fails
    (LP: #1795696)
  * Compare apt.package.Version objects and not the versions' string
    representation. (LP: #1820888)
    This prevented adjusting candidates when the strings sorted differently.
    Also extend tests to catch issue.
  * Fall back to adjusting more packages' candidates
    when a package from an allowed origin can't be marked to install/upgrade.
    (LP: #1821101)

 -- Balint Reczey <email address hidden> Mon, 29 Apr 2019 12:23:14 +0200

This bug was fixed in the package unattended-upgrades - 1.10ubuntu5.1

---------------
unattended-upgrades (1.10ubuntu5.1) disco; urgency=medium

  * Skip trying to upgrade held packages in call_adjusted() (LP: #1824804)
  * Follow all kinds of transitive dependencies when adjusting dependencies
    (LP: #1824804)
  * Detect unchanged moved conffiles.
    When a package moves a conffile properly without any change no conffile
    prompt needs to be shown thus the package can be upgraded unattended even
    when the conffile is changed locally. (LP: #1823872)
  * Don't crash collecting transitive dependencies when package has no candidate
    (LP: #1825886)
  * Use mark_install_adjusted() in rewind_cache()
    The original cache had packages marked with adjustments thus rewinding
    should also do adjustments to reach the same state.
    Also not using mark_install_adjusted() crashes when apt raises error on
    held packages. (LP: #1826157)
    - test_rewind: Update test to check if adjusted rewinding took place

 -- Balint Reczey <email address hidden> Fri, 26 Apr 2019 15:00:50 +0200

Please add Breaks: unattended-upgrades (<< 1.1ubuntu1.18.04.11) to qemu-guest-agent for Bionic and similar breaks for other affected releases to prevent unfixed u-u from considering the upgrade the package. This allows installation of other security fixes.

@rbalint:
While it isn't clear from the bug updates alone IRC chat gave me the impression that this will be handled by you and the security Team (to push to -security) - is that correct or is any help needed that I miss?

@paelzer I would appreciate if you could handle this, I did not want to just drive by and update the package I usually don't touch and I just wanted to give a heads-up to the Security Team.

Since the fix relies on u-u's behavior I did not want to just throw the idea in a comment without testing it and since I tested it it would have made little sense to keep the patch to myself. :-)

I will also be away next week and I can't follow up on potential regressions (not that they would be likely to occur).

There are qemu packages for testing in the security team PPA here:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

This bug was fixed in the package qemu - 1:2.12+dfsg-3ubuntu8.8

---------------
qemu (1:2.12+dfsg-3ubuntu8.8) cosmic-security; urgency=medium

  * Add qemu-guest-agent Breaks: for unattended-upgrades versions not being
    able to install it to avoid qemu-guest-agent blocking other security
    updates. (LP: #1823872)

 -- Balint Reczey <email address hidden> Thu, 23 May 2019 15:09:48 +0200

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.14

---------------
qemu (1:2.11+dfsg-1ubuntu7.14) bionic-security; urgency=medium

  * Add qemu-guest-agent Breaks: for unattended-upgrades versions not being
    able to install it to avoid qemu-guest-agent blocking other security
    updates. (LP: #1823872)

 -- Balint Reczey <email address hidden> Thu, 23 May 2019 15:09:48 +0200