Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled

I also posted the same report on the qemu-devel mailinglist. Maybe they have any comments.
https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02270.html

Thanks for reporting this issue, I can reproduce it.

I can't reproduce this in Yakkety with qemu 2.6, which means it's a bad backport to earlier releases.

This bug was fixed in the package qemu - 2.0.0+dfsg-2ubuntu1.27

---------------
qemu (2.0.0+dfsg-2ubuntu1.27) trusty-security; urgency=medium

  * SECURITY REGRESSION: crash on migration with memory stats enabled
    (LP: #1612089)
    - debian/patches/CVE-2016-5403.patch: disable for now pending
      investigation.

 -- Marc Deslauriers <email address hidden> Fri, 12 Aug 2016 08:48:20 -0400

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.4

---------------
qemu (1:2.5+dfsg-5ubuntu10.4) xenial-security; urgency=medium

  * SECURITY REGRESSION: crash on migration with memory stats enabled
    (LP: #1612089)
    - debian/patches/CVE-2016-5403.patch: disable for now pending
      investigation.

 -- Marc Deslauriers <email address hidden> Fri, 12 Aug 2016 08:46:19 -0400

I run the cloud archive version and got the issue

qemu-kvm: 1:2.3+dfsg-5ubuntu9.4~cloud1 (trusty)

Hello Gaudenz, or anyone else affected,

Accepted qemu into kilo-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:kilo-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-kilo-needed to verification-kilo-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-kilo-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Gaudenz, or anyone else affected,

Accepted qemu into liberty-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:liberty-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-liberty-needed to verification-liberty-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-liberty-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Gaudenz, or anyone else affected,

Accepted qemu into icehouse-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:icehouse-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-icehouse-needed to verification-icehouse-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-icehouse-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

We encountered this problem with qemu-system 1:2.3+dfsg-5ubuntu9.4~cloud1 in trusty-updates liberty

qemu-system 1:2.3+dfsg-5ubuntu9.4~cloud2 in trusty-proposed liberty has fixed it for us. :)

@mdeslaur, do you have an ETA for the package that will address the CVE without regressing live migrations? I'd be interested to know for Trusty and Xenial, please. Or maybe there is another bug I should track for that? Thanks

Folks, thanks for working on this!

Today we looked at OSSN-0069 (https://bugs.launchpad.net/ossn/+bug/1534652), which we could reproduce on older instances. Since we have Liberty, which already has the fix, we could fix this by simply live-migrating all concerned instances.

But when testing this, most instances (the older ones) shut themselves off after live-migration *shriek*.

I found that when I install the new Qemu packages from liberty-proposed on a nova-compute node, then live migrations toward that node no longer lead to these crashes. Yay!

So I'm looking forward to these packages appearing in the regular Ubuntu Cloud Archive.

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.4~cloud0
---------------

 qemu (1:2.5+dfsg-5ubuntu10.4~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 qemu (1:2.5+dfsg-5ubuntu10.4) xenial-security; urgency=medium
 .
   * SECURITY REGRESSION: crash on migration with memory stats enabled
     (LP: #1612089)
     - debian/patches/CVE-2016-5403.patch: disable for now pending
       investigation.

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:2.3+dfsg-5ubuntu9.4~cloud2
---------------

 qemu (1:2.3+dfsg-5ubuntu9.4~cloud2) trusty-liberty; urgency=medium
 .
   * SECURITY REGRESSION: crash on migration with memory stats enabled
     (LP: #1612089)
     - debian/patches/CVE-2016-5403.patch: disable for now pending
       investigation.

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:2.2+dfsg-5expubuntu9.7~cloud7
---------------

 qemu (1:2.2+dfsg-5expubuntu9.7~cloud7) trusty-kilo; urgency=medium
 .
   * SECURITY REGRESSION: crash on migration with memory stats enabled
     (LP: #1612089)
     - debian/patches/CVE-2016-5403.patch: disable for now pending
       investigation.

If it is of any help, Stefan Hajnoczi has been working with me to help fix the regressions introduced by the CVE-2016-5403 fix (upstream QEMU commit afd9096, which is in 2.6.1 stable release) in a follow-up 2.6.2 release.

So far the following patches have been identified as being needed in order to correct the behavior introduced with the CVE fix. The upstream QEMU commit IDs are:

commit bccdef6b1a204db0f41ffb6e24ce373e4d7890d4
Author: Stefan Hajnoczi <email address hidden>
Date: Mon Aug 15 13:54:15 2016 +0100

    virtio: recalculate vq->inuse after migration

commit 58a83c61496eeb0d31571a07a51bc1947e3379ac
Author: Stefan Hajnoczi <email address hidden>
Date: Mon Aug 15 13:54:16 2016 +0100

    virtio: decrement vq->inuse in virtqueue_discard()

commit 4b7f91ed0270a371e1933efa21ba600b6da23ab9
Author: Stefan Hajnoczi <email address hidden>
Date: Wed Sep 7 11:51:25 2016 -0400

    virtio: zero vq->inuse in virtio_reset()

commit 104e70cae78bd4afd95d948c6aff188f10508a9c
Author: Ladi Prosek <email address hidden>
Date: Wed Sep 7 17:20:47 2016 +0200

    virtio-balloon: discard virtqueue element on reset

I believe it is the last of these which addresses the issue reported in this bug.

Thanks Michael for working on this, and listing the commits, that definitely helps!

On RHEL, the live migration crash was also noticed [1]. Michael S. Tsirkin has identified that backporting those two patches from 2.7 fixed the issue:

 virtio: decrement vq->inuse in virtqueue_discard()
 virtio: recalculate vq->inuse after migration

This confirms the findings of Michael Roth in comment 20, so it would be nice to have them SRU'ed.

1: https://bugzilla.redhat.com/show_bug.cgi?id=1372763

They will be included with the next round of security updates, possibly next week.

Thanks for the timeline update Marc.

Updates for this have now been released:
https://www.ubuntu.com/usn/usn-3125-1/

With this series of patches fixing CVE-2016-5403 some live migrations crash with error:

9006: error : qemuProcessReportLogError:1813 : internal error: early end of file from monitor, possible problem: 2016-11-15T12:55:51.353085Z qemu-system-x86_64: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x2
2016-11-15T12:55:51.353122Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:05.0/virtio-balloon'
2016-11-15T12:55:51.353265Z qemu-system-x86_64: load of migration failed: Operation not permitted
Inconsistency detected by ld.so: dl-close.c: 762: _dl_close: Assertion `map->l_init_called' failed!

Version of the affected qemu is 1:2.5+dfsg-5ubuntu10.6.

s10: please file a new bug so this new regression can be tracked properly. Thanks.

Looks like s10 never did file a separate bug for the regression, but we got hit by the same issue, so I filed https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1647389.