Bad page state in process genwqe_gunzip pfn:3c275 in the genwqe device driver

Default Comment by Bridge

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1559194/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

------- Comment From <email address hidden> 2016-03-31 06:36 EDT-------
Just a heads-up: the root cause for this problem is still under investigation.

Setting to incomplete for now, pending more information.

Hello this is being re-escalated by hws again, are there any actions that are expected from our side on this ticket? Or should this priority be lowered?

------- Comment From <email address hidden> 2016-04-12 11:40 EDT-------
IBM is working on a upstream fix. Therefore changed severity from ship -> high , for further tracking purposes.

As far as I understand this is a runtime bug, but is not install time critical. If we respin a kernel for any reason between now and final release we will try to include a fix for this bug if a backported patch for 4.4 stable exists. Othwerwise we will work on landing this as part of the SRU (Stable Release Update) cadence for the kernel. Therefore opening tasks for both xenial and y-series, in anticipation for a fix.

Is there an upstream fix for this problem? Should this ticket be closed, there were no updates for a while about this issue.

------- Comment From <email address hidden> 2016-08-09 10:12 EDT-------
This is the commit. does that help?

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?h=master-next&id=ab72705a2737c90c03d24b91afa10ef1b7403448

------- Comment From <email address hidden> 2016-09-02 07:48 EDT-------
(In reply to comment #29)
> This is the commit. does that help?
>
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/
> commit/?h=master-next&id=ab72705a2737c90c03d24b91afa10ef1b7403448

The error is still reproducible with kernel 4.4.0-33
Additional warning message before the BUG is available:

------------[ cut here ]------------
WARNING: at /build/linux-o03cxz/linux-4.4.0/arch/s390/include/asm/pci_dma.h:141
Modules linked in: qeth_l2 ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common genwqe_card qeth crc_itu_t qdio ccwgroup vmur dm_multipath dasd_eckd_mod dasd_mod
CPU: 2 PID: 3293 Comm: genwqe_gunzip Not tainted 4.4.0-33-generic #52-Ubuntu
task: 0000000032c7e270 ti: 00000000324e4000 task.ti: 00000000324e4000
Krnl PSW : 0404c00180000000 0000000000156346 (dma_update_cpu_trans+0x9e/0xa8)
R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
Krnl GPRS: 00000000324e7bcd 0000000000c3c34a 0000000027628298 000000003215b400
0000000000000400 0000000000001fff 0000000000000400 0000000116853000
07000000324e7b1e 0000000000000001 0000000000000001 0000000000000001
0000000000001000 0000000116854000 0000000000156402 00000000324e7a38
Krnl Code: 000000000015633a: 95001000 cli 0(%r1),0
000000000015633e: a774ffc3 brc 7,1562c4
#0000000000156342: a7f40001 brc 15,156344
>0000000000156346: 92011000 mvi 0(%r1),1
000000000015634a: a7f4ffbd brc 15,1562c4
000000000015634e: 0707 bcr 0,%r7
0000000000156350: c00400000000 brcl 0,156350
0000000000156356: eb7ff0500024 stmg %r7,%r15,80(%r15)
Call Trace:
([<00000000001563e0>] dma_update_trans+0x90/0x228)
[<00000000001565dc>] s390_dma_unmap_pages+0x64/0x160
[<00000000001567c2>] s390_dma_free+0x62/0x98
[<000003ff801310ce>] __genwqe_free_consistent+0x56/0x70 [genwqe_card]
[<000003ff801316d0>] genwqe_free_sync_sgl+0xf8/0x160 [genwqe_card]
[<000003ff8012bd6e>] ddcb_cmd_cleanup+0x86/0xa8 [genwqe_card]
[<000003ff8012c1c0>] do_execute_ddcb+0x110/0x348 [genwqe_card]
[<000003ff8012c914>] genwqe_ioctl+0x51c/0xc20 [genwqe_card]
[<000000000032513a>] do_vfs_ioctl+0x3b2/0x518
[<0000000000325344>] SyS_ioctl+0xa4/0xb8
[<00000000007b86c6>] system_call+0xd6/0x264
[<000003ff9e8e520a>] 0x3ff9e8e520a
Last Breaking-Event-Address:
[<0000000000156342>] dma_update_cpu_trans+0x9a/0xa8
---[ end trace 35996336235145c8 ]---
BUG: Bad page state in process jbd2/dasdb1-8 pfn:3215b
page:000003d100c856c0 count:-1 mapcount:0 mapping: (null) index:0x0
flags: 0x3fffc0000000000()
page dumped because: nonzero _count

------- Comment (attachment only) From <email address hidden> 2016-09-02 08:48 EDT-------

------- Comment From <email address hidden> 2016-09-02 09:22 EDT-------
From the dmesg it looks like this time ext4 page allocation stumbles upon the doubly freed page first, but it is immediately after the page got corrupted by the double free (indicated by the WARNING), so this just means that ext4 happened to be the first to get its fingers on the corrupted page during a page alloc. It could hit anyone, and we also see later another occurrence where copy_pte_range() stumbles over another corrupted page (no WARNING before that because it is a WARN_ONCE).

We still need to find the root cause for the double free and the resulting page corruption (count -1), and for that we only have the WARNING trace as reliable hint for a double free. So my analysis from comment #5 is still valid, even though this time genwqe itself is not the one who stumbled over the corrupted page, it was still involved in the double free (anyone can see the corrupted page afterwards, genwqe was just a more likely candidate because it was an active consumer at the time).

BTW, instead of "double free" of course a call of dma_free() on previously unmapped addresses would result in the same issue, but a double free is much more likely, e.g. caused by broken error handling with "off by one" or other issues. Speaking of error handling, the "genwqe 0001:00:00.0: [genwqe_map_pages] err: no dma addr daddr=ffffffffffffffff!" messages may be a good starting point to verify the genwqe error handling and the page freeing strategy. Those messages by itself are no problem and even expected given the nature of the test (online/offline and failing rpcit), but of course there is some error handling involved which may have issues that could lead to a double free.

------- Comment From <email address hidden> 2016-09-06 10:21 EDT-------
I think I found the bug in the genwqe code:

ddcb_cmd_fixups() -> genwqe_alloc_sync_sgl() (fails in f/lpage, but sgl->sgl != NULL and f/lpage maybe also != NULL) -> ddcb_cmd_cleanup() -> genwqe_free_sync_sgl() (double free, because sgl->sgl != NULL and f/lpage maybe also != NULL)

In this scenario we would have exactly the kind of double free that would explain the WARNING / Bad page state, and as expected it is caused by broken error handling (cleanup).

Not being familiar with the genwqe code, it would be good if Frank could have a look at the patch.

Using the Ubuntu git source, tag Ubuntu-4.4.0-33.52, I can reproduce the "Bad page state" issue, and with the patch on top I cannot reproduce it any more.

Alex, I'll attach a debian kernel package with the patch applied, please verify if it also solves the issue for you.

Frank, I'll attach the patch, please comment.

------- Comment on attachment From <email address hidden> 2016-09-06 10:23 EDT-------

Fix double free in the genwqe code, resulting from missing pointer clearing in genwqe_alloc_sync_sgl() error handling.

------- Comment on attachment From <email address hidden> 2016-09-06 10:25 EDT-------

Debian kernel package with genwqe-bad-page.patch applied.

------- Comment From <email address hidden> 2016-09-12 10:45 EDT-------
(In reply to comment #36)

> Alex, I'll attach a debian kernel package with the patch applied, please
> verify if it also solves the issue for you.

Thanks, with the attached kernel I could not reproduce this issue.

------- Comment on attachment From <email address hidden> 2016-09-06 10:25 EDT-------

Debian kernel package with genwqe-bad-page.patch applied.

Hello IBM - is this patch going upstream ? It is not currently in a consumable form, i.e., no provenance.

------- Comment on attachment From <email address hidden> 2016-09-06 10:25 EDT-------

Debian kernel package with genwqe-bad-page.patch applied.

------- Comment From <email address hidden> 2016-10-06 09:13 EDT-------
We are waiting for review/approval from the genwqe owner fromFrank Haverkamp, assigning LTC bugzilla to him.

Frank, please handle ASAP in order to meet the current merge window for 4.9.

------- Comment on attachment From <email address hidden> 2016-09-06 10:25 EDT-------

Debian kernel package with genwqe-bad-page.patch applied.

Default Comment by Bridge

------- Comment on attachment From <email address hidden> 2016-09-06 10:25 EDT-------

Debian kernel package with genwqe-bad-page.patch applied.

------- Comment From <email address hidden> 2016-10-19 11:41 EDT-------
Patch was submitted upstream, should be added in a few days, here is the LKML discussion link: https://marc.info/?l=linux-kernel&m=147688806928316&w=2

------- Comment on attachment From <email address hidden> 2016-09-06 10:25 EDT-------

Debian kernel package with genwqe-bad-page.patch applied.

https://lists.ubuntu.com/archives/kernel-team/2016-October/080462.html

------- Comment on attachment From <email address hidden> 2016-09-06 10:25 EDT-------

Debian kernel package with genwqe-bad-page.patch applied.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Debian kernel package with genwqe-bad-page.patch applied.

------- Comment From <email address hidden> 2016-11-21 08:05 EDT-------
Verified , could not be reproduced anymore..

Debian kernel package with genwqe-bad-page.patch applied.

As per comment #37 I'm tagging this bug as verified.

The verification of the Stable Release Update for linux-lts-xenial has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package linux - 4.4.0-51.72

---------------
linux (4.4.0-51.72) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1644611

  * 4.4.0-1037-snapdragon #41: kernel panic on boot (LP: #1644596)
    - Revert "dma-mapping: introduce the DMA_ATTR_NO_WARN attribute"
    - Revert "powerpc: implement the DMA_ATTR_NO_WARN attribute"
    - Revert "nvme: use the DMA_ATTR_NO_WARN attribute"

linux (4.4.0-50.71) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1644169

  * xenial 4.4.0-49.70 kernel breaks LXD userspace (LP: #1644165)
    - Revert "UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts by
      default"
    - Revert "UBUNTU: SAUCE: (namespace) fs: Don't remove suid for CAP_FSETID for
      userns root"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Don't remove suid for
      CAP_FSETID in s_user_ns""
    - Revert "UBUNTU: SAUCE: (namespace) fs: Allow superblock owner to change
      ownership of inodes"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Allow superblock owner to
      change ownership of inodes with unmappable ids""
    - Revert "UBUNTU: SAUCE: (namespace) security/integrity: Harden against
      malformed xattrs"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: ima/evm: Allow root in s_user_ns
      to set xattrs""
    - Revert "(namespace) dquot: For now explicitly don't support filesystems
      outside of init_user_ns"
    - Revert "(namespace) quota: Handle quota data stored in s_user_ns in
      quota_setxquota"
    - Revert "(namespace) quota: Ensure qids map to the filesystem"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: quota: Convert ids relative to
      s_user_ns""
    - Revert "(namespace) Revert "UBUNTU: SAUCE: quota: Require that qids passed
      to dqget() be valid and map into s_user_ns""
    - Revert "(namespace) vfs: Don't create inodes with a uid or gid unknown to
      the vfs"
    - Revert "(namespace) vfs: Don't modify inodes with a uid or gid unknown to
      the vfs"
    - Revert "UBUNTU: SAUCE: (namespace) fuse: Translate ids in posix acl xattrs"
    - Revert "UBUNTU: SAUCE: (namespace) posix_acl: Export
      posix_acl_fix_xattr_userns() to modules"
    - Revert "(namespace) vfs: Verify acls are valid within superblock's
      s_user_ns."
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Update posix_acl support to
      handle user namespace mounts""
    - Revert "(namespace) fs: Refuse uid/gid changes which don't map into
      s_user_ns"
    - Revert "(namespace) Revert "UBUNTU: SAUCE: fs: Refuse uid/gid changes which
      don't map into s_user_ns""
    - Revert "(namespace) mnt: Move the FS_USERNS_MOUNT check into sget_userns"

linux (4.4.0-49.70) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1640921

  * Infiniband driver (kernel module) needed for Azure (LP: #1641139)
    - SAUCE: RDMA Infiniband for Windows Azure
    - [Config] CONFIG_HYPERV_INFINIBAND_ND=m
    - SAUCE: Makefile RDMA infiniband driver for Windows Azure
    - [Config] Add hv_network_direct.ko to generic inclusion list
    - SAUCE: RDMA Infiniband for Windows Azure is dependent on amd64...

This bug was fixed in the package linux - 4.8.0-28.30

---------------
linux (4.8.0-28.30) yakkety; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1641083

  * lxc-attach to malicious container allows access to host (LP: #1639345)
    - Revert "UBUNTU: SAUCE: (noup) ptrace: being capable wrt a process requires
      mapped uids/gids"
    - (upstream) mm: Add a user_ns owner to mm_struct and fix ptrace permission
      checks

  * [Feature] AVX-512 new instruction sets (avx512_4vnniw, avx512_4fmaps)
    (LP: #1637526)
    - x86/cpufeature: Add AVX512_4VNNIW and AVX512_4FMAPS features

  * zfs: importing zpool with vdev on zvol hangs kernel (LP: #1636517)
    - SAUCE: (noup) Update zfs to 0.6.5.8-0ubuntu4.1

  * Move some device drivers build from kernel built-in to modules
    (LP: #1637303)
    - [Config] CONFIG_TIGON3=m for all arches
    - [Config] CONFIG_VIRTIO_BLK=m, CONFIG_VIRTIO_NET=m

  * I2C touchpad does not work on AMD platform (LP: #1612006)
    - pinctrl/amd: Configure GPIO register using BIOS settings

  * guest experiencing Transmit Timeouts on CX4 (LP: #1636330)
    - powerpc/64: Re-fix race condition between going idle and entering guest
    - powerpc/64: Fix race condition in setting lock bit in idle/wakeup code

  * QEMU throws failure msg while booting guest with SRIOV VF (LP: #1630554)
    - KVM: PPC: Always select KVM_VFIO, plus Makefile cleanup

  * [Feature] KBL - New device ID for Kabypoint(KbP) (LP: #1591618)
    - SAUCE: mfd: lpss: Fix Intel Kaby Lake PCH-H properties

  * hio: SSD data corruption under stress test (LP: #1638700)
    - SAUCE: hio: set bi_error field to signal an I/O error on a BIO
    - SAUCE: hio: splitting bio in the entry of .make_request_fn

  * cleanup primary tree for linux-hwe layering issues (LP: #1637473)
    - [Config] switch Vcs-Git: to yakkety repository
    - [Packaging] handle both linux-lts* and linux-hwe* as backports
    - [Config] linux-tools-common and linux-cloud-tools-common are one per series
    - [Config] linux-source-* is in the primary linux namespace
    - [Config] linux-tools -- always suggest the base package

  * SRU: sync zfsutils-linux and spl-linux changes to linux (LP: #1635656)
    - SAUCE: (noup) Update spl to 0.6.5.8-2, zfs to 0.6.5.8-0ubuntu4 (LP:
      #1635656)

  * [Feature] SKX: perf uncore PMU support (LP: #1591810)
    - perf/x86/intel/uncore: Add Skylake server uncore support
    - perf/x86/intel/uncore: Remove hard-coded implementation for Node ID mapping
      location
    - perf/x86/intel/uncore: Handle non-standard counter offset

  * [Feature] Purley: Memory Protection Keys (LP: #1591804)
    - x86/pkeys: Add fault handling for PF_PK page fault bit
    - mm: Implement new pkey_mprotect() system call
    - x86/pkeys: Make mprotect_key() mask off additional vm_flags
    - x86/pkeys: Allocation/free syscalls
    - x86: Wire up protection keys system calls
    - generic syscalls: Wire up memory protection keys syscalls
    - pkeys: Add details of system call use to Documentation/
    - x86/pkeys: Default to a restrictive init PKRU
    - x86/pkeys: Allow configuration of init_pkru
    - x86/pkeys: Add self-tests

  * kernel invalid ...

Debian kernel package with genwqe-bad-page.patch applied.

------- Comment From <email address hidden> 2016-11-30 12:30 EDT-------
Verified with Ubuntu 16.04.1-LTS, the issue can no longer be reproduced.

root@s8330026:~# uname -a
Linux s8330026 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:29:36 UTC 2016 s390x s390x s390x GNU/Linux

Debian kernel package with genwqe-bad-page.patch applied.

This bug was fixed in the package linux - 4.8.0-30.32

---------------
linux (4.8.0-30.32) yakkety; urgency=low

  * CVE-2016-8655 (LP: #1646318)
    - packet: fix race condition in packet_set_ring

 -- Brad Figg <email address hidden> Thu, 01 Dec 2016 08:02:53 -0800

Debian kernel package with genwqe-bad-page.patch applied.

Debian kernel package with genwqe-bad-page.patch applied.