NFS regression causes subsequent mounts from same superblock to silently use previous mount options

Hi Philip,

Thank you for your report. Just to clarify, this is also a security issue for Feisty correct? Thanks.

Leann,

Yes, the problem is also reproducible in Feisty.

It should be obvious from my description above, but I wanted to specifically mention that this regression makes it impossible to mount two directories from the same block device with different options. An example of a case that worked in Edgy but no longer works in Feisty/Gutsy is using sec=sys for one mount point and sec=krb5 for another.

I've attached a tested patch which applies cleanly to linux-source-2.6.22.9-14 and resolves the issue on Gutsy. Note, however, that I've only tested the patch with an NFSv3 client.

The patch above also applies cleanly to Feisty's linux-source-2.6.20-16.32 kernel and builds. The system boots successfully and this bug is not reproducible with the patched kernel. Other NFSv3 mounts continue to work based on my limited testing.

I backported the patch above to the Gutsy kernel based on the patch at:
   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=e89a5a43b95cdc4305b7c8e8121a380f02476636

I just wanted to add a note that these patches were automatically merged into the Hardy Heron kernel when it was sync'd with mainline. Thanks!

Leanne,

It has been a few weeks since I submitted this bug and I haven't heard anything from the teams responsible for it. I did my best to hand over the bug on a silver platter (at the cost of a fair deal of my own time), so a couple answers would set my head at ease:

- Has the bug been confirmed? As a security issue?
- Is someone working on or planning on getting an update out for Gutsy? Feisty?
- Are there any updates at all from the kernel team?

Thanks,
Philip

Hi Philip,

I apologize for any lack of communication/updates regarding your report. The effort and time you have invested in getting this issue reosolved has not been overlooked and has been extremely helpful to us. I hope I can answer some of your questions:

1) Yes this bug has been confirmed and is a security issue.
2) Yes, a member of the kernel team is actively backporting this fix to Gutsy. For some reason I thought they had already posted a comment and updated you on this report.
3) I'll nudge the kernel team again to post a comment here regarding the current status.

Again, thank you so much for your help and your continued intersted in getting this resolved.

Leanne,

Thanks a lot for the update. It's good to know that a fix is in the pipeline. I hope my previous message didn't come across as too snarky.

From the kernel team. Thanks for your work on this bug. Your patch has been evaluated and integrated into the kernel. It should be released very soon in the next lum security updates.

2.6.22 wont be in hardy.

Phillip, can you cleanup the Affects/targets for this bug. It's hard to see where it's fixed already. Note, "Affects: linux" should only be for hardy at the moment.

I've cleaned up the tasks. This will be published as part of USN-558-1 shortly.

Was this fix applied in kernel 2.6.22-14.47?

Two people reported kernel oops this mornings with the new kernel released on Dec. 18 and nfs4 shares. See

https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/148600/comments/3

https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/148600/comments/4

Claude,

I apologize for missing an important part of the patch in my original submission which caused the NFSv4 regression that you are experiencing. If I'd know that Canonical wasn't going to do quality assurance of kernel patches to their stable release branch (despite my warning that I'd only tested NFSv3 clients) I would have tested the NFSv4 portion of the code myself.

After seeing this report, I set up an NFSv4 server to test, reproduce and fix the problem with my patch. NFSv4 and NFSv4 + Kerberos mounts are now working with the revised patch (attached). Note: this patch applies to the 2.6.22-14.46 kernel. I will attach a patch to 2.6.22-14.47 that fixes this regression momentarily.

Thanks,
Philip

Reopening due to introduction of NFSv4 problem in patch

Thanks Philip,

Don't feel too bad... we all make mistakes :) I reverted to 21.6.22-14.46 temporarily.

The regression is being investigated. Sorry about the trouble; we should have a solution published shortly. For anyone that needs to roll-back to the earlier kernel, you can use:

 IMGPKG=$(dpkg -S /boot/vmlinuz-2.6.22-14* | cut -d: -f1)
 sudo apt-get install $IMGPKG=2.6.22-14.46

(Basically, force a down-grade of the kernel-image-2.6.22 flavour that is installed)

Oops, ignore that attachment...wrong bug

As promised, here is the fix to unbreak NFSv4 in the 2.6.22-14.47 kernel.

I independently tested:
- Mounting NFSv4 export (sec=sys and sec=krb5)
- Reading/writing from partition, including seeks (tail)

Claude,

If you are willing to test the patch (or a new nfs.ko or linux-image-2.6.22-14-generic.deb, disclaiming the potential danger of running untrusted kernel code) in your setup to ensure that it fixes the problem you are experiencing, please contact me on IRC (<email address hidden>:#ubuntu-bugs) or respond here and I'll upload the fixed module to this bug.

Kees,

Feel free to contact me on IRC if you have any questions.

Thanks,
Philip

OK Philip,

I'm trying to reach you through IRC as we speak.

Here's the nfs.ko compiled against 2.6.22-14.47. Make sure that if you use this you first back up the current one in /lib/modules. You can attempt to reinsert the module into your running kernel or reboot.

Claude tested the above patched module and reported that everything worked well.

Just tried the rebuilt nfs.ko linked above, it appears to work. However it seems to be causing issues with the nvidia propriety module, it won't load. I'll post the specifics when I have a second box running to post from.

Ignore above problem, tracked it down to something else entirely

Charles,

You may need to run 'dpkg-reconfigure linux-restricted-modules-2.6.22-14-generic' to rebuild your module list to include the volatile modules ('depmod -a' on a fully booted system should also work). You can check to see if this is the cause by comparing the output of the following command with your machine:

$ modprobe -l |grep nvidia.ko
/lib/modules/2.6.22-14-generic/volatile/nvidia.ko

The patched kernel module resolved the problem for me. Thank you.

-- David P.

Is there already a plan to release the fix for NFSv4 problem? I only ask as there haven't been any changes for almost a month now. We would like to migrate a bunch of machines to Ubuntu, and having a working default kernel would save us quite some work.

Thanks in advance!

Philip,

is it possible to push your patch to the Ubuntu distro pool? I'm using linux-image-2.6.22-14-xen 2.6.22-14.47 and also would like to get NFSv4 working as I'm using some Solaris in domU and would like to share ZFS-based storage from it with dom0 Ubuntu.

Thanks!
Karel
PS: otherwise, is it hard to rebuild Ubuntu kernel? How is it different from Debian?

This bug was fixed in the package linux-source-2.6.22 - 2.6.22-14.51

---------------
linux-source-2.6.22 (2.6.22-14.51) gutsy-security; urgency=low

  [Amit Kucheria]

  * Poulsbo: Mass update of all patches from moblin repo
  * Update config.lpia to reflect new patches
  * [sata_sil][sata->ide-bridg] failed to set xfermode
    - LP: #153096
  * Poulsbo: remove extra patch

  [Kees Cook]

  * fix NFSv4 client mount regression
    - LP: #164231

  [Tim Gardner]

  * Support of new AMD PowerNow! (family 0x11 and beyond)
    - LP: #185649

  [Upstream Kernel Changes]

  * minixfs: limit minixfs printks on corrupted dir i_size (CVE-2006-6058)
  * [JFFS2] Fix ACL vs. mode handling. (CVE-2007-4849)
  * [IEEE80211]: avoid integer underflow for runt rx frames (CVE-2007-4997)
  * [TCP]: Make sure write_queue_from does not begin with NULL ptr
    (CVE-2007-5501)
  * wait_task_stopped: Check p->exit_state instead of TASK_TRACED
    (CVE-2007-5500)
  * fix DLM regression
  * CVE-2008-0001: Use access mode instead of open flags to determine
    needed permissions
  * hrtimers: avoid overflow for large relative timeouts (CVE-2007-5966)
  * isdn: avoid copying overly-long strings (CVE-2007-6063)
  * I4L: fix isdn_ioctl memory overrun vulnerability (CVE-2007-6151)
  * vfs: coredumping fix (CVE-2007-6206)
  * tmpfs: restore missing clear_highpage (CVE-2007-6417)
  * [UBUNTU] fs/dlm: Fix regression introduced with last security fix.

 -- Tim Gardner <email address hidden> Mon, 28 Jan 2008 13:46:21 -0700

This bug was fixed in 2.6.20-16.34

---
 linux-source-2.6.20 (2.6.20-16.34) feisty-security; urgency=low

   [Kees Cook]

   * fix NFSv4 client mount regression
     - GIT-SHA 7a9e181ce37e0a862261b1814ee3ef358036dd5e
     - Bug #164231
   * ppc64: fix corrupted sigcontext during FPU stress (CVE-2007-3107)
     - GIT-SHA 7117942afa58a6331cc2abc498541b2784cf79ac

   [Upstream Kernel Changes]

   * minixfs: limit minixfs printks on corrupted dir i_size (CVE-2006-6058)
   * [IPV6]: Do no rely on skb->dst before it is assigned. (CVE-2007-4567)
   * [JFFS2] Fix ACL vs. mode handling. (CVE-2007-4849)
   * [IEEE80211]: avoid integer underflow for runt rx frames (CVE-2007-4997)
   * USB: fix DoS in pwc USB video driver (CVE-2007-5093)
   * Fix debug regression in video/pwc
   * wait_task_stopped: Check p->exit_state instead of TASK_TRACED
     (CVE-2007-5500)
   * fix DLM regression
   * CVE-2008-0001: Use access mode instead of open flags to determine
     needed permissions
   * hrtimers: avoid overflow for large relative timeouts (CVE-2007-5966)
   * isdn: avoid copying overly-long strings (CVE-2007-6063)
   * I4L: fix isdn_ioctl memory overrun vulnerability (CVE-2007-6151)
   * vfs: coredumping fix (CVE-2007-6206)
   * tmpfs: restore missing clear_highpage (CVE-2007-6417)
   * [UBUNTU] fs/dlm: Fix regression introduced with last security fix.

Unless I'm mistaken, this bug is already resolved in Hardy since Hardy received the upstream kernel patch automatically.