Severe NFS performance degradation after LP #2003053

Bug #2022098 reported by Khaled El Mously
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned
Focal
Fix Released
Medium
Unassigned
Jammy
Fix Released
Medium
Unassigned
linux-gke (Ubuntu)
Confirmed
Critical
Khaled El Mously
Focal
Fix Released
Critical
Khaled El Mously
Jammy
Fix Released
Critical
Khaled El Mously

Bug Description

The fix to LP #2003053 has caused massively increased NFS server access in some use-cases, which caused severe performance degradation to the point of being unusable in some cases.

The solution to this issue, at least temporarily and at least for linux-gke, is to make the new behaviour optional using the "nfs_fasc=1" module parameter. Without this parameter specified, (or specified as =0) will keep the old behaviour.

Regresion potential:
 - Regression potential is considered low considering the scope of the change and is limited to NFS only.

no longer affects: linux (Ubuntu)
Changed in linux-gke (Ubuntu):
assignee: nobody → Khaled El Mously (kmously)
Changed in linux-gke (Ubuntu Focal):
assignee: nobody → Khaled El Mously (kmously)
Changed in linux-gke (Ubuntu Jammy):
assignee: nobody → Khaled El Mously (kmously)
Changed in linux-gke (Ubuntu):
importance: Undecided → Critical
Changed in linux-gke (Ubuntu Focal):
importance: Undecided → Critical
Changed in linux-gke (Ubuntu Jammy):
importance: Undecided → Critical
description: updated
description: updated
Changed in linux-gke (Ubuntu Focal):
status: New → Fix Committed
Changed in linux-gke (Ubuntu Jammy):
status: New → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2022098

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Changed in linux (Ubuntu Focal):
status: New → Incomplete
Changed in linux (Ubuntu Jammy):
status: New → Incomplete
Changed in linux (Ubuntu Focal):
importance: Undecided → Critical
Changed in linux (Ubuntu Jammy):
importance: Undecided → Critical
status: Incomplete → In Progress
Changed in linux (Ubuntu Focal):
status: Incomplete → In Progress
Stefan Bader (smb)
Changed in linux (Ubuntu Jammy):
importance: Critical → Medium
Changed in linux (Ubuntu Focal):
importance: Critical → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-gke - 5.4.0-1101.108

---------------
linux-gke (5.4.0-1101.108) focal; urgency=medium

  * focal/linux-gke: 5.4.0-1101.108 -proposed tracker (LP: #2022106)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

  * Severe NFS performance degradation after LP #2003053 (LP: #2022098)
    - SAUCE: Make NFS file-access stale cache behaviour opt-in

 -- Khalid Elmously <email address hidden> Thu, 01 Jun 2023 16:22:47 -0400

Changed in linux-gke (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-gke - 5.15.0-1035.40

---------------
linux-gke (5.15.0-1035.40) jammy; urgency=medium

  * jammy/linux-gke: 5.15.0-1035.40 -proposed tracker (LP: #2022105)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

  * Severe NFS performance degradation after LP #2003053 (LP: #2022098)
    - SAUCE: Make NFS file-access stale cache behaviour opt-in

 -- Khalid Elmously <email address hidden> Thu, 01 Jun 2023 16:59:59 -0400

Changed in linux-gke (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-gke-5.15/5.15.0-1036.41~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-gke-5.15 verification-needed-focal
Stefan Bader (smb)
Changed in linux (Ubuntu Jammy):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.15.0-77.84 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux verification-needed-jammy
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-154.171 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux
tags: added: verification-done-focal verification-done-jammy
removed: verification-needed-focal verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-gke (Ubuntu):
status: New → Confirmed
IHME SA (ihmesa)
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

I am removing the duplication of this bug due to the fact that this requires a separate fix from bug 2015827, and the fix is still applied and queued for release.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-nvidia-tegra-igx/5.15.0-1002.2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-nvidia-tegra-igx verification-needed-jammy
removed: verification-done-jammy
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-nvidia-tegra-5.15/5.15.0-1016.16~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-nvidia-tegra-5.15 verification-needed-focal
removed: verification-done-focal
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-nvidia-tegra/5.15.0-1016.16 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-nvidia-tegra
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (21.8 KiB)

This bug was fixed in the package linux - 5.4.0-156.173

---------------
linux (5.4.0-156.173) focal; urgency=medium

  * focal/linux: 5.4.0-156.173 -proposed tracker (LP: #2026585)

  * CVE-2023-3390
    - netfilter: nf_tables: incorrect error path handling with NFT_MSG_NEWRULE

  * Focal update: v5.4.241 upstream stable release (LP: #2023930)
    - scsi: ses: Handle enclosure with just a primary component gracefully
    - x86/PCI: Add quirk for AMD XHCI controller that loses MSI-X state in D3hot
    - cgroup/cpuset: Wake up cpuset_attach_wq tasks in cpuset_cancel_attach()
    - treewide: Replace DECLARE_TASKLET() with DECLARE_TASKLET_OLD()
    - smb3: fix problem with null cifs super block with previous patch
    - pinctrl: amd: Use irqchip template
    - pinctrl: amd: disable and mask interrupts on probe
    - pinctrl: amd: Disable and mask interrupts on resume
    - pwm: cros-ec: Explicitly set .polarity in .get_state()
    - pwm: sprd: Explicitly set .polarity in .get_state()
    - wifi: mac80211: fix invalid drv_sta_pre_rcu_remove calls for non-uploaded
      sta
    - icmp: guard against too small mtu
    - net: don't let netpoll invoke NAPI if in xmit context
    - sctp: check send stream number after wait_for_sndbuf
    - ipv6: Fix an uninit variable access bug in __ip6_make_skb()
    - gpio: davinci: Add irq chip flag to skip set wake
    - sunrpc: only free unix grouplist after RCU settles
    - NFSD: callback request does not use correct credential for AUTH_SYS
    - xhci: also avoid the XHCI_ZERO_64B_REGS quirk with a passthrough iommu
    - USB: serial: cp210x: add Silicon Labs IFS-USB-DATACABLE IDs
    - usb: typec: altmodes/displayport: Fix configure initial pin assignment
    - USB: serial: option: add Telit FE990 compositions
    - USB: serial: option: add Quectel RM500U-CN modem
    - iio: adc: ti-ads7950: Set `can_sleep` flag for GPIO chip
    - iio: dac: cio-dac: Fix max DAC write value check for 12-bit
    - tty: serial: sh-sci: Fix transmit end interrupt handler
    - tty: serial: sh-sci: Fix Rx on RZ/G2L SCI
    - tty: serial: fsl_lpuart: avoid checking for transfer complete when
      UARTCTRL_SBK is asserted in lpuart32_tx_empty
    - nilfs2: fix potential UAF of struct nilfs_sc_info in nilfs_segctor_thread()
    - nilfs2: fix sysfs interface lifetime
    - ALSA: hda/realtek: Add quirk for Clevo X370SNW
    - perf/core: Fix the same task check in perf_event_set_output
    - ftrace: Mark get_lock_parent_ip() __always_inline
    - can: j1939: j1939_tp_tx_dat_new(): fix out-of-bounds memory access
    - tracing: Free error logs of tracing instances
    - net_sched: prevent NULL dereference if default qdisc setup failed
    - drm/panfrost: Fix the panfrost_mmu_map_fault_addr() error path
    - ring-buffer: Fix race while reader and writer are on the same page
    - mm/swap: fix swap_info_struct race between swapoff and get_swap_pages()
    - irqdomain: Look for existing mapping only once
    - irqdomain: Refactor __irq_domain_alloc_irqs()
    - irqdomain: Fix mapping-creation race
    - Revert "pinctrl: amd: Disable and mask interrupts on resume"
    - ALSA: emu10k1: fix capture interrupt handler unlinking
    - ALSA: hd...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (83.7 KiB)

This bug was fixed in the package linux - 5.15.0-79.86

---------------
linux (5.15.0-79.86) jammy; urgency=medium

  * jammy/linux: 5.15.0-79.86 -proposed tracker (LP: #2026531)

  * Jammy update: v5.15.111 upstream stable release (LP: #2025095)
    - ASOC: Intel: sof_sdw: add quirk for Intel 'Rooks County' NUC M15
    - ASoC: soc-pcm: fix hw->formats cleared by soc_pcm_hw_init() for dpcm
    - x86/hyperv: Block root partition functionality in a Confidential VM
    - iio: adc: palmas_gpadc: fix NULL dereference on rmmod
    - ASoC: Intel: bytcr_rt5640: Add quirk for the Acer Iconia One 7 B1-750
    - selftests mount: Fix mount_setattr_test builds failed
    - asm-generic/io.h: suppress endianness warnings for readq() and writeq()
    - x86/cpu: Add model number for Intel Arrow Lake processor
    - wireguard: timers: cast enum limits members to int in prints
    - wifi: mt76: mt7921e: Set memory space enable in PCI_COMMAND if unset
    - arm64: Always load shadow stack pointer directly from the task struct
    - arm64: Stash shadow stack pointer in the task struct on interrupt
    - PCI: pciehp: Fix AB-BA deadlock between reset_lock and device_lock
    - PCI: qcom: Fix the incorrect register usage in v2.7.0 config
    - IMA: allow/fix UML builds
    - USB: dwc3: fix runtime pm imbalance on probe errors
    - USB: dwc3: fix runtime pm imbalance on unbind
    - hwmon: (k10temp) Check range scale when CUR_TEMP register is read-write
    - hwmon: (adt7475) Use device_property APIs when configuring polarity
    - posix-cpu-timers: Implement the missing timer_wait_running callback
    - blk-mq: release crypto keyslot before reporting I/O complete
    - blk-crypto: make blk_crypto_evict_key() return void
    - blk-crypto: make blk_crypto_evict_key() more robust
    - ext4: use ext4_journal_start/stop for fast commit transactions
    - staging: iio: resolver: ads1210: fix config mode
    - tty: Prevent writing chars during tcsetattr TCSADRAIN/FLUSH
    - xhci: fix debugfs register accesses while suspended
    - tick/nohz: Fix cpu_is_hotpluggable() by checking with nohz subsystem
    - MIPS: fw: Allow firmware to pass a empty env
    - ipmi:ssif: Add send_retries increment
    - ipmi: fix SSIF not responding under certain cond.
    - kheaders: Use array declaration instead of char
    - wifi: mt76: add missing locking to protect against concurrent rx/status
      calls
    - pwm: meson: Fix axg ao mux parents
    - pwm: meson: Fix g12a ao clk81 name
    - soundwire: qcom: correct setting ignore bit on v1.5.1
    - pinctrl: qcom: lpass-lpi: set output value before enabling output
    - ring-buffer: Sync IRQ works before buffer destruction
    - crypto: api - Demote BUG_ON() in crypto_unregister_alg() to a WARN_ON()
    - crypto: safexcel - Cleanup ring IRQ workqueues on load failure
    - rcu: Avoid stack overflow due to __rcu_irq_enter_check_tick() being kprobe-
      ed
    - reiserfs: Add security prefix to xattr name in reiserfs_security_write()
    - KVM: nVMX: Emulate NOPs in L2, and PAUSE if it's not intercepted
    - relayfs: fix out-of-bounds access in relay_file_read
    - writeback, cgroup: fix null-ptr-deref write in bdi_split_work_to_wbs
 ...

Changed in linux (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-ibm/5.4.0-1055.60 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-ibm' to 'verification-done-focal-linux-ibm'. If the problem still exists, change the tag 'verification-needed-focal-linux-ibm' to 'verification-failed-focal-linux-ibm'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-ibm-v2 verification-needed-focal-linux-ibm
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.4.0-1114.120 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-azure' to 'verification-done-focal-linux-azure'. If the problem still exists, change the tag 'verification-needed-focal-linux-azure' to 'verification-failed-focal-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-azure-v2 verification-needed-focal-linux-azure
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-aws/5.15.0-1044.49 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws' to 'verification-done-jammy-linux-aws'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws' to 'verification-failed-jammy-linux-aws'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-aws-v2 verification-needed-jammy-linux-aws
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-bluefield/5.4.0-1069.75 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-bluefield' to 'verification-done-focal-linux-bluefield'. If the problem still exists, change the tag 'verification-needed-focal-linux-bluefield' to 'verification-failed-focal-linux-bluefield'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-bluefield-v2 verification-needed-focal-linux-bluefield
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.15.0-1046.53 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure' to 'verification-done-jammy-linux-azure'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure' to 'verification-failed-jammy-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-azure-v2 verification-needed-jammy-linux-azure
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-aws-5.15/5.15.0-1046.51~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-done-focal-linux-aws-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-failed-focal-linux-aws-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-aws-5.15-v2 verification-needed-focal-linux-aws-5.15
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-xilinx-zynqmp/5.15.0-1024.28 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-done-jammy-linux-xilinx-zynqmp'. If the problem still exists, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-failed-jammy-linux-xilinx-zynqmp'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-xilinx-zynqmp-v2 verification-needed-jammy-linux-xilinx-zynqmp
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.