[Azure] PCI: hv: Do not set PCI_COMMAND_MEMORY to reduce VM boot time

Bug #1972662 reported by Tim Gardner
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-azure (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Medium
Tim Gardner
Impish
Fix Released
Medium
Tim Gardner
Jammy
Fix Released
Medium
Tim Gardner

Bug Description

SRU Justification

[Impact]

A VM on Azure can have 14 GPUs, and each GPU may have a huge MMIO BAR,
e.g. 128 GB. Currently the boot time of such a VM can be 4+ minutes, and
most of the time is used by the host to unmap/map the vBAR from/to pBAR
when the VM clears and sets the PCI_COMMAND_MEMORY bit: each unmap/map
operation for a 128GB BAR needs about 1.8 seconds, and the pci-hyperv
driver and the Linux PCI subsystem flip the PCI_COMMAND_MEMORY bit
eight times (see pci_setup_device() -> pci_read_bases() and
pci_std_update_resource()), increasing the boot time by 1.8 * 8 = 14.4
seconds per GPU, i.e. 14.4 * 14 = 201.6 seconds in total.

Fix the slowness by not turning on the PCI_COMMAND_MEMORY in pci-hyperv.c,
so the bit stays in the off state before the PCI device driver calls
pci_enable_device(): when the bit is off, pci_read_bases() and
pci_std_update_resource() don't cause Hyper-V to unmap/map the vBARs.
With this change, the boot time of such a VM is reduced by
1.8 * (8-1) * 14 = 176.4 seconds.

[Test Case]

Microsoft tested

[Where things could go wrong]

PCI BAR setup could fail or be incorrect.

[Other Info]

SF: #00336342

Tim Gardner (timg-tpi)
affects: linux (Ubuntu) → linux-azure (Ubuntu)
Changed in linux-azure (Ubuntu):
status: New → Fix Released
Changed in linux-azure (Ubuntu Jammy):
assignee: nobody → Tim Gardner (timg-tpi)
importance: Undecided → Medium
status: New → In Progress
Changed in linux-azure (Ubuntu Impish):
assignee: nobody → Tim Gardner (timg-tpi)
importance: Undecided → Medium
status: New → In Progress
Tim Gardner (timg-tpi)
Changed in linux-azure (Ubuntu Focal):
assignee: nobody → Tim Gardner (timg-tpi)
importance: Undecided → Medium
status: New → In Progress
Tim Gardner (timg-tpi)
Changed in linux-azure (Ubuntu Focal):
status: In Progress → Fix Committed
Changed in linux-azure (Ubuntu Impish):
status: In Progress → Fix Committed
Changed in linux-azure (Ubuntu Jammy):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.15.0-1006.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-jammy
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.13.0-1024.28 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-impish
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.4.0-1081.84 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Tim Gardner (timg-tpi) wrote :

Microsoft tested. Marking verification done.

tags: added: verification-done-focal verification-done-impish verification-done-jammy
removed: verification-needed-focal verification-needed-impish verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (12.2 KiB)

This bug was fixed in the package linux-azure - 5.13.0-1026.30

---------------
linux-azure (5.13.0-1026.30) impish; urgency=medium

  * impish/linux-azure: 5.13.0-1026.30 -proposed tracker (LP: #1974334)

  * [Azure] WARNING: CPU: 0 PID: 499 at include/linux/dma-mapping.h:555
    netvsc_probe+0x3c9/0x3e0 (LP: #1975717)
    - Drivers: hv: vmbus: Rework use of DMA_BIT_MASK(64)
    - Drivers: hv: vmbus: Fix initialization of device object in
      vmbus_device_register()

  * [Azure] hv_netvsc: Add support for XDP_REDIRECT (LP: #1972832)
    - hv_netvsc: Add comment of netvsc_xdp_xmit()
    - hv_netvsc: Add support for XDP_REDIRECT

  * linux-azure: Patch Set for ARM64 Images 20.04 and 18.04 (LP: #1970468)
    - Drivers: hv: vmbus: Replace smp_store_mb() with virt_store_mb()
    - Drivers: hv: balloon: Support status report for larger page sizes
    - Drivers: hv: balloon: Disable balloon and hot-add accordingly

  * [Azure] PCI: hv: Do not set PCI_COMMAND_MEMORY to reduce VM boot time
    (LP: #1972662)
    - PCI: hv: Do not set PCI_COMMAND_MEMORY to reduce VM boot time

  * config CONFIG_HISI_PMU for kunpeng920 (LP: #1956086)
    - [Config] azure: CONFIG_HISI_PMU=m

  [ Ubuntu: 5.13.0-46.51 ]

  * CVE-2022-21499
    - SAUCE: debug: Lock down kgdb

  [ Ubuntu: 5.13.0-45.50 ]

  * impish/linux: 5.13.0-45.50 -proposed tracker (LP: #1974347)
  * CVE-2022-1158
    - KVM: x86/mmu: do compare-and-exchange of gPTE via the user address
  * config CONFIG_HISI_PMU for kunpeng920 (LP: #1956086)
    - [Config] CONFIG_HISI_PMU=m
  * re-apply missing overlayfs SAUCE patch (LP: #1967924)
    - SAUCE: overlayfs: fix incorrect mnt_id of files opened from map_files
  * Impish update: upstream stable patchset 2022-04-20 (LP: #1969666)
    - mac80211_hwsim: report NOACK frames in tx_status
    - mac80211_hwsim: initialize ieee80211_tx_info at hw_scan_work
    - i2c: bcm2835: Avoid clock stretching timeouts
    - ASoC: rt5668: do not block workqueue if card is unbound
    - ASoC: rt5682: do not block workqueue if card is unbound
    - regulator: core: fix false positive in regulator_late_cleanup()
    - KVM: arm64: vgic: Read HW interrupt pending state from the HW
    - tipc: fix a bit overflow in tipc_crypto_key_rcv()
    - cifs: fix double free race when mount fails in cifs_get_root()
    - selftests/seccomp: Fix seccomp failure by adding missing headers
    - i2c: cadence: allow COMPILE_TEST
    - i2c: qup: allow COMPILE_TEST
    - net: usb: cdc_mbim: avoid altsetting toggling for Telit FN990
    - usb: gadget: don't release an existing dev->buf
    - usb: gadget: clear related members when goto fail
    - exfat: reuse exfat_inode_info variable instead of calling EXFAT_I()
    - exfat: fix i_blocks for files truncated over 4 GiB
    - tracing: Add test for user space strings when filtering on string pointers
    - serial: stm32: prevent TDR register overwrite when sending x_char
    - ata: pata_hpt37x: fix PCI clock detection
    - drm/amdgpu: check vm ready by amdgpu_vm->evicting flag
    - tracing: Add ustring operation to filtering string pointers
    - ALSA: intel_hdmi: Fix reference to PCM buffer address
    - riscv/efi_stub: Fix get_boot_hartid_from_fdt...

Changed in linux-azure (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (105.5 KiB)

This bug was fixed in the package linux-azure - 5.15.0-1008.9

---------------
linux-azure (5.15.0-1008.9) jammy; urgency=medium

  * jammy/linux-azure: 5.15.0-1008.9 -proposed tracker (LP: #1974294)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.04.18)

  * [Azure] WARNING: CPU: 0 PID: 499 at include/linux/dma-mapping.h:555
    netvsc_probe+0x3c9/0x3e0 (LP: #1975717)
    - Drivers: hv: vmbus: Rework use of DMA_BIT_MASK(64)
    - Drivers: hv: vmbus: Fix initialization of device object in
      vmbus_device_register()

  * config CONFIG_HISI_PMU for kunpeng920 (LP: #1956086)
    - [Config] azure: CONFIG_HISI_PMU=m

  * linux: CONFIG_SERIAL_8250_MID=y (LP: #1967338)
    - [Config] azure: CONFIG_SERIAL_8250_MID=y

  * Support AMD P-State cpufreq control mechanism (LP: #1956509) // Enable
    speakup kernel modules to allow the speakup screen reader to function
    (LP: #1967702)
    - [Config] azure: Update configs after rebase

  * Azure: swiotlb patch needed for CVM (LP: #1971701) // [Azure][CVM] Fix
    swiotlb_max_mapping_size() for potential bounce buffer allocation failure in
    storvsc (LP: #1973169)
    - SAUCE: swiotlb: Max mapping size takes min align mask into account

  * Azure: swiotlb patch needed for CVM (LP: #1971701)
    - SAUCE: treewide: Replace the use of mem_encrypt_active() with
      cc_platform_has()
    - SAUCE: swiotlb: use bitmap to track free slots
    - SAUCE: swiotlb: allocate memory in a cache-friendly way
    - SAUCE: swiotlb: Split up single swiotlb lock

  * jammy/linux-azure: Update cifs to 5.15 backport (LP: #1970977)
    - improve error message when mount options conflict with posix
    - cifs: call cifs_reconnect when a connection is marked
    - cifs: call helper functions for marking channels for reconnect
    - cifs: mark sessions for reconnection in helper function
    - treewide: Replace zero-length arrays with flexible-array members
    - smb3: fix incorrect session setup check for multiuser mounts
    - cifs: truncate the inode and mapping when we simulate fcollapse
    - cifs: use a different reconnect helper for non-cifsd threads
    - cifs: do not skip link targets when an I/O fails
    - cifs: convert the path to utf16 in smb2_query_info_compound
    - cifs: change smb2_query_info_compound to use a cached fid, if available
    - cifs: fix bad fids sent over wire
    - cifs: fix incorrect use of list iterator after the loop
    - move more common protocol header definitions to smbfs_common
    - smb3: move defines for ioctl protocol header and SMB2 sizes to smbfs_common
    - smb3: move defines for query info and query fsinfo to smbfs_common
    - smb3: cleanup and clarify status of tree connections
    - smb3: fix ksmbd bigendian bug in oplock break, and move its struct to
      smbfs_common
    - fs: Remove ->readpages address space operation
    - cifs: fix potential race with cifsd thread
    - cifs: remove check of list iterator against head past the loop body
    - cifs: force new session setup and tcon for dfs
    - cifs: update internal module number
    - cifs: Check the IOCB_DIRECT flag, not O_DIRECT
    - cifs: Split the smb3_a...

Changed in linux-azure (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-azure - 5.4.0-1083.87

---------------
linux-azure (5.4.0-1083.87) focal; urgency=medium

  [ Ubuntu: 5.4.0-117.132 ]

  * CVE-2022-1966
    - netfilter: nf_tables: add nft_set_elem_expr_alloc()
    - netfilter: nf_tables: disallow non-stateful expression in sets earlier

 -- Thadeu Lima de Souza Cascardo <email address hidden> Wed, 01 Jun 2022 21:52:55 -0300

Changed in linux-azure (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.