5.19 not reporting cgroups v1 blkio.throttle.io_serviced

A few clarifications from IRC:
1. We run all of our Ubuntu 22.04 LTS nodes with the kernel args 'systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=true' to force cgroups v1 as, unfortunately, we cannot safely turn on cgroups v2 yet (that's another pile of work I want to do!).
2. If you install 'linux-modules-extra-aws', 'modprobe bfq', and then 'echo bfq > /sys/block/nvme0n1/queue/scheduler' you will see stats in the '/sys/fs/cgroup/blkio/blkio.bfq.io_service*' files.
3. However, we continue to only see 0's in the '/sys/fs/cgroup/blkio/blkio.throttle.io_service*' files.

Potentially an upstream change but, definitely something that breaks with the '5.19.0.1022.23~22.04.6' Jammy package update. For me, this likely means I need to pin everything to the older 5.15 package pending cgroups v2 working or a fix to this. Obviously I'd prefer having this fixed so that we can get to 5.19 and stick w/ cgroups v1. I'd also offer a note that pushing 5.19 to Jammy without this support feels like a breaking change. I'm more worried that _other_ cgroups v1 controllers aren't working in a way I haven't noticed yet. Anyway, thanks so much for the help so far and gimme a holler if I can test/confirm anything else!

It might be worth to report this also upstream, apparently there was a behavior change with the blkio controller in cgroup1 that happend at some point between 5.15 and 5.19. I'll do more tests and will let you know if I find anything relevant. Thanks for reporting this!

I've updated this to report the issue for linux-azure and linux-gcp as their jammy-updates repo's have recently updated to kernel 5.19 and appear to be affected as well. I could use some guidance on reporting this upstream as 5.19 doesn't seem to be a supported kernel version (looking https://www.kernel.org/) so it's unclear the correct way to go about that.

Related, https://packages.ubuntu.com/jammy-updates/linux-gcp-lts-22.04 doesn't exist as of right now. If that could be published similar to https://packages.ubuntu.com/jammy-updates/linux-azure-lts-22.04 and https://packages.ubuntu.com/jammy-updates/linux-aws-lts-22.04 that'd be a huge help for me.

Potential upstream fix: https://<email address hidden>/

However this seems to partially restore the old behavior of cgroup v1, because we still need to set io throttling limits in order to get the io statistics.

We may need an additional fix like this to completely restore the old behavior: https://lore.kernel.org/lkml/ZEwY5Oo+5inO9UFf@righiandr-XPS-13-7390/

I'll follow the upstream thread, if we come up with a reasonable fix I'll take care of preparing a proper SRU for it.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2016186

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

This bug is awaiting verification that the linux/5.19.0-44.45 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hey Andrea,
Thanks for the help getting this all fixed up. I see that the change is committed for Lunar and Kinetic.

Is there a good way for me to follow when this'll land for the Ubuntu Jammy linux-aws, linux-gcp, and linux-azure packages?

This bug is awaiting verification that the linux-nvidia-5.19/5.19.0-1014.14 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-hwe-6.2/6.2.0-23.23~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-6.2/6.2.0-1003.3~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi Jared you can follow the release of all of the Jammy cloud kernel here https://kernel.ubuntu.com/sru/dashboards/web/kernel-stable-board.html , where this fix is included in the 2023.05.15 cycle. They should be released to updates in the next week.

- Luke

This bug was fixed in the package linux - 5.19.0-45.46

---------------
linux (5.19.0-45.46) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-45.46 -proposed tracker (LP: #2023057)

  * Kinetic update: upstream stable patchset 2023-05-23 (LP: #2020599)
    - wifi: cfg80211: Partial revert "wifi: cfg80211: Fix use after free for wext"

linux (5.19.0-44.45) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-44.45 -proposed tracker (LP: #2019827)

  * Linux 5.19 amdgpu: NULL pointer on GCN2 and invalid load on GCN1
    (LP: #2018470)
    - drm/amdgpu: Fix for BO move issue

  * CVE-2023-32233
    - netfilter: nf_tables: deactivate anonymous set from preparation phase

  * CVE-2023-2612
    - SAUCE: shiftfs: prevent lock unbalance in shiftfs_create_object()

  * CVE-2023-31436
    - net: sched: sch_qfq: prevent slab-out-of-bounds in qfq_activate_agg

  * CVE-2023-1380
    - wifi: brcmfmac: slab-out-of-bounds read in brcmf_get_assoc_ies()

  * conntrack mark is not advertised via netlink (LP: #2016269)
    - netfilter: ctnetlink: revert to dumping mark regardless of event type

  * 5.19 not reporting cgroups v1 blkio.throttle.io_serviced (LP: #2016186)
    - SAUCE: blk-throttle: Fix io statistics for cgroup v1

  * [SRU] Backport request for hpwdt from upstream 6.1 to Jammy (LP: #2008751)
    - watchdog/hpwdt: Enable HP_WATCHDOG for ARM64 systems.
    - watchdog/hpwdt: Include nmi.h only if CONFIG_HPWDT_NMI_DECODING
    - [Config] Add arm64 option to CONFIG_HP_WATCHDOG

  * vmwgfx fails to reserve graphics buffer on aarch64 leading to blank display
    (LP: #2007001)
    - SAUCE: Revert "video/aperture: Disable and unregister sysfb devices via
      aperture helpers"

  * Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
    (LP: #2012335)
    - ice: Allow operation with reduced device MSI-X

  * Dell: Enable speaker mute hotkey LED indicator (LP: #2015972)
    - platform/x86: dell-laptop: Register ctl-led for speaker-mute

  * [SRU]With "Performance per Watt (DAPC)" enabled in the BIOS, Bootup time is
    taking longer than expected (LP: #2008527)
    - cpufreq: ACPI: Defer setting boost MSRs

  * [SRU][Jammy] CONFIG_PCI_MESON is not enabled (LP: #2007745)
    - [Config] arm64: Enable PCI_MESON module

  * Kinetic update: upstream stable patchset 2023-05-08 (LP: #2018948)
    - HID: asus: use spinlock to protect concurrent accesses
    - HID: asus: use spinlock to safely schedule workers
    - powerpc/mm: Rearrange if-else block to avoid clang warning
    - ARM: OMAP2+: Fix memory leak in realtime_counter_init()
    - arm64: dts: qcom: qcs404: use symbol names for PCIe resets
    - arm64: dts: qcom: msm8996-tone: Fix USB taking 6 minutes to wake up
    - arm64: dts: qcom: sm8150-kumano: Panel framebuffer is 2.5k instead of 4k
    - arm64: dts: qcom: sm6125: Reorder HSUSB PHY clocks to match bindings
    - arm64: dts: imx8m: Align SoC unique ID node unit address
    - ARM: zynq: Fix refcount leak in zynq_early_slcr_init
    - arm64: dts: mediatek: mt8183: Fix systimer 13 MHz clock description
    - arm64: dts: qcom: sdm845-db845c: fix audio codec interrupt pin name
    - arm64: dts: qcom: sc7180: correct SPMI bus addres...

This bug was fixed in the package linux - 6.2.0-23.23

---------------
linux (6.2.0-23.23) lunar; urgency=medium

  * lunar/linux: 6.2.0-23.23 -proposed tracker (LP: #2019845)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
    - debian/dkms-versions -- update from kernel-versions (main/2023.05.15)

  * Fix flicker display problem on some panels which support PSR2 (LP: #2002968)
    - drm/i915/psr: Add continuous full frame bit together with single

  * Kernel 6.1 bumped the disk consumption on default images by 15%
    (LP: #2015867)
    - [Packaging] introduce a separate linux-lib-rust package

  * Update I915 PSR calculation on Linux 6.2 (LP: #2018655)
    - drm/i915: Fix fast wake AUX sync len
    - drm/i915: Explain the magic numbers for AUX SYNC/precharge length

  * Computer with Intel Atom CPU will not boot with Kernel 6.2.0-20
    (LP: #2017444)
    - [Config]: Disable CONFIG_INTEL_ATOMISP

  * udev fails to make prctl() syscall with apparmor=0 (as used by maas by
    default) (LP: #2016908)
    - SAUCE: (no-up) Stacking v38: Fix prctl() syscall with apparmor=0

  * CVE-2023-32233
    - netfilter: nf_tables: deactivate anonymous set from preparation phase

  * CVE-2023-2612
    - SAUCE: shiftfs: prevent lock unbalance in shiftfs_create_object()

  * CVE-2023-31436
    - net: sched: sch_qfq: prevent slab-out-of-bounds in qfq_activate_agg

  * CVE-2023-1380
    - wifi: brcmfmac: slab-out-of-bounds read in brcmf_get_assoc_ies()

  * 5.19 not reporting cgroups v1 blkio.throttle.io_serviced (LP: #2016186)
    - SAUCE: blk-throttle: Fix io statistics for cgroup v1

  * LSM stacking and AppArmor for 6.2: additional fixes (LP: #2017903)
    - SAUCE: (no-up) apparmor: fix policy_compat perms remap for file dfa
    - SAUCE: (no-up) apparmor: fix profile verification and enable it
    - SAUCE: (no-up) apparmor: fix: add missing failure check in
      compute_xmatch_perms
    - SAUCE: (no-up) apparmor: fix: kzalloc perms tables for shared dfas

  * Lunar update: v6.2.12 upstream stable release (LP: #2017219)
    - Revert "pinctrl: amd: Disable and mask interrupts on resume"
    - drm/amd/display: Pass the right info to drm_dp_remove_payload
    - drm/i915: Workaround ICL CSC_MODE sticky arming
    - ALSA: emu10k1: fix capture interrupt handler unlinking
    - ALSA: hda/sigmatel: add pin overrides for Intel DP45SG motherboard
    - ALSA: i2c/cs8427: fix iec958 mixer control deactivation
    - ALSA: hda: patch_realtek: add quirk for Asus N7601ZM
    - ALSA: hda/realtek: Add quirks for Lenovo Z13/Z16 Gen2
    - ALSA: firewire-tascam: add missing unwind goto in
      snd_tscm_stream_start_duplex()
    - ALSA: emu10k1: don't create old pass-through playback device on Audigy
    - ALSA: hda/sigmatel: fix S/PDIF out on Intel D*45* motherboards
    - ALSA: hda/hdmi: disable KAE for Intel DG2
    - Bluetooth: L2CAP: Fix use-after-free in l2cap_disconnect_{req,rsp}
    - Bluetooth: Fix race condition in hidp_session_thread
    - bluetooth: btbcm: Fix logic error in forming the board name.
    - Bluetooth: Free potentially unfreed SCO connection
    - Bluetooth: hci_conn: Fix possible UAF
    - btrfs: restore the thread_pool=...

This bug is awaiting verification that the linux-azure/6.2.0-1009.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!