| 2023-04-13 21:20:43 |
Jared Ledvina (Datadog) |
bug |
|
|
added bug |
| 2023-04-13 21:22:10 |
Jared Ledvina (Datadog) |
description |
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15 linux-aws package to the recently pusblished v5.19 package and rebooted. It seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect the actual disk usage. No other system configuration changes were applied just the kernel upgrade and reboot. I've also verified that simply rebooting a v5.15 where this does work doesn't break the reporting. These EC2 instances are running with cgroups v1 due to other compatability issues and I suspect that might be the issue. So far, I cannot find any differences. mtab shows the same v1 mount setup, the kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd love to get 5.19 working for us but, we really need the data from these files. |
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15 linux-aws package to the recently pusblished v5.19 package and rebooted. It seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect the actual disk usage. No other system configuration changes were applied just the kernel upgrade and reboot. I've also verified that simply rebooting a v5.15 where this does work doesn't break the reporting. These EC2 instances are running with cgroups v1 due to other compatability issues and I suspect that might be the issue. So far, I cannot find any differences. mtab shows the same v1 mount setup, the kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd love to get 5.19 working for us but, we really need the data from these files.
Info:
Prior version that works: Linux ip-10-128-168-154 5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Upgraded version that's broken: Linux ip-10-128-166-219 5.19.0-1022-aws #23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
EC2 instances built off of the published 22.04 LTS AMI in us-east-1. |
|
| 2023-04-26 18:20:06 |
Jared Ledvina (Datadog) |
bug task added |
|
linux-gcp (Ubuntu) |
|
| 2023-04-26 18:20:13 |
Jared Ledvina (Datadog) |
bug task added |
|
linux-azure (Ubuntu) |
|
| 2023-05-09 04:33:01 |
Andrea Righi |
bug task added |
|
linux (Ubuntu) |
|
| 2023-05-09 04:33:14 |
Andrea Righi |
bug task deleted |
linux-aws (Ubuntu) |
|
|
| 2023-05-09 04:33:19 |
Andrea Righi |
bug task deleted |
linux-azure (Ubuntu) |
|
|
| 2023-05-09 04:33:23 |
Andrea Righi |
bug task deleted |
linux-gcp (Ubuntu) |
|
|
| 2023-05-09 04:33:57 |
Andrea Righi |
nominated for series |
|
Ubuntu Kinetic |
|
| 2023-05-09 04:33:57 |
Andrea Righi |
bug task added |
|
linux (Ubuntu Kinetic) |
|
| 2023-05-09 04:33:57 |
Andrea Righi |
nominated for series |
|
Ubuntu Mantic |
|
| 2023-05-09 04:33:57 |
Andrea Righi |
bug task added |
|
linux (Ubuntu Mantic) |
|
| 2023-05-09 04:33:57 |
Andrea Righi |
nominated for series |
|
Ubuntu Lunar |
|
| 2023-05-09 04:33:57 |
Andrea Righi |
bug task added |
|
linux (Ubuntu Lunar) |
|
| 2023-05-09 04:53:38 |
Andrea Righi |
description |
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15 linux-aws package to the recently pusblished v5.19 package and rebooted. It seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect the actual disk usage. No other system configuration changes were applied just the kernel upgrade and reboot. I've also verified that simply rebooting a v5.15 where this does work doesn't break the reporting. These EC2 instances are running with cgroups v1 due to other compatability issues and I suspect that might be the issue. So far, I cannot find any differences. mtab shows the same v1 mount setup, the kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd love to get 5.19 working for us but, we really need the data from these files.
Info:
Prior version that works: Linux ip-10-128-168-154 5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Upgraded version that's broken: Linux ip-10-128-166-219 5.19.0-1022-aws #23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
EC2 instances built off of the published 22.04 LTS AMI in us-east-1. |
[Impact]
Commit f382fb0bcef4 ("block: remove legacy IO schedulers") introduced a behavior change in the blkio throttle cgroup subsystem: IO statistics are not reported anymore unless a throttling rule is explicitly defined, because the current code only counts bios that are actually throttled.
This behavior change is breaking the user-space (see original bug report below).
[Test case]
- mount cgroup v1
- create a blkio cgroup
- move a task into the blkio cgroup
- perform some I/O (i.e., dd)
- read the IO stats for the cgroup (blkio.throttle.io_serviced and blkio.throttle.io_service_bytes in cgroupfs)
- IO stats are all 0, unless a throttle rule is defined
Previous behavior (kernel 5.15) was showing I/O statistics even without throttling rules defined.
[Fix]
Apply / backport this fix:
https://lore.kernel.org/lkml/20230507170631.89607-1-hanjinke.666@bytedance.com/t/
[Regression potential]
The fix is affecting the block IO cgroup subsystem, we may see potential regressions in this particular cgroup subsystem with this fix applied.
[Original bug report]
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15 linux-aws package to the recently pusblished v5.19 package and rebooted. It seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect the actual disk usage. No other system configuration changes were applied just the kernel upgrade and reboot. I've also verified that simply rebooting a v5.15 where this does work doesn't break the reporting. These EC2 instances are running with cgroups v1 due to other compatability issues and I suspect that might be the issue. So far, I cannot find any differences. mtab shows the same v1 mount setup, the kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd love to get 5.19 working for us but, we really need the data from these files.
Info:
Prior version that works: Linux ip-10-128-168-154 5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Upgraded version that's broken: Linux ip-10-128-166-219 5.19.0-1022-aws #23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
EC2 instances built off of the published 22.04 LTS AMI in us-east-1. |
|
| 2023-05-09 05:00:06 |
Ubuntu Kernel Bot |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2023-05-09 05:00:06 |
Ubuntu Kernel Bot |
linux (Ubuntu Kinetic): status |
New |
Incomplete |
|
| 2023-05-09 05:00:07 |
Ubuntu Kernel Bot |
linux (Ubuntu Lunar): status |
New |
Incomplete |
|
| 2023-05-09 05:16:38 |
Andrea Righi |
description |
[Impact]
Commit f382fb0bcef4 ("block: remove legacy IO schedulers") introduced a behavior change in the blkio throttle cgroup subsystem: IO statistics are not reported anymore unless a throttling rule is explicitly defined, because the current code only counts bios that are actually throttled.
This behavior change is breaking the user-space (see original bug report below).
[Test case]
- mount cgroup v1
- create a blkio cgroup
- move a task into the blkio cgroup
- perform some I/O (i.e., dd)
- read the IO stats for the cgroup (blkio.throttle.io_serviced and blkio.throttle.io_service_bytes in cgroupfs)
- IO stats are all 0, unless a throttle rule is defined
Previous behavior (kernel 5.15) was showing I/O statistics even without throttling rules defined.
[Fix]
Apply / backport this fix:
https://lore.kernel.org/lkml/20230507170631.89607-1-hanjinke.666@bytedance.com/t/
[Regression potential]
The fix is affecting the block IO cgroup subsystem, we may see potential regressions in this particular cgroup subsystem with this fix applied.
[Original bug report]
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15 linux-aws package to the recently pusblished v5.19 package and rebooted. It seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect the actual disk usage. No other system configuration changes were applied just the kernel upgrade and reboot. I've also verified that simply rebooting a v5.15 where this does work doesn't break the reporting. These EC2 instances are running with cgroups v1 due to other compatability issues and I suspect that might be the issue. So far, I cannot find any differences. mtab shows the same v1 mount setup, the kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd love to get 5.19 working for us but, we really need the data from these files.
Info:
Prior version that works: Linux ip-10-128-168-154 5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Upgraded version that's broken: Linux ip-10-128-166-219 5.19.0-1022-aws #23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
EC2 instances built off of the published 22.04 LTS AMI in us-east-1. |
[Impact]
Commit f382fb0bcef4 ("block: remove legacy IO schedulers") introduced a behavior change in the blkio throttle cgroup subsystem: IO statistics are not reported anymore unless a throttling rule is explicitly defined, because the current code only counts bios that are actually throttled.
This behavior change is potentially breaking some user-space
applications that are relying on the old behavior (see original bug
report below).
[Test case]
- mount cgroup v1
- create a blkio cgroup
- move a task into the blkio cgroup
- perform some I/O (i.e., dd)
- read the IO stats for the cgroup (blkio.throttle.io_serviced and blkio.throttle.io_service_bytes in cgroupfs)
- IO stats are all 0, unless a throttle rule is defined
Previous behavior (kernel 5.15) was showing I/O statistics even without throttling rules defined.
[Fix]
Apply / backport this fix:
https://lore.kernel.org/lkml/20230507170631.89607-1-hanjinke.666@bytedance.com/t/
[Regression potential]
The fix is affecting the block IO cgroup subsystem, we may see potential regressions in this particular cgroup subsystem with this fix applied.
[Original bug report]
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15 linux-aws package to the recently pusblished v5.19 package and rebooted. It seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect the actual disk usage. No other system configuration changes were applied just the kernel upgrade and reboot. I've also verified that simply rebooting a v5.15 where this does work doesn't break the reporting. These EC2 instances are running with cgroups v1 due to other compatability issues and I suspect that might be the issue. So far, I cannot find any differences. mtab shows the same v1 mount setup, the kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd love to get 5.19 working for us but, we really need the data from these files.
Info:
Prior version that works: Linux ip-10-128-168-154 5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Upgraded version that's broken: Linux ip-10-128-166-219 5.19.0-1022-aws #23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
EC2 instances built off of the published 22.04 LTS AMI in us-east-1. |
|
| 2023-05-11 15:35:53 |
Luke Nowakowski-Krijger |
linux (Ubuntu Kinetic): status |
Incomplete |
Fix Committed |
|
| 2023-05-11 15:35:55 |
Luke Nowakowski-Krijger |
linux (Ubuntu Lunar): status |
Incomplete |
Fix Committed |
|
| 2023-05-17 22:01:04 |
Ubuntu Kernel Bot |
tags |
|
kernel-spammed-kinetic-linux verification-needed-kinetic |
|
| 2023-06-06 16:34:26 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-kinetic-linux verification-needed-kinetic |
kernel-spammed-jammy-linux-nvidia-5.19 kernel-spammed-kinetic-linux verification-needed-jammy verification-needed-kinetic |
|
| 2023-06-08 18:41:31 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-nvidia-5.19 kernel-spammed-kinetic-linux verification-needed-jammy verification-needed-kinetic |
kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-5.19 kernel-spammed-kinetic-linux verification-needed-jammy verification-needed-kinetic |
|
| 2023-06-09 20:55:36 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-5.19 kernel-spammed-kinetic-linux verification-needed-jammy verification-needed-kinetic |
kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-5.19 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-kinetic-linux verification-needed-jammy verification-needed-kinetic |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
linux (Ubuntu Kinetic): status |
Fix Committed |
Fix Released |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2022-4269 |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2023-1380 |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2023-1670 |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2023-1859 |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2023-2612 |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2023-30456 |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2023-31436 |
|
| 2023-06-15 22:20:36 |
Launchpad Janitor |
cve linked |
|
2023-32233 |
|
| 2023-06-15 22:21:51 |
Launchpad Janitor |
linux (Ubuntu Lunar): status |
Fix Committed |
Fix Released |
|
| 2023-07-19 13:14:43 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-5.19 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-kinetic-linux verification-needed-jammy verification-needed-kinetic |
kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-5.19 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-kinetic-linux kernel-spammed-lunar-linux-azure verification-needed-jammy verification-needed-kinetic verification-needed-lunar |
|
