Activity log for bug #2030709

Date Who What changed Old value New value Message
2023-08-08 06:01:33 Po-Hsu Lin bug added bug
2023-08-08 08:54:58 Po-Hsu Lin tags 5.15 gkeop jammy ubuntu-ltp-controllers 5.15 gkeop jammy sru-20230710 ubuntu-ltp-controllers
2023-08-08 09:00:24 Po-Hsu Lin summary memcg_regression_test in ubuntu_ltp_controllers cause softlockup on Google g1-small with J-gkeop memcg_regression_test in ubuntu_ltp_controllers cause soft lockup on Google g1-small with J-gkeop
2023-08-08 10:01:10 Po-Hsu Lin description Issue found with 5.15.0-1025.30 and can be reproduced with 5.15.0-1024-gkeop on google instance g1-small only. The test and the system will hang with 4th test case inside this test. Test output: COMMAND: /opt/ltp/bin/ltp-pan -e -S -a 1345 -n 1345 -p -f /tmp/ltp-8BAdmWLWz8/alltests -l /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log -C /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed -T /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf LOG File: /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log FAILED COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed TCONF COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf Running tests....... <<<test_start>>> tag=memcg_regression stime=1691472909 cmdline="memcg_regression_test.sh" contacts="" analysis=exit <<<test_output>>> incrementing stop memcg_regression_test 1 TINFO: timeout per run is 0h 5m 0s memcg_regression_test 1 TINFO: test starts with cgroup version 2 memcg_regression_test 1 TPASS: no kernel bug was found memcg_regression_test 2 TCONF: Cgroup v2 found, skipping test memcg_regression_test 3 TPASS: no kernel bug was found dmesg output: [ 296.923589] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [memcg_test_4.sh:1983] [ 324.923599] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [memcg_test_4.sh:1983] [ 352.923640] watchdog: BUG: soft lockup - CPU#0 stuck for 78s! [memcg_test_4.sh:1983] [ 380.923622] watchdog: BUG: soft lockup - CPU#0 stuck for 104s! [memcg_test_4.sh:1983] [ 408.923634] watchdog: BUG: soft lockup - CPU#0 stuck for 130s! [memcg_test_4.sh:1983] [ 436.923645] watchdog: BUG: soft lockup - CPU#0 stuck for 156s! [memcg_test_4.sh:1983] Issue found with 5.15.0-1025.30 and can be reproduced with 5.15.0-1024-gkeop on google instance g1-small only. There is no such issue in J-gcp and J-gke. The test and the system will hang with 4th test case inside this test. Test output: COMMAND: /opt/ltp/bin/ltp-pan -e -S -a 1345 -n 1345 -p -f /tmp/ltp-8BAdmWLWz8/alltests -l /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log -C /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed -T /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf LOG File: /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log FAILED COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed TCONF COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf Running tests....... <<<test_start>>> tag=memcg_regression stime=1691472909 cmdline="memcg_regression_test.sh" contacts="" analysis=exit <<<test_output>>> incrementing stop memcg_regression_test 1 TINFO: timeout per run is 0h 5m 0s memcg_regression_test 1 TINFO: test starts with cgroup version 2 memcg_regression_test 1 TPASS: no kernel bug was found memcg_regression_test 2 TCONF: Cgroup v2 found, skipping test memcg_regression_test 3 TPASS: no kernel bug was found dmesg output: [ 296.923589] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [memcg_test_4.sh:1983] [ 324.923599] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [memcg_test_4.sh:1983] [ 352.923640] watchdog: BUG: soft lockup - CPU#0 stuck for 78s! [memcg_test_4.sh:1983] [ 380.923622] watchdog: BUG: soft lockup - CPU#0 stuck for 104s! [memcg_test_4.sh:1983] [ 408.923634] watchdog: BUG: soft lockup - CPU#0 stuck for 130s! [memcg_test_4.sh:1983] [ 436.923645] watchdog: BUG: soft lockup - CPU#0 stuck for 156s! [memcg_test_4.sh:1983]
2023-08-08 10:19:15 Po-Hsu Lin description Issue found with 5.15.0-1025.30 and can be reproduced with 5.15.0-1024-gkeop on google instance g1-small only. There is no such issue in J-gcp and J-gke. The test and the system will hang with 4th test case inside this test. Test output: COMMAND: /opt/ltp/bin/ltp-pan -e -S -a 1345 -n 1345 -p -f /tmp/ltp-8BAdmWLWz8/alltests -l /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log -C /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed -T /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf LOG File: /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log FAILED COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed TCONF COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf Running tests....... <<<test_start>>> tag=memcg_regression stime=1691472909 cmdline="memcg_regression_test.sh" contacts="" analysis=exit <<<test_output>>> incrementing stop memcg_regression_test 1 TINFO: timeout per run is 0h 5m 0s memcg_regression_test 1 TINFO: test starts with cgroup version 2 memcg_regression_test 1 TPASS: no kernel bug was found memcg_regression_test 2 TCONF: Cgroup v2 found, skipping test memcg_regression_test 3 TPASS: no kernel bug was found dmesg output: [ 296.923589] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [memcg_test_4.sh:1983] [ 324.923599] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [memcg_test_4.sh:1983] [ 352.923640] watchdog: BUG: soft lockup - CPU#0 stuck for 78s! [memcg_test_4.sh:1983] [ 380.923622] watchdog: BUG: soft lockup - CPU#0 stuck for 104s! [memcg_test_4.sh:1983] [ 408.923634] watchdog: BUG: soft lockup - CPU#0 stuck for 130s! [memcg_test_4.sh:1983] [ 436.923645] watchdog: BUG: soft lockup - CPU#0 stuck for 156s! [memcg_test_4.sh:1983] Issue found with 5.15.0-1025.30 and can be reproduced with 5.15.0-1024-gkeop on google instance g1-small only. There is no such issue in J-gcp and J-gke. The test and the system will hang with 4th test case inside this test. Test output: COMMAND: /opt/ltp/bin/ltp-pan -e -S -a 1345 -n 1345 -p -f /tmp/ltp-8BAdmWLWz8/alltests -l /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log -C /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed -T /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf LOG File: /opt/ltp/results/LTP_RUN_ON-2023_08_08-05h_35m_08s.log FAILED COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.failed TCONF COMMAND File: /opt/ltp/output/LTP_RUN_ON-2023_08_08-05h_35m_08s.tconf Running tests....... <<<test_start>>> tag=memcg_regression stime=1691472909 cmdline="memcg_regression_test.sh" contacts="" analysis=exit <<<test_output>>> incrementing stop memcg_regression_test 1 TINFO: timeout per run is 0h 5m 0s memcg_regression_test 1 TINFO: test starts with cgroup version 2 memcg_regression_test 1 TPASS: no kernel bug was found memcg_regression_test 2 TCONF: Cgroup v2 found, skipping test memcg_regression_test 3 TPASS: no kernel bug was found dmesg output from console (ssh session can't get this far): [ 296.923589] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [memcg_test_4.sh:1983] [ 324.923599] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [memcg_test_4.sh:1983] [ 352.923640] watchdog: BUG: soft lockup - CPU#0 stuck for 78s! [memcg_test_4.sh:1983] [ 380.923622] watchdog: BUG: soft lockup - CPU#0 stuck for 104s! [memcg_test_4.sh:1983] [ 408.923634] watchdog: BUG: soft lockup - CPU#0 stuck for 130s! [memcg_test_4.sh:1983] [ 436.923645] watchdog: BUG: soft lockup - CPU#0 stuck for 156s! [memcg_test_4.sh:1983]