ubuntu-kernel-tests

mem-on-off-test.sh from memory-hotplug in ubuntu_kernel_selftests failed on X-gcp-4.15 / F-5.4 zVM

Bug #1897764 reported by Po-Hsu Lin on 2020-09-29

This bug affects 2 people

	Status	Importance	Assigned to
ubuntu-kernel-tests	Fix Released	Undecided	Krzysztof Kozlowski
linux (Ubuntu)	Invalid	Undecided	Unassigned
Bionic	Invalid	Undecided	Unassigned
Focal	Invalid	Undecided	Unassigned
Groovy	Invalid	Undecided	Unassigned

Bug Description

Issue found on Focal 5.4.0-49.53 zVM kernel04

Test failed with:
# selftests: memory-hotplug: mem-on-off-test.sh
# Test scope: 2% hotplug memory
# online all hot-pluggable memory in offline state:
# SKIPPED - no hot-pluggable memory in offline state
# offline 2% hot-pluggable memory in online state
# trying to offline 1 out of 16 memory block(s):
# online->offline memory0
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 0: unexpected fail
# online->offline memory1
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 1: unexpected fail
# online->offline memory10
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 10: unexpected fail
# online->offline memory11
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 11: unexpected fail
# online->offline memory12
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 12: unexpected fail
# online->offline memory13
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 13: unexpected fail
# online->offline memory14
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 14: unexpected fail
# online->offline memory15
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 15: unexpected fail
# online->offline memory2
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 2: unexpected fail
# online->offline memory3
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 3: unexpected fail
# online->offline memory4
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 4: unexpected fail
# online->offline memory5
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 5: unexpected fail
# online->offline memory6
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 6: unexpected fail
# online->offline memory7
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 7: unexpected fail
# online->offline memory8
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 8: unexpected fail
# online->offline memory9
# ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
# offline_memory_expect_success 9: unexpected fail
# FAILED - unable to offline some memory blocks, device busy?
# online all hot-pluggable memory in offline state:
# SKIPPED - no hot-pluggable memory in offline state
# Test with memory notifier error injection
not ok 1 selftests: memory-hotplug: mem-on-off-test.sh # exit=1

This issue can be found on 5.4.0-46.50, 5.4.0-45.49 zVM as well.
Passed with 5.4.0-44.48
Passed with 5.4.0-43.47
Failed with 5.4.0-42.46

Looks like it's not very stable.

See original description

Po-Hsu Lin (cypressyew) on 2020-09-29

description:	updated
description:	updated

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2020-09-30:

This "FAILED - unable to offline some memory blocks, device busy?" can be found on some instances (n1-standard-64) with X-gcp 4.15.0-1085.96 as well

summary:

mem-on-off-test.sh from memory-hotplug in ubuntu_kernel_selftests failed
- on F-5.4 zVM
+ on X-gcp-4.15 / F-5.4 zVM

Revision history for this message

Kelsey Steele (kelsey-steele) wrote on 2020-10-01:

Found on Focal 'aws : 5.4.0-1026.26 : amd64' on multiple instances

tags:

added: amd64

Kelsey Steele (kelsey-steele) on 2020-10-07

tags:

added: aws

Revision history for this message

Francis Ginther (fginther) wrote on 2020-10-19:

Seen on groovy 5.8 on t2.small.

tags:	added: groovy
tags:	added: 5.8

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2021-01-20:

Found on Focal GCP 5.4.0-1036.37

tags:

added: gcp sru-20210104

Revision history for this message

Kelsey Steele (kelsey-steele) wrote on 2021-01-28:

spotted on azure : 4.15.0-1106.118 : amd64

tags:

added: 4.15 azure

Ian May (ian-may) on 2021-02-18

tags:

added: sru-20210125

Revision history for this message

Francis Ginther (fginther) wrote on 2021-02-19:

Seen with linux-gcp 5.4.0-1037.40.

Kleber Sacilotto de Souza (kleber-souza) on 2021-03-04

Changed in linux (Ubuntu Focal):
status:	New → Confirmed
Changed in linux (Ubuntu Bionic):
status:	New → Confirmed

Revision history for this message

Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote on 2021-03-04: Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1897764

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete

Kelsey Steele (kelsey-steele) on 2021-04-06

tags:

added: sru-20210315

Kelsey Steele (kelsey-steele) on 2021-04-07

tags:

added: hwe

Revision history for this message

Marcelo Cerri (mhcerri) wrote on 2021-04-12:

Also seen in xenial linux-fips 4.4.0-1060.66 for sru-20210315

tags:

added: 4.4 fips ppc64el

Kleber Sacilotto de Souza (kleber-souza) on 2021-04-20

Changed in linux (Ubuntu Groovy):
status:	New → Confirmed

Revision history for this message

Krzysztof Kozlowski (krzk) wrote on 2021-04-27:

38. 02/25 16:08:02 DEBUG| utils:0153| [stdout] # Test with memory notifier error injection
39. 02/25 16:08:02 DEBUG| utils:0153| [stdout] # ./mem-on-off-test.sh: line 267: echo: write error: Invalid argument
40. 02/25 16:08:02 DEBUG| utils:0153| [stdout] # ./mem-on-off-test.sh: line 283: echo: write error: Invalid argument
41. 02/25 16:08:03 DEBUG| utils:0153| [stdout] # offline_memory_expect_fail 13: unexpected success
42. 02/25 16:08:03 DEBUG| utils:0153| [stdout] # offline_memory_expect_fail 14: unexpected success

groovy/aws 5.8.0-1031.33

tags:

added: sru-20210412

Revision history for this message

Krzysztof Kozlowski (krzk) wrote on 2021-04-30:

#10

Also on: groovy/oracle 5.8.0-1027.28

tags:

added: oracle

Krzysztof Kozlowski (krzk) on 2021-05-26

tags:

added: sru-20210510

Revision history for this message

Krzysztof Kozlowski (krzk) wrote on 2021-06-15:

#11

Found on focal/azure 5.8.0-1034.36~20.04.1-azure

tags:

added: sru-20210531

Krzysztof Kozlowski (krzk) on 2021-06-15

Changed in ubuntu-kernel-tests:
assignee:	nobody → Krzysztof Kozlowski (krzk)

Revision history for this message

Krzysztof Kozlowski (krzk) wrote on 2021-06-15:

#12

Sent patch: https://lists.ubuntu.com/archives/kernel-team/2021-June/121361.html

Changed in ubuntu-kernel-tests:
status:	New → In Progress

Krzysztof Kozlowski (krzk) on 2021-06-23

Changed in ubuntu-kernel-tests:
status:	In Progress → Fix Released
Changed in linux (Ubuntu Bionic):
status:	Confirmed → Invalid
Changed in linux (Ubuntu Focal):
status:	Confirmed → Invalid
Changed in linux (Ubuntu Groovy):
status:	Confirmed → Invalid
Changed in linux (Ubuntu):
status:	Incomplete → Invalid

Revision history for this message

Kleber Sacilotto de Souza (kleber-souza) wrote on 2021-07-07:

#13

This is still affecting at least the generic kernels for amd64 and ppc64el arches.

Changed in linux (Ubuntu Bionic):
status:	Invalid → Confirmed
Changed in linux (Ubuntu Focal):
status:	Invalid → Confirmed
Changed in linux (Ubuntu Groovy):
status:	Invalid → Confirmed

Krzysztof Kozlowski (krzk) on 2021-08-20

Changed in linux (Ubuntu Bionic):
status:	Confirmed → Invalid
Changed in linux (Ubuntu Focal):
status:	Confirmed → Invalid
Changed in linux (Ubuntu Groovy):
status:	Confirmed → Invalid

Revision history for this message

Luke Nowakowski-Krijger (lukenow) wrote on 2021-09-17:

#14

Observed on F/gcp 5.4.0-1053.57 on e2-standard-2, f1-micro, n1-highcpu-4

Krzysztof Kozlowski (krzk) on 2021-10-29

tags:

added: hinted impish sru-20211018

Revision history for this message

Bartlomiej Zolnierkiewicz (bzolnier) wrote on 2021-12-16 (last edit on 2021-12-16):

#15

Observed also on 2021.11.29/focal/linux-gcp/5.4.0-1059.63 on f1-micro and g1-small instances

Revision history for this message

Bartlomiej Zolnierkiewicz (bzolnier) wrote on 2022-05-09:

#16

Also seen in bionic linux-ibm-5.4 5.4.0-1021.23~18.04.1 for sru-20220418

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.