1) It was not backported fully, missing the debug message at its bottom. That led me to initially think, when debugging, that the patch hasn't been included at all. Later I could check kernel tree and saw the it was manually merged, missing that debug message part.
2) Since functional piece of code was backported, it should, still, have worked, and it did not. Something in trusty is not making it to work like expected. I'm still investigating it through a kdump taken during a shutdown hang on trusty, but, I don't want to hold the kernel release because of that:
[ 144.864794] session4: iscsi_eh_device_reset LU Reset [sc ffff880037a9a100 lun 1]
[ 144.865907] session4: iscsi_eh_device_reset dev reset result = FAILED
[ 144.866875] session4: iscsi_eh_target_reset tgt Reset [sc ffff880037a9a100 tgt iqn.2017.tgtd:tid5.lun]
[ 144.868343] session4: iscsi_eh_target_reset tgt iqn.2017.tgtd:tid5.lun reset result = FAILED
[ 144.869580] session4: iscsi_eh_session_reset wait for relogin
The iscsi transport layer, did not queue the other commands, after the first one timed out, but instead of warning the upper layer, it proceeded with a "device reset", causing the "relogin" logic to be stuck since the transport layer was already gone.
This is the hang backtrace:
crash> bt
PID: 5980 TASK: ffff880057e19800 CPU: 0 COMMAND: "halt"
#0 [ffff880037b53a50] __schedule at ffffffff8173af59
#1 [ffff880037b53ab8] schedule at ffffffff8173b3e9
#2 [ffff880037b53ac8] schedule_timeout at ffffffff8173a5d5
#3 [ffff880037b53b70] io_schedule_timeout at ffffffff8173bacb
#4 [ffff880037b53ba0] wait_for_completion_io_timeout at ffffffff8173c1e6
#5 [ffff880037b53bf8] blk_execute_rq at ffffffff8134b9fb
#6 [ffff880037b53ca8] scsi_execute at ffffffff814f1a77
#7 [ffff880037b53cf0] scsi_execute_req_flags at ffffffff814f2cfc
#8 [ffff880037b53d58] sd_sync_cache at ffffffff81500626
#9 [ffff880037b53dd0] sd_shutdown at ffffffff81500bb9
#10 [ffff880037b53df0] device_shutdown at ffffffff814a4495
#11 [ffff880037b53e20] kernel_power_off at ffffffff81096b75
#12 [ffff880037b53e30] SYSC_reboot at ffffffff81096d4b
#13 [ffff880037b53f70] sys_reboot at ffffffff81096ebe
#14 [ffff880037b53f80] system_call_fastpath at ffffffff81748170
RIP: 00007ff48aa45bc6 RSP: 00007ffef9bc4be8 RFLAGS: 00010246
RAX: 00000000000000a9 RBX: ffffffff81748170 RCX: 000000000000001e
RDX: 000000004321fedc RSI: 0000000028121969 RDI: fffffffffee1dead
RBP: 0000000000000000 R8: fefefefefefefeff R9: 0000000000000000
R10: 00007ff48ad13c8c R11: 0000000000000202 R12: ffffffff81096ebe
R13: ffff880037b53f78 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 00000000000000a9 CS: 0033 SS: 002b
Taken by a watchdog timeout on a hang kvm system that had network interface shutdown before the transport layer could log out.
I'm failing trusty's patch because:
1) It was not backported fully, missing the debug message at its bottom. That led me to initially think, when debugging, that the patch hasn't been included at all. Later I could check kernel tree and saw the it was manually merged, missing that debug message part.
2) Since functional piece of code was backported, it should, still, have worked, and it did not. Something in trusty is not making it to work like expected. I'm still investigating it through a kdump taken during a shutdown hang on trusty, but, I don't want to hold the kernel release because of that:
This is the output for my tests:
https:/ /pastebin. ubuntu. com/p/B9fp5y5gq K/
And you can check (1) with:
[ 144.862511] session4: iscsi_eh_ cmd_timed_ out return nh
It should be "or shutdown" there, together.
Differently than other tests, in trusty's kernel (3.13) you can see:
[ 144.860010] session4: iscsi_eh_ cmd_timed_ out scsi cmd ffff880037a9a100 timedout cmd_timed_ out sc on shutdown, handled cmd_timed_ out return nh
[ 144.861400] session4: iscsi_eh_
[ 144.862511] session4: iscsi_eh_
[ 144.863483] session4: iscsi_queuecommand iscsi: cmd 0x3 is not queued (2)
[ 144.864794] session4: iscsi_eh_ device_ reset LU Reset [sc ffff880037a9a100 lun 1] device_ reset dev reset result = FAILED target_ reset tgt Reset [sc ffff880037a9a100 tgt iqn.2017. tgtd:tid5. lun] target_ reset tgt iqn.2017. tgtd:tid5. lun reset result = FAILED session_ reset wait for relogin
[ 144.865907] session4: iscsi_eh_
[ 144.866875] session4: iscsi_eh_
[ 144.868343] session4: iscsi_eh_
[ 144.869580] session4: iscsi_eh_
The iscsi transport layer, did not queue the other commands, after the first one timed out, but instead of warning the upper layer, it proceeded with a "device reset", causing the "relogin" logic to be stuck since the transport layer was already gone.
This is the hang backtrace:
crash> bt completion_ io_timeout at ffffffff8173c1e6 req_flags at ffffffff814f2cfc call_fastpath at ffffffff81748170
PID: 5980 TASK: ffff880057e19800 CPU: 0 COMMAND: "halt"
#0 [ffff880037b53a50] __schedule at ffffffff8173af59
#1 [ffff880037b53ab8] schedule at ffffffff8173b3e9
#2 [ffff880037b53ac8] schedule_timeout at ffffffff8173a5d5
#3 [ffff880037b53b70] io_schedule_timeout at ffffffff8173bacb
#4 [ffff880037b53ba0] wait_for_
#5 [ffff880037b53bf8] blk_execute_rq at ffffffff8134b9fb
#6 [ffff880037b53ca8] scsi_execute at ffffffff814f1a77
#7 [ffff880037b53cf0] scsi_execute_
#8 [ffff880037b53d58] sd_sync_cache at ffffffff81500626
#9 [ffff880037b53dd0] sd_shutdown at ffffffff81500bb9
#10 [ffff880037b53df0] device_shutdown at ffffffff814a4495
#11 [ffff880037b53e20] kernel_power_off at ffffffff81096b75
#12 [ffff880037b53e30] SYSC_reboot at ffffffff81096d4b
#13 [ffff880037b53f70] sys_reboot at ffffffff81096ebe
#14 [ffff880037b53f80] system_
RIP: 00007ff48aa45bc6 RSP: 00007ffef9bc4be8 RFLAGS: 00010246
RAX: 00000000000000a9 RBX: ffffffff81748170 RCX: 000000000000001e
RDX: 000000004321fedc RSI: 0000000028121969 RDI: fffffffffee1dead
RBP: 0000000000000000 R8: fefefefefefefeff R9: 0000000000000000
R10: 00007ff48ad13c8c R11: 0000000000000202 R12: ffffffff81096ebe
R13: ffff880037b53f78 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 00000000000000a9 CS: 0033 SS: 002b
Taken by a watchdog timeout on a hang kvm system that had network interface shutdown before the transport layer could log out.