[linux-azure][hibernation] ]VM hangs after hibernation/resume if the VM has SRIOV NIC and has been deallocated

Bug #1894895 reported by Dexuan Cui
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-azure (Ubuntu)
Invalid
Undecided
Marcelo Cerri
Focal
Fix Committed
Medium
Unassigned
Groovy
Invalid
Undecided
Unassigned

Bug Description

[Impact]

Description of problem:
On Azure, if the VM is Stopped(deallocated) and later Started, the VF NIC's VMBus Instance GUID may change, and as a result hibernation/resume can hang forever.

This happens to the latest stable release of the linux-azure 5.4.0-1023.23 kernel and the latest mainline linux kernel.

[Test Case]

How reproducible:
100%

Steps to Reproduce:
1. Start a VM in Azure that supports Accelerated Networking, and enable hibernation properly (please refer to https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1880032/comments/14 )

2. Do hibernation from serial console
# systemctl hibernate

4. After the VM state changes to "Stopped", click "Stop" button from Azure portal to change the VM state to Stopped(deallocated)

5. Wait for some time (e.g. 10 minutes? 1 hour?), and click the "Start" button to start the VM, and then check the boot-up process from the serial console.

Actual results:
Can not boot up. VM hangs after resume.

Starting Resume from hibernation us…6c7-2c0c-491e-adcf-b625d69faf76...
[ 19.822747] PM: resume from hibernation
[ 19.836693] Freezing user space processes ... (elapsed 0.003 seconds) done.
[ 19.846968] OOM killer disabled.
[ 19.850236] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 20.542934] PM: Using 1 thread(s) for decompression
[ 20.548250] PM: Loading and decompressing image data (559580 pages)...
[ 22.844964] PM: Image loading progress: 0%
[ 28.131327] PM: Image loading progress: 10%
[ 32.346480] PM: Image loading progress: 20%
[ 37.453971] PM: Image loading progress: 30%
[ 40.834525] PM: Image loading progress: 40%
[ 42.980629] PM: Image loading progress: 50%
[ 44.342959] PM: Image loading progress: 60%
[ 45.506197] PM: Image loading progress: 70%
[ 46.800445] PM: Image loading progress: 80%
[ 48.010185] PM: Image loading progress: 90%
[ 49.045671] PM: Image loading done
[ 49.050419] PM: Read 2238320 kbytes in 28.48 seconds (78.59 MB/s)
[ 49.074198] printk: Suspending console(s) (use no_console_suspend to debug)

(The VM hangs here forever)

[Regression Potential]

The fix touches vmbus and can compromise the hyper-v guest drivers. However the change is simple and just adds an additional timeout.

[Other info]

BUG FIX:
A workaround patch is available and is being reviewed: https://lkml.org/lkml/2020/9/4/1270

Final fix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=19873eec7e13fda140a0ebc75d6664e57c00bfb1

Revision history for this message
Dexuan Cui (decui) wrote :
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

5.8 already received the fix by upstream stable updates (LP:#1896795) and hibernation is not supported in 4.15, because of that I'm targeting only 5.4.

Changed in linux-azure (Ubuntu Focal):
status: New → In Progress
Marcelo Cerri (mhcerri)
description: updated
Revision history for this message
Marcelo Cerri (mhcerri) wrote :
description: updated
Stefan Bader (smb)
Changed in linux-azure (Ubuntu Focal):
importance: Undecided → Medium
Changed in linux-azure (Ubuntu):
status: New → Invalid
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

The fix was already reviewed/acked in the mailing list.

Ian May (ian-may)
Changed in linux-azure (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

The fix was already applied to groovy via an upstream stable update (LP:#1896795)

Changed in linux-azure (Ubuntu Groovy):
status: New → Invalid
Changed in linux-azure (Ubuntu):
assignee: nobody → Marcelo Cerri (mhcerri)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.