Power consumption higher after suspend - Ubuntu 20.10 on Asus UX425JA

This change was made by a bot.

Can you please run the following and attach "active" here:
$ find /sys/devices 2>/dev/null | grep runtime_status | while read i; do grep -H . $i; done | egrep -v 'suspended|unsupported' > active

I have run the command and attached the output. I did it both just after boot and after suspend/resume - same result both times, so I have attached just one output "active".

I have been poking around to see any differences in drivers/system state between after boot and after suspend/resume. I have found one difference - using the following command:
sudo lspci -vv | grep -E '(^\S|\s+LnkCtl:)'
For the Nvme device the LnkCtl - changes from ASPM L1 Enabled to ASPM Disabled. I have attached output of the command before and after suspend/resume. If it is not relevant please just ignore.

The NVMe device didn't restore the LTR value.
Can you please boot with kernel parameter "pci.dyndbg log_buf_len=16M", reproduce the issue and attach dmesg?

I have done the following - turned off pc. At boot pressed "e" added - pci.dyndbg log_buf_len=16M - to the parameteres after VT-handoff. Logged in, suspend/resumed and added output of "sudo dmesg".

Thanks for the log. The NVMe drive dropped off the PCI bus after resume.

Thanks, please test latest mainline kernel:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11-rc4/amd64/

Thanks a lot for the help!
I have tried kernel 5.11-rc4.

It does not solve the issue. I have tried 2 test/reboots.
Power consumption seems better with new kernel just after boot - around 1.5W but after suspend/resume it is around 4.6W witch is much the same. The ASPM state is like with the old kernel - ASPM L1 Enabled before suspend/resume and after ASPM Disabled.

I will be off for tonight.

Can you please attach "acpidump"?

Also, what does "cat /sys/bus/pci/devices/0000:00:1d.0/firmware_node/path" say?

Output of "sudo acpidump > acpidump" is attached.
"cat /sys/bus/pci/devices/0000:00:1d.0/firmware_node/path" - outputs "\_SB_.PCI0.RP09"

Thanks, please test this kernel:
https://people.canonical.com/~khfeng/lp1912057/

I have tested the rc4+ kernel (two reboots with similar results).

I will try to summarize the results:

Rc4+: ASPM L1 enabled both before and after suspend/resume!

Rc4+ before suspend/resume:
Power consumption around 4.3 W
Idle stats: pc2-pc3 is reached
ASPM L1 enabled for NVMe (and all others)

Rc4+ after suspend/resume:
Power consumption around 4.3 W
Idle stats: pc2-pc3 is reached
ASPM L1 enabled for NVMe (and all others)

Rc4+ breaks some Fn-keys and touchpad. In powertop I get 600 wakeups/s and 10% CPU usage. The Webcam is active 100%. I have attached output from powertop.

Rc4 before suspend/resume:
Power consumption around 1.4W
Idle stats: pc2-pc3-pc7-pc8-pc10 is reached
ASPM L1 enabled for NVMe (and all others)

Rc4 after suspend/resume:
Power consumption around 4.3W
Idle stats: pc2-pc3 is reached
ASPM Disabled for NVMe (and L1 enabled for all others)

Rc4 - In powertop I get 70 wakeups/s and 1% CPU usage.

Please give this a try:
https://people.canonical.com/~khfeng/lp1912057-5.8/

Can you please also attach dmesg under test kernel?

Thanks for the new kernel. I have tried it.

Before suspend/resume:
Power consumption around 2.4 W
Idle stats: pc2-pc3-pc6 is reached
ASPM L1 enabled for NVMe

After suspend/resume:
Power consumption around 4.9 W
Idle stats: pc2-pc3 is reached
ASPM Disabled for NVMe

and keys & touchpad is working.

dmesg is attached.

Can you please boot the vanilla 5.11-rc4 kernel, run `# echo 'file drivers/pci/* +p' > /sys/kernel/debug/dynamic_debug/control`, suspend/resume, and attach dmesg?

I have not used dynamic_debug before - hope I did it right.
As regular user: "sudo -i"
Then as root: "echo 'file drivers/pci/* +p' > /sys/kernel/debug/dynamic_debug/control"
followed by "dmesg > dmesg.rc4".

Thanks! Can you please also do the same for the kernel in comment #14?

That's really helpful!
Can you please test this kernel, which solves another similar issue I am working on:
https://people.canonical.com/~khfeng/acs-hack-take-2/

I did a similar test as you asked in comment #20.

Before suspend/resume:
Powertop: around 1.5W - pc2,3,7,8,10
ASPM L1 enabled for all controllers

After suspend/resume:
Powertop: around 4.8W - pc2,3
ASPM Disabled for NVMe

Fn-keys and touchpad working.

The test kernel (accidentally) disabled pcieport driver, and made the issue went away.
Two things happened:
1) The rootport stays at D0
2) PME service isn't enabled

Let's try 1) first. Will upload a kernel soon.

https://people.canonical.com/~khfeng/lp1912057-bridge-d0/

Before suspend/resume:
Powertop: around 1.4W - pc2,3,7,8,10
ASPM L1 enabled for all controllers

After suspend/resume:
Powertop: around 4.6W - pc2,3
ASPM Disabled for NVMe

Fn-keys and touchpad working.

Thanks, apparently the root port doesn't stay at D0.
Please give this one a try:
https://people.canonical.com/~khfeng/lp1912057-bridge-d0-2/

I tried the ~khfeng/lp1912057-bridge-d0-2.

Before suspend/resume:
Powertop: around 1.4W - pc2,3,7,8,10
ASPM L1 enabled for all controllers

After suspend/resume:
Powertop: around 4.6W - pc2,3
ASPM Disabled for NVMe

Attached dmesg after "echo 'file drivers/pci/* +p' > /sys/kernel/debug/dynamic_debug/control" and suspend/resume.

So, saving parent device stat in child's suspend routine doesn't keep parent device at D0.

Please test this one instead, it prevents NVMe with HMB shutting down:
https://people.canonical.com/~khfeng/lp1912057-keep-hmb-nvme-d0/

Good news!

Before suspend/resume:
Powertop: around 1.5W - pc2,3,7,8,10
ASPM L1 enabled for all controllers

After suspend/resume:
Powertop: around 1.5W - pc2,3,7,8,10
ASPM L1 enabled for all controllers

Fn-keys and touchpad working.

Dmesg is attached.

After using the kernel from #31 for 3 days and having many suspend/resumes the system has resumed with ASPM L1 enabled and same power as before suspend every time.

The issue is resolved in latest mainline kernel:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11/amd64/
I will change the status as resolved.

Sorry for the belated response, I was on vacation.

It's interesting that it's fixed by 5.11, because I am not seeing any change that can help the issue.

Anyway, glad it's sill fixed.

Hi again,

I have reopened this issue. It is not fixed in mainline 5.11.0, 5.11.7 or 5.11.10.

I think I was using your kernel and mixed up the version numbers when I sent the message in #34. Since last I have updated BIOS version from V304->V306. For the last month I have been using the laptop with a USB-hub so I have not payed much attention to the power consumption - this is why I did not see this before now.
Let me know if you need me to send more information.

@Henrik,
When you try the kernel(#31), is issue gone?

@koba
Using kernel from #31 - the power consumption is the same before and after suspend/resume and ASPM L1 is enabled for the NVMe after resume. Yes the issue is gone.

@Henrik,
Would you please help to try this kernel?
https://drive.google.com/drive/folders/1ixfPnW4N-P_UGS-YkbluCVBuxDnPzXZv?usp=sharing
Please also append the dmesg after you reproduce it.

@Koba
I have tested the kernel from #39. It fixes the issue. I have attached dmesg. (Installing the headers failed).

@Henrik
Thanks for your help.
Would you please try another kernel that disables HMB prior to s2idle? and
also help to collect dmesg, thanks
https://drive.google.com/drive/folders/1xMArRlMkED20kSu1pTKuRPvBT8kh__L2?usp=sharing

@Koba
I have tested kernel in #41 (5.10.0-1025lpv1912057b) and attached dmesg. This kernel does not fix the issue. ASPM is Disabled for NVMe and higher power consumption after suspend/resume.

@Henrik,
Thanks, would you please try this kernel again that I want to collect
the power state of nvme during suspend&resume?
please also collect dmesg, thanks
https://drive.google.com/drive/folders/1hU4OflO8nE4ShC0rOahpGsu8LhcF_UC-?usp=sharing

@Koba
No problem. I will do.
I have tested kernel in #43 (5.10.0-1025lpv1912057c) and attached dmesg. This kernel does not fix the issue. ASPM is Disabled for NVMe and higher power consumption after suspend/resume.

@Henrik,
Thanks for your hard work, Would you please try another kernel that
avoids nvme to fall into d3cold.
Please also collect dmesg, thanks
https://drive.google.com/drive/folders/1Du-A8EJYxUwS7zkYkfox7JFeU6u8puEZ?usp=sharing

@Koba
I can only see the buildinfo file in the last folder.

@Henrik, please check it again. Something wrong during uploading and
it was fixed.

@Koba, I have tried kernel from #45 (5.10.0-1025lpv1912057d). After suspend, it fails when resuming. The laptop wakes up and I am able to log in, but then it freezes. So no dmesg.

@Henrik, I'm very appreciative you could help me to figure out this
issue, thanks.
Sorry for the crash, would you please help to try this and help to
collect dmesg?
https://drive.google.com/drive/folders/10GpLDqM6OXQ11tDmX5zq6ppA5wsCKeIQ?usp=sharing
Thanks again.

Hi Koba,
This time the laptop did not resume at all. It did boot first and seemed to function normally but I did not test and quite fast after 30-60sec suspended it. Do you want a dmesg just after boot?

@Henrik,
Please fire the cmd to gather dmesg after you load the last test
kernel(5.10.0-1025lpv1912057e) and reboot at the first time.
#journalctl -k -b -1

After that, please load the official 5.10-oem-1025 and disable d3cold for nvme
#echo 0 | sudo tee /sys/bus/pci/devices/0000\:58\:00.0/d3cold_allowed
Then try to s2idle&resume and measure the power consumption .

@Koba,
Here is the dmesg from kernel(5.10.0-1025lpv1912057e).
For the official 5.10-oem-1025 it seems like disabling d3cold for nvme fixes the issue. ASPM L1 Enabled for nvme and power consumption after suspend/resume is as before. I have attached output from powertop.

@Henrik,
Thanks.
here's another test kernel that disable d3cold_allowed for nvme with hmb.
https://drive.google.com/drive/folders/1gK67OYm6-jeKLj7GIQNZLCaHuaOwVTr1?usp=sharing

Please check the status of d3cold_allowed(it must be 0) before
s2idle&resume.
sudo cat /sys/bus/pci/devices/0000\:58\:00.0/d3cold_allowed

@Koba,
Thanks for your help.
I have tried the kernel from #55.
sudo cat /sys/bus/pci/devices/0000\:58\:00.0/d3cold_allowed showed "0".
The suspend & resume did end up in a freeze. I have attached dmesg (journalctl -k -b -1 on next boot).
Just as a note I saw the power consumption during suspend was around 1.0W compared to 0.5W using the 5.10-oem-1025 and "echo 0 | sudo tee...." from #51. This was using an external AC power meter - so it includes the power loss in the AC adapter.

@Henrik,
Sorry for causing your inconvenience.
don't be in a hurry and if you have free time, please help to try this
test kernel remove irrelevant codes.
https://drive.google.com/drive/folders/1wRGJNPU0LmmyJMkVLZvG7xLKqinsOZU0?usp=sharing

@Koba,
I have tried kernel 5.10.0-1025lpv1912057g. After boot cat /sys/bus/pci/devices/0000\:58\:00.0/d3cold_allowed showed "0".
After suspend/resume the power consumption is higher. ASPM is disabled for Nvme. I have attached dmesg.

@Henrik,
Thanks, would you please try this that follows the original path of
d3cold_allowed to disable nvme d3cold.
https://drive.google.com/drive/folders/19-dxtVTFNq5WtN3KOxjOrAXQ_1kYrnbh?usp=sharing

@Koba,
I have tried kernel 5.10.0-1025lpv1912057h. After boot cat /sys/bus/pci/devices/0000\:58\:00.0/d3cold_allowed showed "0".
After suspend/resume power consumption is the same and ASPM L1 is enabled for NVMe. I have attached dmesg. I will test the kernel for a longer time period and see if the power consumption changes.

@Henrik,
Thanks a lot for your efforts, can I have you test-by signature?

@Koba,
Sorry, but I do not know what "test-by signature" is.

@Henrik,
For example, someone help to test a solution and the developer come out a
patch,
The developer would leave the guy's signature in his patch.
If you want this way, please give your email and the full name.

commit 4514d991d99211f225d83b7e640285f29f0755d0
Author: Rafael J. Wysocki <email address hidden>
Date: Tue Mar 16 16:51:40 2021 +0100

    PCI: PM: Do not read power state in pci_enable_device_flags()
...
    Link:
https://<email address hidden>/
    Reported-by: Maximilian Luz <email address hidden>
    Tested-by: Maximilian Luz <email address hidden>
    Signed-off-by: Rafael J. Wysocki <email address hidden>
    Reviewed-by: Mika Westerberg <email address hidden>

@Henrik,
would you mind to verify this that wakes nvme from d3.
https://drive.google.com/drive/folders/1O30CIxm4YZVexBUc919yI5D3Jmw8M0V4?usp=sharing

@Koba,
I have tested kernel 5.10.0-1025lpv1912057i.
After boot cat /sys/bus/pci/devices/0000\:58\:00.0/d3cold_allowed showed "0".
After suspend/resume power consumption is the same as before and ASPM L1 is enabled for NVMe. I have attached dmesg.

My full name is Henrik Juul Hansen and email is <email address hidden>. You are welcome to put my name on the things I have tested. I have updated my Launchpad info so it should be visible to others.

@Henrik,
forget to reset the patch that disables d3cold_allowed and please try this
again. thanks
the d3cold_allowed shoube 1.
https://drive.google.com/drive/folders/17qeSGLKz8hR8TLsr35qS_4RSDUZcpaLu?usp=sharing

@Koba
I tried kernel 5.13.0-051300rc2-generic. After boot d3cold_allowed was "1", ASPM L1 enabled for nvme (and bluetooth did not work). After suspend/resume ASPM L1 was disabled for nvme. Power consumption higher. dmesg is attached.

@Henrik, would you please help to collect lspci -vvv
#sudo lspci -vvv

@Koba, I have attached output from "sudo lspci -vvv" before and after suspend/resume for kernel 5.13.0-051300rc2-generic.

Just a follow-up to anyone else is using this hardware.
I have been using kernel 5.11.0-17-generic and "echo 0 | sudo tee /sys/bus/pci/devices/0000\:58\:00.0/d3cold_allowed" just after boot. Now I have uptime 4 days 23 hours with many suspend/resumes and power consumption is remaining as it is just after boot.

https://lkml.org/lkml/2021/6/25/52

Henrik, please test this kernel:
https://people.canonical.com/~khfeng/lp1927866-upstream/

Kai-Heng,
I have tried kernel 5.13.0+ from #73.
Power consumption is the same and NVMe has ASPM L1 enabled both before and after suspend/resume. I have been running the kernel for a few hours now and also done some suspend/resume and it seems to work fine.
I have attached dmesg and powertop output.