nvidia soft lockup after blanking

X11 desktop freezes sporadic and quite frequently (about every other day). I assume it's related to the nvidia driver. An other system with same setup and AMD graphics works stable.

ssh remote login and reboot is still possible.

I've attached also a dmesg directly after the freeze happened.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: nvidia-driver-460 460.32.03-0ubuntu0.20.04.1
ProcVersionSignature: Ubuntu 5.4.0-65.73-generic 5.4.78
Uname: Linux 5.4.0-65-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: KDE
Date: Wed Feb 10 09:07:10 2021
InstallationDate: Installed on 2020-12-02 (70 days ago)
InstallationMedia: Ubuntu-Server 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
SourcePackage: nvidia-graphics-drivers-460
UpgradeStatus: No upgrade log present (probably fresh install)