Default 192MB crashkernel reservation is never enough (kdump fails on 20.04)

hello, may I ask how did you capture the log? My kdump also stuck but unfortunately does not print anything. the system looks dead though. the vm got a memory size of 4GiB

From my notes:

=====
OK, I did the following and made some forward progress:

/etc/default/grub: added
 console=ttyS0
update-grub
reboot
virsh console vm843
echo 8 > /proc/sys/kernel/printk
rm -fr /var/crash/*
sync
echo c > /proc/sysrq-trigger

I now see the capture kernel panic messages.
=====

The important item here is the additional serial console which allowed me to view the messages in an xterm and scroll back (on a vnc console, there is no such ability).
eric

________________________________
From: <email address hidden> <email address hidden> on behalf of norman shen <email address hidden>
Sent: Wednesday, June 16, 2021 1:57 AM
To: Eric DeVolder <email address hidden>
Subject: [Bug 1908090] Re: ubuntu 20.04 kdump fails

hello, may I ask how did you capture the log? My kdump also stuck but
unfortunately does not print anything. the system looks dead though. the
vm got a memory size of 4GiB

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1908090

Title:
  ubuntu 20.04 kdump fails

Status in kexec-tools package in Ubuntu:
  New

Bug description:
  When linux-crashdump (5.4.0.58.61) is enabled on Ubuntu 20.04 LTS,
  everything appears to be in good working order, according to
  "systemctl status kdump-tools" and "kdump-config status". However,
  upon an actual crash, the system hangs, and no crash files are
  produced. I've investigated and have learned that the capture kernel
  does indeed start, but it is unable to unpack the rootfs/initrd, and
  thus fails and hangs.

  [ 1.070469] Trying to unpack rootfs image as initramfs...
  [ 1.333182] swapper/0 invoked oom-killer: gfp_mask=0x100cc2(GFP_HIGHUSER), order=0, oom_score_adj=0
  [ 1.335074] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-26-generic #30-Ubuntu
  [ 1.336396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
  [ 1.336396] Call Trace:
  [ 1.336396] dump_stack+0x6d/0x9a
  [ 1.336396] dump_header+0x4f/0x1eb
  [ 1.336396] out_of_memory.part.0.cold+0x39/0x83
  [ 1.336396] out_of_memory+0x6d/0xd0
  ...
  [ 1.413202] ---[ end Kernel panic - not syncing: System is deadlocked on memory ]---

  On this system with 8G of memory, the crash memory as specified on the kernel command line is "crashkernel=512M-:192M". I changed the 192M to 256M, and now kdump works.

  Not sure how the 192M value is chosen, but it does not work. I think
  this used value used to work for 16.04 and maybe 18.04 (I didn't try),
  but is no longer useful for 20.04.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1908090/+subscriptions

Status changed to 'Confirmed' because the bug affects multiple users.

Are you sure you have this bug filed against the correct package?

At a glance, it looks like you should file it against linux-meta (see https://packages.ubuntu.com/focal/linux-crashdump )

kexec-tools "provides tools to load a kernel into memory and then "reboot"
 directly into that kernel using the kexec system call, bypassing the normal
 boot process."

I was/am unsure as to which package determines the crashkernel= settings. If it is another package, then by all means please update accordingly.
Thanks,
eric

________________________________
From: <email address hidden> <email address hidden> on behalf of Todd Taft <email address hidden>
Sent: Thursday, July 1, 2021 2:51 AM
To: Eric DeVolder <email address hidden>
Subject: [Bug 1908090] Re: ubuntu 20.04 kdump fails

Are you sure you have this bug filed against the correct package?

At a glance, it looks like you should file it against linux-meta (see
https://packages.ubuntu.com/focal/linux-crashdump )

kexec-tools "provides tools to load a kernel into memory and then "reboot"
 directly into that kernel using the kexec system call, bypassing the normal
 boot process."

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1908090

Title:
  ubuntu 20.04 kdump fails

Status in kexec-tools package in Ubuntu:
  Confirmed

Bug description:
  When linux-crashdump (5.4.0.58.61) is enabled on Ubuntu 20.04 LTS,
  everything appears to be in good working order, according to
  "systemctl status kdump-tools" and "kdump-config status". However,
  upon an actual crash, the system hangs, and no crash files are
  produced. I've investigated and have learned that the capture kernel
  does indeed start, but it is unable to unpack the rootfs/initrd, and
  thus fails and hangs.

  [ 1.070469] Trying to unpack rootfs image as initramfs...
  [ 1.333182] swapper/0 invoked oom-killer: gfp_mask=0x100cc2(GFP_HIGHUSER), order=0, oom_score_adj=0
  [ 1.335074] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-26-generic #30-Ubuntu
  [ 1.336396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
  [ 1.336396] Call Trace:
  [ 1.336396] dump_stack+0x6d/0x9a
  [ 1.336396] dump_header+0x4f/0x1eb
  [ 1.336396] out_of_memory.part.0.cold+0x39/0x83
  [ 1.336396] out_of_memory+0x6d/0xd0
  ...
  [ 1.413202] ---[ end Kernel panic - not syncing: System is deadlocked on memory ]---

  On this system with 8G of memory, the crash memory as specified on the kernel command line is "crashkernel=512M-:192M". I changed the 192M to 256M, and now kdump works.

  Not sure how the 192M value is chosen, but it does not work. I think
  this used value used to work for 16.04 and maybe 18.04 (I didn't try),
  but is no longer useful for 20.04.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1908090/+subscriptions

Same issue here, with the same workaround for me:

Edit /etc/default/grub.d/kdump-tools.cfg

Change the crashkernel part to read
    crashkernel=512M-:256M

Yes, me too. The same fix fixed it.

This is still an issue on Ubuntu Server 20.04.4 LTS 5.4.0.122-generic using linux-crashdump-5.4.0.122.123.

The workaround in comment #6 didn't work but did work increasing it to 512M-:1024M .

I ran into this today. 192MB seems not sufficient at all - even with an LXD VM with only 1G of RAM. 384M seems to work reliably for basic empty VMs though we often find larger servers need more.

I think we need to revise this default, it has support for setting defaults based on how much RAM the host has. I think we can leverage that to try set a better default out of the box and increase it for big servers where we won't notice the lost RAM.

This really, really should be raised.

Not setting the default higher risks a loss of coredumps from production systems due to memory exhaustion when the host boots the kdump kernel. You would think that something like that was tested in all cases, but I did not and just got bit on what was fortunately a toy system...

If you really want to be conservative, you could even say kdump with systems <= 2GiB of RAM or something should only use 192M still (though as per Trent's observations -- it seems like the floor needs to be raised, period? or initrd needs to go on a diet?)