kernel panic when trying to reboot in bionic

Bug #1759445 reported by Eric Vasquez
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Undecided
Unassigned
linux (Ubuntu)
Confirmed
High
Joseph Salisbury
Bionic
Confirmed
High
Joseph Salisbury

Bug Description

cpe_foundation test deployment of Bionic failed.
After some investigation, it looks like the nodes deployed and installed bionic, but never came back from a reboot.

Accessing the ILO console of a node in question (all nodes failed), it revealed a kernel panic (attached)

Revision history for this message
Eric Vasquez (envas) wrote :
Revision history for this message
Eric Vasquez (envas) wrote :
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Eric,

I experience the same issue in non-MAAS deployed machines after upgrading to the latest kernel. AS such, I'm marking this as invalid for MAAS and opening a task for the kernel.

Changed in maas:
status: New → Invalid
summary: - MAAS 2.3.1 fails to deploy Bionic due to kernel panic
+ Bionic due to kernel panic
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1759445

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: bionic
Revision history for this message
Jason Hobbs (jason-hobbs) wrote : Re: Bionic due to kernel panic

This bug is a kernel panic when rebooting at the end of a MAAS deployment of bionic; there is no way to run apport-collect.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
summary: - Bionic due to kernel panic
+ kernel panic when trying to reboot in bionic
tags: added: foundations-engine
Changed in linux (Ubuntu Bionic):
importance: Undecided → High
status: Confirmed → Triaged
assignee: nobody → Joseph Salisbury (jsalisbury)
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

We can no longer reproduce this.

Changed in linux (Ubuntu Bionic):
status: Triaged → Incomplete
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

We reproduced it again... looking to try the testing now.

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

So far we've only been able to produce this by doing bionic deploys.

One thing that stands out in the rsyslog for bionic deploys is this failure:

http://paste.ubuntu.com/p/y8xXc7PYjp/

Apr 2 17:48:35 leafeon blkdeactivate[1782]: /sbin/blkdeactivate: line 345: /bin/sort: No such file or directory

Could it be related?

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

It's possible that the message is related, but it would be difficult to track down without shortening the number of commits to search through between 4.4 and 4.15.

It sounds like you can't reproduce this error with Artful and 4.13 based kernels. Is that correct?

If so, it would be worth testing the 4.14 final kernel and possibly some of the 4.15 release candidates if 4.14. is good. All the kernels are available from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

After updating firmware on the servers, we can't reproduce it at all anymore.

Revision history for this message
Chris Gregan (cgregan) wrote :

We need to continue to investigate this issue as anyone upgrading a certified hardware with older firware will be stuck. This will be a bad user experience.

Changed in linux (Ubuntu Bionic):
status: Incomplete → Confirmed
Revision history for this message
Chris Gregan (cgregan) wrote :

Mark is pressing to potentially change dist upgrade to detect older firmware versions prior to upgrade to ensure the latest is in place before Ubuntu upgrade

Revision history for this message
John George (jog) wrote :

Possibly a reproduction but in this case Xenial is deployed and rsyslog shows the kernel upgraded to 4.4.0-119-generic. These are automated runs and we don't have the console output to confirm a kernel panic.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.