encounter general protection fault while pxe booting from MaaS server

Bug #1958623 reported by sandeep kumar pabboj
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
dellserver
New
Undecided
Unassigned
grub2 (Ubuntu)
Fix Released
Medium
Unassigned
Jammy
Fix Released
Medium
Unassigned
grub2-signed (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
grub2-unsigned (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Medium
Unassigned
Focal
Fix Released
Medium
Unassigned
Jammy
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
grub can cause a general protection fault in EFI by freeing the wrong number of bytes for an address it allocated, causing the machine not to boot.

[Test plan]
1. Have a Dell 14G server with ubuntu OS and MAAS 2.9.2 application installed. Configure MaaS Deploy Settings to Centos7.
2. Configure another Dell 14G SUT with PXE on boot and enlist it in MAAS.
3. Deploy Centos7 in SUT through MaaS.
4. Power off the system after Centos deployment.
5. Power on the system, SUT goes to MaaS-PXE and we now should be able to boot to installed Centos-7 (Issue is RSOD here).

[Where problems could occur]
We could free too little or too much memory if the code is wrong? Basically all this does is change a free() call to use sizeof() instead of a hardcoded value that was used previously to allocate the buffer. Which should lead to the same result as now in the worst case.

[Original bug report]
Observing RSOD when ubuntu installed server setup with MaaS application as per below procedure and pointing out different OS images from grub to deploying OS for connected server.

certification-static.canonical.com/docs/MAAS_Advanced_Network_Installation_And_Configuration.pdf)

Steps to reproduce the issue,
1. Have a server with ubuntu OS and MAAS 2.9.2 application installed

2. Make sure remote server is configured with PXE and enlisted into MAAS application.

[In remote server enable PXE boot in the BIOD network settings and also configure uEFI boot mode and PXE boot device in BIOS settings]

4. Go to MAAS server and use curl command to check the current grub configuration for new enlisted server and make sure current grub configuration mapping to the "MAC address of NIC which is pointing to the CentOS image to deploy or to install into remote server which is enlisted into MAAS. [please refer "grub_commission" snapshot for more details]

5. After above check power on the new enlisted server and run the PXE boot via NIC port.
RSOD should be able to triggered at stage of "booting under maas direction.." [please refer "RSOD_error" snapshot for mroe details]

6. In step 4, if groub config for commissioned machine is pointing to ubuntu boot kernel the RSOD wont be triggered.

Tags: fr-2047

Related branches

CVE References

Revision history for this message
sandeep kumar pabboj (sandeep-kumar-pabboj) wrote :
Revision history for this message
sandeep kumar pabboj (sandeep-kumar-pabboj) wrote :
information type: Private Security → Private
Revision history for this message
sandeep kumar pabboj (sandeep-kumar-pabboj) wrote :

As per our internal evaluation, seems issue is in GRUB utility used by MAAS (2.9.2).

In the source code of grub(../grub-core/loader/i386/efi/linux.c) at line #215 "gub_efi_allocate_pages_max" allocates ONE PAGE but while freeing the pages in line #365 FOUR PAGES are got freed in "grub_efi_free_pages",

/*Line 215*/ params = grub_efi_allocate_pages_max (0x3fffffff,
/*Line 216*/ BYTES_TO_PAGES(sizeof(*params)));
/*Line 365*/ grub_efi_free_pages ((grub_efi_physical_address_t)(grub_addr_t)params,
/*Line 366*/ BYTES_TO_PAGES(16384));

And RSOD is not observed after changing the above code as like below in line #366,
/* Line 365*/ grub_efi_free_pages ((grub_efi_physical_address_t)(grub_addr_t)params,
/* Line 366*/ BYTES_TO_PAGES(sizeof(*params));

Kindly help us verifying and confirming above observations.

Michael Reed (mreed8855)
information type: Private → Public
Revision history for this message
Jeff Lane  (bladernr) wrote :

what version of grub is being used?

Also, is the reproducible using MAAS 3.1?

Revision history for this message
sandeep kumar pabboj (sandeep-kumar-pabboj) wrote :

Hi Jeff,

Yes with MAAS 3.1 also we see the same issue.

grub version is "grub-efi-amd64-2.04-1ubuntu44.2-grubnetx64".

Revision history for this message
Jeff Lane  (bladernr) wrote :

Thanks for trying that Sandeep. We'll have to escalate that internally and see if we can get someone to take a deeper look.

Revision history for this message
sandeep kumar pabboj (sandeep-kumar-pabboj) wrote :

@jeff lane
please let us know if any updates on this issue.

Revision history for this message
Jeff Lane  (bladernr) wrote :

Ack... Michael is going to refer this to the teams that manage grub and the installer to see how they want to proceed.

Revision history for this message
Michael Reed (mreed8855) wrote :

I have forwarded this bug internally to the appropriate teams and it is under review.

tags: added: fr-2047
Michael Reed (mreed8855)
Changed in grub2 (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Julian Andres Klode (juliank) wrote (last edit ):

This should be fixed in grub 2.06-2ubuntu5 which will land in jammy in the next couple days - it's currently in proposed, but it takes a while for the signed images to be accepted.

We should still SRU that to bionic and focal - after the point release

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.06-2ubuntu5

---------------
grub2 (2.06-2ubuntu5) jammy; urgency=medium

  [ Julian Andres Klode ]
  * Free correct size when freeing params, rather than 16 Ki (LP: #1958623)
  * Build with FUSE3 (LP: #1935659)
  * Only run os-prober on first run and if it previously found other OS
    (LP: #1955109)

  [ Heinrich Schuchardt ]
  * Rename grub-core/loader/efi/linux.c
  * Add patches for GRUB on RISC-V
  * fat: fix listing the root directory
  * Enable building for RISC-V (LP: #1876620)

  [ Julian Andres Klode ]
  * Re-enable peimage code on other archs outside secure boot; this
    fixes LP: #1947046 when not booting in secure boot mode (secure
    boot pending security review of the code)

 -- Julian Andres Klode <email address hidden> Fri, 18 Feb 2022 17:21:16 +0100

Changed in grub2 (Ubuntu):
status: New → Fix Released
Revision history for this message
Jeff Lane  (bladernr) wrote :

@julian I added focal and bionic tasks per your comment #10, should we rewrite this as an SRU bug now, or would you ratner a separate SRU bug to handle focal and bionic?

Revision history for this message
Julian Andres Klode (juliank) wrote :

No, we'll use this bug, but the tasks need to move to grub2-unsigned (where we build the supported UEFI binaries)

no longer affects: grub2 (Ubuntu Bionic)
no longer affects: grub2 (Ubuntu Focal)
Changed in grub2-unsigned (Ubuntu Jammy):
status: New → Fix Released
Changed in grub2-signed (Ubuntu Jammy):
status: New → Fix Released
Jeff Lane  (bladernr)
tags: added: servcert-224
tags: added: servcert-225
Jeff Lane  (bladernr)
tags: added: servcert-226
Jeff Lane  (bladernr)
tags: added: servcert-227
Jeff Lane  (bladernr)
tags: added: servcert-228
tags: removed: servcert-224 servcert-225 servcert-226 servcert-227 servcert-228
Revision history for this message
Julian Andres Klode (juliank) wrote :

Updated the description to avoid ominous acronyms and describe the issue at hand (TIL RSOD = red screen of death).

summary: - encounter RSOD while pxe booting from MaaS server
+ encounter general protection fault while pxe booting from MaaS server
description: updated
Revision history for this message
Julian Andres Klode (juliank) wrote :

Since we want this SRUed I'm going to need someone who can update the bug with a test case for the SRU and then do the testing of the SRUs in focal and bionic; as I don't think the original bug report is universally reproducible.

Revision history for this message
Sujith Pandel (sujithpandel) wrote :

Test case steps:

1. Have a Dell 14G server with ubuntu OS and MAAS 2.9.2 application installed. Configure MaaS Deploy Settings to Centos7.
2. Configure another Dell 14G SUT with PXE on boot and enlist it in MAAS.
3. Deploy Centos7 in SUT through MaaS.
4. Power off the system after Centos deployment.
5. Power on the system, SUT goes to MaaS-PXE and we now should be able to boot to installed Centos-7 (Issue is RSOD here).

We can test the fix once it is made available.

description: updated
tags: added: foundations-todo
Changed in grub2-unsigned (Ubuntu Bionic):
importance: Undecided → Medium
Changed in grub2-unsigned (Ubuntu Focal):
importance: Undecided → Medium
Changed in grub2-unsigned (Ubuntu Bionic):
status: New → Triaged
Changed in grub2-unsigned (Ubuntu Focal):
status: New → Triaged
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu14

---------------
grub2-unsigned (2.06-2ubuntu14) kinetic; urgency=medium

  * SECURITY UPDATE: Fix out of bounds writes due specially crafted fonts.
    - add debian/patches/font-Fix-several-integer-overflows-in-grub_font_construct.patch
    - add debian/patches/font-Fix-an-integer-underflow-in-blit_comb.patch
    - CVE-2022-2601, CVE-2022-3775
    - LP: #1996950
  * Fix various issues as a result of fuzzing, static analysis and code
    review:
    - add debian/patches/font-Reject-glyphs-exceeds-font-max_glyph_width-or-font-m.patch
    - add debian/patches/font-Fix-size-overflow-in-grub_font_get_glyph_internal.patch
    - add debian/patchces/font-Remove-grub_font_dup_glyph.patch
    - add debian/patches/font-Fix-integer-overflow-in-ensure_comb_space.patch
    - add debian/patches/font-Fix-integer-overflow-in-BMP-index.patch
    - add debian/patches/font-Fix-integer-underflow-in-binary-search-of-char-index.patch
    - add debian/patches/fbutil-Fix-integer-overflow.patch
    - add debian/patches/font-Harden-grub_font_blit_glyph-and-grub_font_blit_glyph.patch
    - add debian/patches/font-Assign-null_font-to-glyphs-in-ascii_font_glyph.patch
    - add debian/patches/normal-charset-Fix-an-integer-overflow-in-grub_unicode_ag.patch
  * Enforce verification of fonts when secure boot is enabled:
    - add debian/patches/kern-efi-sb-Enforce-verification-of-font-files.patch
  * Bundle unicode.pf2 in a squashfs memdisk attached to the signed EFI binary
    - update debian/control
    - update debian/build-efi-image
    - add debian/patches/font-Try-opening-fonts-from-the-bundled-memdisk.patch
  * Fix LP: #1997006 - add support for performing measurements to RTMRs
    - add debian/patches/commands-efi-tpm-Refine-the-status-of-log-event.patch
    - add debian/patches/commands-efi-tpm-Use-grub_strcpy-instead-of-grub_memcpy.patch
    - add debian/patches/efi-tpm-Add-EFI_CC_MEASUREMENT_PROTOCOL-support.patch
  * Fix the squashfs tests during the build
    - remove debian/patches/ubuntu-fix-reproducible-squashfs-test.patch
    - add debian/patches/tests-Explicitly-unset-SOURCE_DATE_EPOCH-before-running-f.patch
  * Bump SBAT generation:
    - update debian/sbat.ubuntu.csv.in
  * Source package generated from src:grub2 using make -f ./debian/rules
    generate-grub2-unsigned

 -- Chris Coulson <email address hidden> Wed, 16 Nov 2022 14:40:42 +0000

Changed in grub2-unsigned (Ubuntu Focal):
status: Triaged → Fix Released
Changed in grub2-unsigned (Ubuntu Bionic):
status: Triaged → Fix Released
Benjamin Drung (bdrung)
tags: removed: foundations-todo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.