Kernels & kernel drivers fail to build with gcc-9 [error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible]

Status changed to 'Confirmed' because the bug affects multiple users.

This affects package builds for Eoan with Liquorix in launchpad: https://launchpad.net/~damentz/+archive/ubuntu/liquorix/+build/17232577

This prevents compilation of NVIDIA drivers (DKMS) also.

I *suspect* (but am not at all authoritative or knowledgeable enough to say so) that the fix here will actually be to change the build flags used by the kernel set in /usr/src/linux-headers-$(uname -r)/Makefile

As a temporary workaround I changed the compile flags in /usr/src/linux-headers-foo/Makefile around line 634 to

RETPOLINE_CFLAGS_GCC := -mindirect-branch-register
RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch-register

and now it compiled with gcc9

However, even though it compiled it seemed to be broken, so instead I just changed the default compiler to gcc-8 and that worked fine.

Status changed to 'Confirmed' because the bug affects multiple users.

WORKAROUND:

sudo ln -fs gcc-8 /usr/bin/gcc

https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040741.html

This also appears to be breaking the Ubuntu mainline kernel builds:

https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2/BUILD.LOG.amd64

I uploaded a workaround, and it is in eoan-proposed:
https://launchpad.net/ubuntu/+source/nvidia-graphics-drivers-430/430.26-0ubuntu3

Same issue with:

Ubuntu 19.10
xtables-addons-dkms 3.2-1ubuntu2
linux kernel 5.0.0-17 and 5.0.0-20
gcc-9: 9.1.0-6ubuntu2

The compilation complains that:
 echo >&2; \
 echo >&2 " ERROR: Kernel configuration is invalid."; \
 echo >&2 " include/generated/autoconf.h or include/config/auto.conf are missing.";\
 echo >&2 " Run 'make oldconfig && make prepare' on kernel src to fix it."; \

but both files are present. Later on, I get a huge number of similar errors like the following:
 ./include/linux/compiler.h:192:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

Switching to gcc-8 works around this critical issue.

As per comment #4, i made the same change:

doug@serv-ee:~/temp-k-git/linux$ git diff
diff --git a/Makefile b/Makefile
index 3e4868a6498b..68f0858484a5 100644
--- a/Makefile
+++ b/Makefile
@@ -630,8 +630,8 @@ ifdef CONFIG_FUNCTION_TRACER
   CC_FLAGS_FTRACE := -pg
 endif

-RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register
-RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch=thunk-inline -mindirect-branch-register
+RETPOLINE_CFLAGS_GCC := -mindirect-branch-register
+RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch-register
 RETPOLINE_CFLAGS_CLANG := -mretpoline-external-thunk
 RETPOLINE_VDSO_CFLAGS_CLANG := -mretpoline
 RETPOLINE_CFLAGS := $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-option,$(RETPOLINE_CFLAGS_CLANG)))

And while the kernel compile had a constant stream of warnings, the resulting kernel (5.2) works fine.

The origin of this bug is the change in the gcc-9 package on 26 May 2019. Shouldn't the fcf-protection change there just be reverted?

As per the changelog:

gcc-9 (9.1.0-3) experimental; urgency=medium

  * Update to SVN 20190526 (r271629) from the gcc-9-branch.
    - Fix PR libgomp/90527, PR c++/90532, PR libstdc++/90299,
      PR libstdc++/90454, PR debug/90197, PR pch/90326, PR c++/90484,
      PR tree-optimization/90385, PR c++/90383, PR tree-optimization/90303,
      PR tree-optimization/90316, PR tree-optimization/90316,
      PR libstdc++/90220, PR libstdc++/90557, PR sanitizer/90570,
      PR target/90547 (x86), PR libfortran/90038, PR fortran/90498,
      PR libfortran/90038, PR libfortran/90038, PR fortran/54613,
      PR fortran/54613, PR libstdc++/85965, PR target/90530 (PARISC),
      PR c++/90572.
  * Turn on -fstack-clash-protection and -fcf-protection in Ubuntu 19.10 on
    supported architectures.
  * Fix PR bootstrap/87338 on ia64 (James Clarke). Addresses: #927976.
  * Enable LTO builds on 64bit architectures.
  * Update libstdc++ symbols files for gcc-4-compatible builds.
  * Build the nvptx offload compiler on ppc64el.
  * Build the libgomp-hsa plugin.

 -- Matthias Klose <email address hidden> Sun, 26 May 2019 17:59:59 +0200

no, the kernel infrastructure should handle that like any other hardening flags before.

It looks like this is being addressed in recent linux packages.

As per https://launchpad.net/ubuntu/+source/linux/5.2.0-8.9

"SAUCE: add -fcf-protection=none to retpoline flags"

If I make the same changes to the Makefile as per the link in comment #16, then the kernel compiles fine and without the zillions of warnings I mentioned in comment #13. However I can not find this anywhere "upstream" as the notes seem to indicate.

doug@serv-ee:~/temp-k-git/linux$ git diff
diff --git a/Makefile b/Makefile
index 3e4868a6498b..d8ccc47215be 100644
--- a/Makefile
+++ b/Makefile
@@ -630,8 +630,8 @@ ifdef CONFIG_FUNCTION_TRACER
   CC_FLAGS_FTRACE := -pg
 endif

-RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register
-RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch=thunk-inline -mindirect-branch-register
+RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register -fcf-protection=none
+RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch=thunk-inline -mindirect-branch-register -fcf-protection=none
 RETPOLINE_CFLAGS_CLANG := -mretpoline-external-thunk
 RETPOLINE_VDSO_CFLAGS_CLANG := -mretpoline
 RETPOLINE_CFLAGS := $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-option,$(RETPOLINE_CFLAGS_CLANG)))

As a follow-up tp my previous comment #12 (https://bugs.launchpad.net/ubuntu/+source/xtables-addons/+bug/1830961/comments/12), I have discovered 2 new facts:
- I am not able to build dpdk 19.05 on the same system anymore, (cf. build log), unless I modify the gcc symlink to point to gcc-8 instead of gcc-9
- however, the xtables -addons issue encountered on the previous system has not been seen on 2 other Ubuntu Eoan platforms with the exact same versions of:
  + gcc-9 (and/usr/bin/gcc -> gcc-9)
  + linux kernels
  + xtables-addons-dkms/common

This bug was fixed in the package nvidia-graphics-drivers-430 - 430.34-0ubuntu2

---------------
nvidia-graphics-drivers-430 (430.34-0ubuntu2) eoan; urgency=medium

  * debian/dkms_nvidia/patches/disable_fstack-clash-protection_fcf-protection.patch:
    - Refresh for 430.34 (LP: #1830961).

 -- Alberto Milone <email address hidden> Wed, 10 Jul 2019 15:27:36 +0200

Assigned to Seth per comment #16.

Status changed to 'Confirmed' because the bug affects multiple users.

The patch that satmandu mentioned in https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/1830961/comments/16 fixes the v5.2.0 mainline kernel build as well.

Can the patch that was applied to the kernel in https://launchpad.net/ubuntu/+source/linux/5.2.0-8.9 please also be applied to the daily mainline builds? (possible duplicate bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1836373 )

This is fixed in the -proposed kernel, so I don't plan to hack virtualbox sources to drop such flag.

I confirm that on Ubuntu Eoan with:
- linux kernel 5.2.0-8.9
- gcc-9 9.1.0-8ubuntu1

I am able to:
- build DPDK 19.05
- install xtables-addons-dkms.

linux (5.2.0-8.9) eoan; urgency=medium

  * linux: 5.2.0-8.9 -proposed tracker (LP: #1835700)

  * Miscellaneous Ubuntu changes
    - [Packaging] replace zfs and spl build with zfs 0.8.1-1ubuntu1
    - SAUCE: test_bpf: remove expected fail for Ctx heavy transformations test on
      s390
    - SAUCE: add -fcf-protection=none to retpoline flags
    - SAUCE: usbip: ensure strings copied using strncpy are null-terminated
    - SAUCE: usbip: add -Wno-address-of-packed-member to EXTRA_CFLAGS
    - SAUCE: perf jvmti: ensure strncpy result is null-terminated
    - update dkms package versions
    - add removed zfs modules to modules.ignore

  [ Upstream Kernel Changes ]

  * Rebase to v5.2

 -- Seth Forshee <email address hidden> Mon, 08 Jul 2019 07:13:41 -0500

If the new kernel works per comment #25 then no specific DKMS packages need fixing.

Any chance someone could get the recipes fixed for the mainline builds, which also need the -fcf-protection=none retpoline flags added since they're being built using "schroot -c eoan-amd64"?

(I'm not sure who needs to be looped in for that to happen.)

+1 for fixing mainline kernel builds.. been broken for almost two weeks!

maybe Seth Forshee <email address hidden> is the guy to ask

Several posters are asking for the mainline builds to be fixed. The mainline builds are an exact reflection of the upstream git master repositories. Therefore they have to wait for Seth's patch submission, which has been picked up and accepted by the kbuild team, to propagate it's way to the mainline upstream git branch. Earlier today, a kernel 5.3-rc1 merge request was issued by the kbuild team [1]. Once that branch has been "pulled" into the upstream mainline, the Ubuntu mainline daily compiles should start working the next day. In about 8 days there will be kernel 5.3-rc1, which should have the patch.

[1] https://www.spinics.net/lists/linux-kbuild/msg22526.html

Other references:

https://kernel.ubuntu.com/~kernel-ppa/mainline/daily/
https://www.spinics.net/lists/linux-kbuild/msg22298.html
https://patchwork.kernel.org/patch/11037379/

Thanks all!

Mainline builds of daily and 5.3-rc1 appear to have been generated for amd64 last night.

It is nice to see this resolved for 5.3-rc1.

Excuse my ignorance of the process, but will this patch be (perhaps automagically) backported so the next iteration of the 5.2 (and maybe 5.1) stable kernels for amd64 and i386 also get generated successfully?

I think if you need to build any older kernels then the simplest answer is the workaround in comment #8.

Just for the record, this still breaks upgrades from 19.04 (I updated today) because dkms tries to build things (nvidia and virtualbox here) for the 19.04 kernel and fails.

Hmm, I guess we should SRU the --fcf-protection=none change to the 19.04 and 18.04 kernels in that case. I will get this done.

Redditer trying an upgrade from disco -> eoan reported broken upgrade due to this, and like in comment #34 I can reproduce this still.

So I think this a fix backported to 19,04 kernel is definitely required before we release 19.10.

This bug is linked in the backport-iwlwifi-dkms SRU for eoan but it does not provide the required SRU information (Impact, Test Case, Regression Potential [1]). Could you please update the bug description? Thank you!

[1] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

Copied SRU justification from bug 1848922.

Hello satmandu, or anyone else affected,

Accepted backport-iwlwifi-dkms into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/backport-iwlwifi-dkms/7906-0ubuntu2~19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verified version 7906-0ubuntu2~19.10.1 against following kernel headers:

  $ dkms status
  backport-iwlwifi, 7906, 4.15.0-1059-oem, x86_64: installed
  backport-iwlwifi, 7906, 5.0.0-1025-oem-osp1, x86_64: installed
  backport-iwlwifi, 7906, 5.3.0-23-generic, x86_64: installed

The verification of the Stable Release Update for backport-iwlwifi-dkms has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package backport-iwlwifi-dkms - 7906-0ubuntu2~19.10.1

---------------
backport-iwlwifi-dkms (7906-0ubuntu2~19.10.1) eoan; urgency=low

  * debian/patches/0006-NOUPSTREAM-backport-rename-ktime_get_boot_ns-for-v5..patch
    - fix build against v5.3. (LP: #1848922)
  * debian/patches/0005-Makefile.kernel-pass-fno-stack-clash-protection-and-.patch
    - add -fcf-protection=none when using retpoline flags. (LP: #1830961)
    - drop -fno-stack-clash-protection for it's no longer necessary.

 -- You-Sheng Yang <email address hidden> Mon, 21 Oct 2019 20:22:27 +0800

Status changed to 'Confirmed' because the bug affects multiple users.

This fixed my problem. I am reinstalling VirtualBox 6.1.4 and the command "sudo ln -fs gcc-8 /usr/bin/gcc" solve the problem.