text VTs are unavailable on desktop after upgrade to Ubuntu 17.10

This change was made by a bot.

The problem is reproducible if I boot with vt.handoff disabled.

It may be worth mentioning that this system has full-disk encryption, therefore plymouthd is started from the initramfs, not from the root disk.

If I boot without the external monitor attached, my text VTs come up correctly.

They continue to be available if I then dock the laptop.

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.14 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc5

I have confirmed that this issue is reproducible with the 4.13.4-041304-generic mainline build, corresponding to 4.13.0-16.

Will test 4.14-rc5 now.

With 4.14-rc5, the bug *is* fixed. Booted with external monitor, with vt.handoff=1 set, with plymouth started from the initramfs, and with wayland configured instead of X. My VTs work.

Let the bisect begin.

(I mean vt.handoff=7)

4.14rc1 works.

4.13.8 doesn't work.

We can't bisect between 4.13.8 and 4.14-rc1 because they are not linear versions. The bug could have been introduced by a 4.13 stable update.

Can you confirm 4.13 final does have the bug? If it does, I'll start a reverse bisect between 3.13 final and 4.14-rc1.

4.13 final is available from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13/

I started a "Reverse" kernel bisect between v4.13 final and v4.14-rc1. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
aae3dbb4776e7916b6cd442d00159bea27a695c1

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

v4.13 also confirmed bad.

aae3dbb4776e7916b6cd442d00159bea27a695c1 is good.

I built the next test kernel, up to the following commit:
bf1d6b2c76eda86159519bf5c427b1fa8f51f733

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

bf1d6b2c76eda86159519bf5c427b1fa8f51f733 is good.

edc2988c548db05e33b921fed15821010bc74895 is bad, though the symptoms are different. (no output on the external monitor from plymouth; plymouth itself seems to crash and the VT is in text mode, so text input results in characters being shown in the upper lefthand corner over the plymouth background; since plymouth isn't accepting input, I'm not able to boot all the way to userspace and check whether the text VTs are accessible after X or wayland has started, since with plymouth broken the root disk does not unlock.)

04759194dc447ff0b9ef35bc641ce3bb076c2930 is bad; same symptoms as edc2988c548db05e33b921fed15821010bc74895.

a7093ba162d942e29ae27a64f8432db65f7f21ab is bad; same symptoms as 04759194dc447ff0b9ef35bc641ce3bb076c2930 and edc2988c548db05e33b921fed15821010bc74895.

1a3b85ea36d38d5732fdd86b321b10bcaeb53512 is bad (idem.)

f1f537c2e7f565a808623d3244ea73a65839240e is bad (idem.)

8162ae5c47977c3f13910b5c1718081d30e8a65a is bad.

11b86a84bc535a602fcf72ba6b3aa4eaa748764f is bad.

2ce8008711e4837c11e99a94df55406085d0d098 is bad.

31cb9a8575ca04f47ea113434d4782b695638b62 is bad (now back to the original symptoms: input works in plymouth, no external monitor output from plymouth, text VTs don't render).

I built the next test kernel, up to the following commit:
3840ed9548f778717aaab5eab744da798c3ea055

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

3840ed9548f778717aaab5eab744da798c3ea055 is bad.

I built the next test kernel, up to the following commit:
9ea859b11b251df57df5bb9fc146ac014b31071c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

9ea859b11b251df57df5bb9fc146ac014b31071c bad. (One behavior difference: text VTs now give no video signal at all, instead of leaving the graphics on screen. Still counts as bad.)

28eb51f7468a43769bd9dca19a54d97ec7a447ed bad.

e63a94f12b5fc67b2b92a89d4058e7a9021e900e is good.

I started a new "Reverse" bisect of just the merge using:

# good: [1a3b85ea36d38d5732fdd86b321b10bcaeb53512] Merge tag 'usb-4.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb

# bad: [3840ed9548f778717aaab5eab744da798c3ea055] tty: goldfish: Implement support for kernel 'earlycon' parameter

I built the first test kernel, up to the following commit:
ef954844c7ace62f773f4f23e28d2d915adc419f

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

ef954844c7ace62f773f4f23e28d2d915adc419f bad.

ec085c5a51b768947ca481f90b66653e36b3c566 bad.

I built the next test kernel, up to the following commit:
cdb939456f81878cda1f31ba0ef1a3b1d7be07e8

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

cdb939456f81878cda1f31ba0ef1a3b1d7be07e8 is bad.

I built the next test kernel, up to the following commit:
2ce8008711e4837c11e99a94df55406085d0d098

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

I built the next test kernel, up to the following commit:
d01c3289e7d68162e32bc08c2b65dd1a216a7ef8

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

2ce8008711e4837c11e99a94df55406085d0d098 bad

d01c3289e7d68162e32bc08c2b65dd1a216a7ef8 bad

I built the next test kernel, up to the following commit:
ae28d7402a7e7cf3dd0da1f2eb6a52b337873e08

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

ae28d7402a7e7cf3dd0da1f2eb6a52b337873e08 bad

I built the next test kernel, up to the following commit:
7157d2be23da9f8860c69e2b79184a4e02701dad

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

I also built a test kernel up to commit 3840ed9 again. This kernel should fix the bug since it's the last commit added by by the merge base:

commit e63a94f12b5fc67b2b92a89d4058e7a9021e900e
tree 9fddf35df3289bd41c2310937bf2faed4d036603
parent 1a3b85ea36d38d5732fdd86b321b10bcaeb53512
parent 3840ed9548f778717aaab5eab744da798c3ea055
author Linus Torvalds <email address hidden> 1504632648 -0700
committer Linus Torvalds <email address hidden> 1504632648 -0700
mergetag object 3840ed9548f778717aaab5eab744da798c3ea055
 type commit
 tag tty-4.14-rc1
 tagger Greg Kroah-Hartman <email address hidden> 1504542898 +0200

 TTY/Serial updates for 4.14-rc1

The merge has commit 1a3b85ea as the first parent and commit 3840ed9 as the second parent, so all the commits added by the merge should be between these two commits.

git describe --contains 1a3b85e
v4.14-rc1~150

git describe --contains 3840ed9
v4.14-rc1~149^2

And the merge:
git describe --contains e63a94f
v4.14-rc1~149

If a test kernel up to commit e63a94f fixes the bug, then a test kernel up to commit 3840ed9 should as well. If it does not, then I'll have to dig much deeper.

Commit e63a94f is a child of commit 3840ed9:

commit 3840ed9548f778717aaab5eab744da798c3ea055
Author: Miodrag Dinic <email address hidden> 2017-08-29 09:53:20
Committer: Greg Kroah-Hartman <email address hidden> 2017-08-31 12:58:45
Parent: 7157d2be23da9f8860c69e2b79184a4e02701dad (tty: goldfish: Use streaming DMA for r/w operations on Ranchu platforms)
Child: e63a94f12b5fc67b2b92a89d4058e7a9021e900e (Merge tag 'tty-4.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty)
Branches: master, remotes/next/akpm, remotes/next/akpm-base, remotes/next/master, remotes/next/stable, remotes/origin/master
Follows: v4.13-rc5
Precedes: v4.14-rc1

    tty: goldfish: Implement support for kernel 'earlycon' parameter

Test kernels can be found at:

3840ed9: http://kernel.ubuntu.com/~jsalisbury/lp1724911/3840ed9/
e63a94f: http://kernel.ubuntu.com/~jsalisbury/lp1724911/e63a94f/

7157d2b: http://kernel.ubuntu.com/~jsalisbury/lp1724911/7157d2b/

7157d2be23da9f8860c69e2b79184a4e02701dad is bad.

3840ed9548f778717aaab5eab744da798c3ea055 is bad.

and e63a94f12b5fc67b2b92a89d4058e7a9021e900e is confirmed good. So it's not just a question of me having screwed up the testing / reporting of that commit previously.

Thanks for the testing, Steve. I'll look in to this further.

Per your IRC posting, I restarted a reverse bisect using e63a94f12b5fc67b2b92a89d4058e7a9021e900e .. 1a3b85ea36d38d5732fdd86b321b10bcaeb53512.

The first SHA1 requested was ec085c5a51b768947ca481f90b66653e36b3c566. I re-used all of your testing results from the bug comments and ended up with the same results again.

ec085c5a51b768947ca481f90b66653e36b3c566 (rebased) is bad.

cdb939456f81878cda1f31ba0ef1a3b1d7be07e8 (rebased) is bad.

2ce8008711e4837c11e99a94df55406085d0d098 (rebased) is bad.

d01c3289e7d68162e32bc08c2b65dd1a216a7ef8 (rebased) is bad.

ae28d7402a7e7cf3dd0da1f2eb6a52b337873e08 (rebased) is bad.

7157d2be23da9f8860c69e2b79184a4e02701dad (rebased) is bad. Considering there is nothing left in the diff between the rebased 7157d2be23da9f8860c69e2b79184a4e02701dad and e63a94f12b5fc67b2b92a89d4058e7a9021e900e which tested good, this is starting to look like the difference is something other than the tree. Toolchain? kernel config options?

(when I say "nothing left", I meant "nothing left of interest")

3840ed9548f778717aaab5eab744da798c3ea055 (rebased), whose content is equivalent to e63a94f12b5fc67b2b92a89d4058e7a9021e900e, also now fails.

So whatever is different here between the working and failing builds has to do with more than just the git content.

Joe, can you give me a step-by-step guide for how you prepare each build from git during the bisect? I've definitely missed something in my builds, because I've just looked at my initrd for the latest build and find that it's missing the i915.ko module... and that this module isn't even included in the linux-image-4.13.0-041300-generic package. So all of the test results from kernels I built are certainly suspect.

Just found out that i915.ko is winding up in linux-image-extra. That didn't match my understanding of extras, so I haven't been installing that package... time to go back to the beginning.

edc2988c548db05e33b921fed15821010bc74895 is the next kernel I will be retesting, from the original v4.13..v4.14rc1 bisect. That's the first point at which I saw symptoms at boot matching what I saw with the most recent kernel for which I definitely was missing the i915.ko module.

edc2988c548db05e33b921fed15821010bc74895 is good.

e5fa05b96b02f45a91a364d2121451a4da05cd84 is good.

b9c55b6e2cc4369b0688961fa5de0e057f3ec0c4 is good.

9f15a4ab19ab33658dbc9fd37be5210e8c1ac622 is good.

e6090cc9be15f8e3a5e736d427186e39fea7cceb is good.

2bf5ccc25c5bc3c8eaf30c589f7d3a5e7490d3b9 is good.

22a884cfe5a49b6fd63f10ff137906028b4cf923 is good.

bb2eaba6458ace16f7e3504de8788374cb42b43d is good.

b6dcaaac44746cf32ee489639b06d3668e473386 is good.

00a9121b8698ddf7fcb18e107405afacc35f748a is good.

fc5e9d63a8db40e9e3a3b180be33d44af8a3cee8 is good.

An Artful test kernel with a pick of commit fc5e9d63a8db is posted to:
http://kernel.ubuntu.com/~jsalisbury/lp1724911

the 4.13.0-16-generic plus fc5e9d63a8db does not fix it.

4.13.4 + fc5e9d63a8db doesn't work; git bisect shows no other commits between v4.13 and fc5e9d63a8db. Now trying a regular bisect between these two revisions instead of a reverse bisect.

34c8ea400ff6 is good.

8c27cb3566762613a23c080e3db7d0501af9a787 is good.

af3c8d98508d is good.

96080f697786e0a30006fcbcc5b53f350fcb3e9f is bad.

6735a1971a00a29a96aa3ea5dc08912bfee95c51 is bad.

58c7ffc0747a3a9145629d4966291f0586703767 is good.

311ac00deff520820d198cda141109f2a5655d4a is bad.

fb4e3beeffa47619985f190663c6ef424f063a22 is good.

bd664f6b3e376a8ef4990f87d08271cc2d01ba9a is good.

4ca6df134847a6349620b485a3e63f00fb3bfad8 is good.

eafbc20701243d21989c9c9aff5e5d9488e1890a is bad.

4ec654bf3a63d503e3c5336eade5c369ae17db56 is good.

0de98709896d9c02ce3121ec3afb524253a5853f is bad.

c379b897ba1a7f786419e8c36a3438ce856f016a is good.

7581d5ca2bb269cfc2ce2d0cb489aac513167f6b is bad.

04941829b0049d2446c7042ab9686dd057d809a6 is good.

7581d5ca2bb269cfc2ce2d0cb489aac513167f6b is the first bad commit

Thanks for doing all this work, Steve!

I built an Artful test kernel with a revert of commit 7581d5ca2bb269cf. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1724911

I'll also take a look at commits in 4.14-rc1 similar to this one, and see if the fix sticks right out.

Hi Joe,
The fix in 4.14 that's missing from 4.13.4 is 88be58be886f1215cc73dc8c273c985eecd7385c. Can you cherry-pick this into the next bionic upload? (Assuming it's not already there by now due to the time elapsed since we started bisecting ;)

and when I say next bionic upload, I mean next artful upload.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

$ uname -a
Linux virgil 4.13.0-38-generic #43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$

Booted with my external monitor plugged in, and Ctrl-Alt-F2 still gets me to a VT.

This bug was fixed in the package linux - 4.13.0-38.43

---------------
linux (4.13.0-38.43) artful; urgency=medium

  * linux: 4.13.0-38.43 -proposed tracker (LP: #1755762)

  * Servers going OOM after updating kernel from 4.10 to 4.13 (LP: #1748408)
    - i40e: Fix memory leak related filter programming status
    - i40e: Add programming descriptors to cleaned_count

  * [SRU] Lenovo E41 Mic mute hotkey is not responding (LP: #1753347)
    - platform/x86: ideapad-laptop: Increase timeout to wait for EC answer

  * fails to dump with latest kpti fixes (LP: #1750021)
    - kdump: write correct address of mem_section into vmcoreinfo

  * headset mic can't be detected on two Dell machines (LP: #1748807)
    - ALSA: hda/realtek - Support headset mode for ALC215/ALC285/ALC289
    - ALSA: hda - Fix headset mic detection problem for two Dell machines
    - ALSA: hda - Fix a wrong FIXUP for alc289 on Dell machines

  * CIFS SMB2/SMB3 does not work for domain based DFS (LP: #1747572)
    - CIFS: make IPC a regular tcon
    - CIFS: use tcon_ipc instead of use_ipc parameter of SMB2_ioctl
    - CIFS: dump IPC tcon in debug proc file

  * i2c-thunderx: erroneous error message "unhandled state: 0" (LP: #1754076)
    - i2c: octeon: Prevent error message on bus error

  * hisi_sas: Add disk LED support (LP: #1752695)
    - scsi: hisi_sas: directly attached disk LED feature for v2 hw

  * EDAC, sb_edac: Backport 1 patch to Ubuntu 17.10 (Fix missing DIMM sysfs
    entries with KNL SNC2/SNC4 mode) (LP: #1743856)
    - EDAC, sb_edac: Fix missing DIMM sysfs entries with KNL SNC2/SNC4 mode

  * [regression] Colour banding and artefacts appear system-wide on an Asus
    Zenbook UX303LA with Intel HD 4400 graphics (LP: #1749420)
    - drm/edid: Add 6 bpc quirk for CPT panel in Asus UX303LA

  * DVB Card with SAA7146 chipset not working (LP: #1742316)
    - vmalloc: fix __GFP_HIGHMEM usage for vmalloc_32 on 32b systems

  * [Asus UX360UA] battery status in unity-panel is not changing when battery is
    being charged (LP: #1661876) // AC adapter status not detected on Asus
    ZenBook UX410UAK (LP: #1745032)
    - ACPI / battery: Add quirk for Asus UX360UA and UX410UAK

  * ASUS UX305LA - Battery state not detected correctly (LP: #1482390)
    - ACPI / battery: Add quirk for Asus GL502VSK and UX305LA

  * support thunderx2 vendor pmu events (LP: #1747523)
    - perf pmu: Extract function to get JSON alias map
    - perf pmu: Pass pmu as a parameter to get_cpuid_str()
    - perf tools arm64: Add support for get_cpuid_str function.
    - perf pmu: Add helper function is_pmu_core to detect PMU CORE devices
    - perf vendor events arm64: Add ThunderX2 implementation defined pmu core
      events
    - perf pmu: Add check for valid cpuid in perf_pmu__find_map()

  * lpfc.ko module doesn't work (LP: #1746970)
    - scsi: lpfc: Fix loop mode target discovery

  * Ubuntu 17.10 crashes on vmalloc.c (LP: #1739498)
    - powerpc/mm/book3s64: Make KERN_IO_START a variable
    - powerpc/mm/slb: Move comment next to the code it's referring to
    - powerpc/mm/hash64: Make vmalloc 56T on hash

  * ethtool -p fails to light NIC LED on HiSilicon D05 systems (LP: #1748567)
    - net...