Xenial server 16.04.x will have a black screen after PXE installation

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1646277/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

Are there any call traces or errors in the logs after the installation? Once on the the other vt, does the system appear to function properly? If so, it might be good to test some other kernels to see if this is a regression, and test the latest mainline kernel to see if this is already fixed:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9/

Hi
After update mainline kernel ,has fixed this problem
linux-headers-4.9.0-040900-generic_4.9.0-040900.201612111631_amd64.deb
linux-image-4.9.0-040900-generic_4.9.0-040900.201612111631_amd64.deb

So..this issue, kernel team will release next Ubuntu 16.04.2 version with fix?

Thanks for testing. The bug will only be fixed if the commit in mainline that fixes the bug was also cc'd to stable.

We should probably perform a "Reverse" kernel bisect to identify the exact commit that fixes the bug. Then we will know if it in fact was cc'd to stable. If it was not, we can request it in stable and SRU it to Ubuntu.

We first need to identify the last bad kernel and first good one. Can you test the v4.9-rc1 kernel to try and narrow the affect versions down further. It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9-rc1/

If v4.9-rc1 is also good, we would want to test 4.8 final. If it is bad, we would want to test some of the newere 4.9 release candidates.

Thanks in advance!

After update it OK,can boot into tty1 during boot into OS
linux-headers-4.9.0-040900rc1-generic_4.9.0-040900rc1.201610151630_amd64.deb
linux-image-4.9.0-040900rc1-generic_4.9.0-040900rc1.201610151630_amd64.deb

Status changed to 'Confirmed' because the bug affects multiple users.

Can you next test the 4.8 final kernel? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8/

It might also be worthwhile to test the latest upstream 4.4 kernel. This will tell us if the fix was already sent to upstream stable. It is available from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.40/

After update test the 4.8 was pass,it is work in tty1
Also update 4.4.40 same PASS

It is good news that 4.4.40 does not exhibit this bug. That might indicate the fix for the bug has already made it to upstream stable.

The current Ubuntu kernel in -proposed has up to the 4.4.35 updates. Can you give that kernel a test to see if the fix was applied to that version? It can be downloaded from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.35/

That v4.4.35 was PASS

Would it be possible for you to test the -proposed kernel and post back if it resolves this bug?
See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed.

Thank you in advance!

Already enabled proposed ,what are need to install kernel packages and update?

You should only have to run the following from a terminal:

sudo apt-get update
sudo apt-get dist-upgrade

Then reboot and run uname -a to ensure you are booted into the -proposed kernel.

OK,The OS can boot into the proposed kernel after update and dist-updgrade

Thanks for testing. The kernel in -updates(4.4.0-59) now has the 4.4.35 updates. Can you apply the latest updates without using -proposed and confirm this bug is resolved?

OS installer via PXE and then update to 4.4-59 kernel ,that still would stop at tty7 and I haven't looked into why it doesn't switch straight to a tty1 though
 update command as:
 sudo apt-get update
 sudo apt-get dist-upgrade

According to Leo, the issue is reproducible with 4.4.0-59. Changing status to confirmed.

The Xenial kernel is now at 4.4.0-64.85. Can you apply the latest updates and see if the bug still exists?

After update to 4.4.0-64.85,that still would stop at tty7 and it doesn't switch straight to a tty1

Would you say this bug is now resolved?

After update to 4.4.0-64.85, the bug still exist.
It would stop at tty7 and it doesn't switch straight to a tty1.

It sounds like the upstream 4.4 kernel does not have the bug per comments #10 and #12.

It appears this bug may be caused by a SAUCE patch that is not in the upstream kernel. To confirm that, can you test the upstream 4.4.44 kernel, which is what 4.4.0-64.85 is based on?

If the bug does not exist in 4.4.44, we can perform a kernel bisect with the Ubuntu kernels to find that SAUCE commit that introduced this bug.

The 4.4.44 kernel can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.44/

Update to 4.4.44 kernel, the bug didn't happened.

It will import into which version of the official kernel or need to manually update patch kernal?

It sounded like this bug was not happening with -proposed in comment #16. Is that correct?

To perform a kernel bisect, we would need to know the last Ubuntu kernel version that was working and the first that is not. Do we know that information? If not, I can post some links to prior kernel versions for testing.

Yes, the bug was not happening with -proposed in comment #16.

On official kernel 4.4.0-31, we use below command to upgrade the newest kernel. The bug still happened.
sudo apt-get update
sudo apt-get dist-upgrade

The kernel in -updates is now 4.4.0-72.93. Are you only getting up to version 4.4.0-31 when you apply the updates?

On official kernel 4.4.0-31, we use below command upgrade to 4.4.0-72.93. The bug still exist.
sudo apt-get update
sudo apt-get dist-upgrade

Thanks for the response. Do you recall what kernel version you were running with the kernel in comment #16? The current kernel in -proposed is 4.4.0-73.94. However, comment #16 was back in January.

Hi
We have tried more tested and verified,so far still not resolved this issue and but we have workaround them commented "# set vt_handoff=vt.handoff=7"in grub.cfg ,It would can switch straight tty1.however,Do you known what's mean about gfxmode ?
This issue always happened in PXE OS intaller
function gfxmode {
        set gfxpayload="${1}"
        if [ "${1}" = "keep" ]; then
               # set vt_handoff=vt.handoff=7
        else
                set vt_handoff=
        fi

Reply to comment #32.
We didn't remember what kernel version we were running with the kernel in comment #16.

It sounds like the best approach is to bisect through the Ubuntu Xenial kernels. We need to identify the last kernel version that did not have the bug and the first that did. Can you test the following early Xenial kernel:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414

Note, with this test kernel you need to install both the linux-image and linux-image-extra .deb packages.

We have already try many kernel version to check this bug.
And finally find 4.4.44 kernel didn't happened.
Please tell us which kernel version will be imported to fix the bug.
Thanks.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.44/

From all of the prior testing, this bug is happening only with Ubuntu kernels and not in any of the upstream kernels, such as 4.4.44.

We need to bisect through some older Ubuntu kernels to find the last good version and first bad version. This is the request posted in comment #35, to test this early version:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414

It might also be good to test the latest 17.04 kernel to see if the bug is already fixed there. It can be downloaded from:

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/12405875

The link which you provide in comment #37, we can not find the link to download "amd64 build of linux 4.4.0-20.36 in ubuntu xenial RELEASE".

We try 17.04 kernel, the bug still not fixed.

Could you help to reproduce the bug with PXE server on your side? Thanks.

Is there anyone can help me to answer this problem?
Thanks.

@nancy-hsu
Scroll down on https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414 you can find download link for
linux-image-4.4.0-20-generic_4.4.0-20.36_amd64.deb and linux-image-extra-4.4.0-20-generic_4.4.0-20.36_amd64.deb
Here are the links for your quick reference:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414/+files/linux-image-4.4.0-20-generic_4.4.0-20.36_amd64.deb
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/9583414/+files/linux-image-extra-4.4.0-20-generic_4.4.0-20.36_amd64.deb

Thanks for your testing so far. All the upstream kernel (mainline) you have tested (4.9, 4.8, 4.4.40, 4.4.35 and 4.4.44) are fine, the two Ubuntu kernels 4.4.0-59 (based on upstream 4.4.35) and 4.4.0-64 (based on upstream 4.4.44) the bug can be reproduced. As #37, we need to find a working Ubuntu kernel to identify where the bug is first introduced, hence the request to test 4.4.0-20.

Download file:
linux-image-4.4.0-20-generic_4.4.0-20.36_amd64.deb
linux-image-extra-4.4.0-20-generic_4.4.0-20.36_amd64.deb

This issue can be reproduced on 4.4.0-20 kernel.
OS installer via PXE and then update to 4.4.0-20 kernel, that still would stop at tty7 and it doesn't switch straight to a tty1.

Can you see if this bug still exists with the 16.04.3 release? I'm thinking it should not, since prior testing showed that 4.9 and newer kernels did not have the bug.

16.04.3 is available from:
http://releases.ubuntu.com/16.04.3/

This bug still exists with 16.04.3.
Please help to check it, thanks.

Hi Nancy,

I'd like to confirm the symptoms of this bug. For PXE installs only, you get through all the installation steps and then the screen goes black after a reboot? You can then switch to tty1 and access command prompt?

Hi Joseph,

Yes, the symptoms are correct.
1. For PXE installs only.
2. Get through all the installation steps and then the screen goes black after a reboot.
3. Then can switch to tty1 and access command prompt.

And you are also able to reproduce this bug with the various test kernels I post? If so, we should be able to narrow this down without having to install every time.

Please take a look with #41.
All the upstream kernel (mainline) we have tested (4.9, 4.8, 4.4.40, 4.4.35 and 4.4.44) are fine.
The two Ubuntu kernels 4.4.0-59 (based on upstream 4.4.35) and 4.4.0-64 (based on upstream 4.4.44) the bug can be reproduced.
Finally we test kernel 4.4.0-20 the bug can also be reproduced.

Can you next see if this bug existed in 4.4.0-5:
https://launchpad.net/ubuntu/+source/linux/4.4.0-5.20/+build/8999514

Yes, the bug existed in 4.4.0-5.
Reproduce step:
1. PXE install Ubuntu 16.04.3.
2. Downgrade kernel to 4.4.0-5.
3. The screen goes black after a reboot.

Thanks for testing. Can you test this 3.19 kernel:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/13366909

This may not be a regression afterall, so we should figure that out.

The bug didn't happened on kernel 3.19.0-93.
Test step:
1. PXE install Ubuntu 16.04.3.
2. Downgrade kernel to 3.19.0-93.
3. The screen goes normally after a reboot.

Thanks, can you next test 4.2.0-16:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/8099555

Also, this bug might be very related to graphics hw. Some server boards with remote management capabilities use something like Matrox or similarly exotic. In the past we sometimes saw problems switching framebuffer devices. Things that maybe can help are changing grub to console mode or adding nomodeset into kernel parameters. (If you change GRUB setting, be sure to run "sudo update-grub" before rebooting).

The bug existed in 4.2.0-16.
Reproduce step:
1. PXE install Ubuntu 16.04.3.
2. Downgrade kernel to 4.2.0-16.
3. The screen goes black after a reboot.

I built a 4.1 based test kernel. Can you see if this kernel exhibits the bug? It can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1646277/

The bug existed in 4.1.0-3 test kernel.
 Reproduce step:
 1. PXE install Ubuntu 16.04.3.
 2. Downgrade kernel to 4.1.0-3.
 3. The screen goes black after a reboot.

I built a 4.0 based test kernel. Can you see if this kernel exhibits the bug? It can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1646277/

The bug didn't happened on test kernel 4.0.0-4.
 Test step:
 1. PXE install Ubuntu 16.04.3.
 2. Downgrade kernel to 4.0.0-4.
 3. The screen goes normally after a reboot.

I started a kernel bisect between v3.7 final and v3.8-rc1. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
106c1ee52e54ee2a695dddfc86b4aed32f65894b

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

The bug didn't happened on test kernel 4.1.0-1.
 Test step:
 1. PXE install Ubuntu 16.04.3.
 2. Downgrade kernel to 4.1.0-1.
 3. The screen goes normally after a reboot.

I built the next test kernel, up to the following commit:
d9e5b220f962910fb37133bde7e857ce478daf3e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

The bug didn't happened on test kernel 4.1.0-1.
commit: d9e5b220f962910fb37133bde7e857ce478daf3e

I built the next test kernel, up to the following commit:
c6ff6112bd446505fb75a70690b953501bd4837f

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

The bug didn't happened on test kernel 4.1.0-3.
commit: c6ff6112bd446505fb75a70690b953501bd4837f

I built the next test kernel, up to the following commit:
5f54bebd73a3903514d7ab786085b21ea757f92b

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Hide

The bug didn't happened on test kernel 4.1.0-3.
commit: 5f54bebd73a3903514d7ab786085b21ea757f92b

I built the next test kernel, up to the following commit:
dddc618817c4eab6c1b3b1632fc47f2ed6c75fdc

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

The bug didn't happened on test kernel 4.1.0-3.
commit: dddc618817c4eab6c1b3b1632fc47f2ed6c75fdc

I built the next test kernel, up to the following commit:
ae827479a0a7a645ea96e68f5391fa71ad1e8c65

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

The bug didn't happened on test kernel 4.1.0-3.
commit: ae827479a0a7a645ea96e68f5391fa71ad1e8c65

I built the next test kernel, up to the following commit:
369e26773363a6c11186e2c5ec4d83a7c89fee9c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

The bug existed in 4.1.0-3 test kernel.
commint: 369e26773363a6c11186e2c5ec4d83a7c89fee9c
  Reproduce step:
  1. PXE install Ubuntu 16.04.3.
  2. Downgrade kernel to 4.1.0-3.
  3. The screen goes black after a reboot.

I built a test kernel with a revert of commit 369e26773363a6c11186e2c5ec4d83a7c89fee9c.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1646277

Can you test that kernel and report back if it has the bug or not?

The bug existed in 4.1.0-3 test kernel.
 commint: 369e26773363a6c11186e2c5ec4d83a7c89fee9c.(#73)
   Reproduce step:
   1. PXE install Ubuntu 16.04.3.
   2. Downgrade kernel to 4.1.0-3.
   3. The screen goes black after a reboot.

With a current kernel, can you try changing grub to console mode or adding nomodeset into kernel parameters.

After you change a GRUB setting, be sure to run "sudo update-grub" before rebooting.

Hi Joseph,

With a current kernel, that adding nomodeset into kernel parameters.
After change a GRUB setting then run "sudo update-grub" before rebooting.

The bug still existed.
The screen goes black after a reboot.

Did the bug go away with the linux-image-4.1.0-3-generic_4.1.0-3.3~lp1646277Commit369e26773Reverted_amd64.deb kernel posted in comment #73?

The bug still existed with the linux-image-4.1.0-3-generic_4.1.0-3.3~lp1646277Commit369e26773Reverted_amd64.deb kernel posted in comment #73.

The bisect must have gone wrong. It might not be possible to perform a bisect in this case. I think what would be best is to post all the HW info into the bug. Then I can review the commits between Ubuntu-4.0.0-4.7 and Ubuntu-4.1.0-3.3 to see what could be relevant.

Can you run the following command and then attach the report.1646277 file:
apport-bug --save /tmp/report.1646277 linux

I am still hitting this issue on the latest HWE as well as the normal kernel on 16.04. This is a major blocker for automating installs.

Got it to work by following instructions mentioned on this bug report:

https://bugs.launchpad.net/ubuntu/+source/grub-installer/+bug/1616258