18.04 daily installer fails missing kernel

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1746807

18.04 daily:
Lubuntu Alternate 20180130 = works
Ubuntu Server 20180201.2 [iso date: 2018-02-01 06:41] = works
Ubuntu Desktop 20180201 [iso date: 2018-02-01 07:53] = works
Lubuntu Alternate 20180201 [iso date: 2018-02-01 16:32] = fails <--

Lubuntu Alternate 20180202 = fails
Lubuntu Alternate 20180203 = fails

Lubuntu Alternate 20180203 = fails

Confirmed.

Status changed to 'Confirmed' because the bug affects multiple users.

Lubuntu Alternate 20180203.1 = fails

A syslog would be nice.

Attached is the log.

I think this may be a clue:

Stat failed for /media/cdrom/dists/bionic/main/dist-upgrader/binary-all/Packages - stat (2: No such file or directory)
Feb 4 13:39:32 base-installer:
Feb 4 13:39:32 base-installer: E
Feb 4 13:39:32 base-installer: :
Feb 4 13:39:32 base-installer: No CD-ROM could be auto-detected or found using the default mount point.
Feb 4 13:39:32 base-installer:
Feb 4 13:39:32 base-installer:
Feb 4 13:39:32 base-installer: You may try the --cdrom option to set the CD-ROM mount point.

There is a "cdrom" (well a virtual one under VirtualBox).

In another terminal...I looked and there is no /media/cdrom directory

However there is a /media/dists/bionic/main/dist-upgrader/binary-all/Packages
note: the above path is /media/dists/... and not /media/cdrom/dists...

Is the changed mount point breaking it finding the packages?

p.s. Adam, thank you for your feedback.

Lubuntu Alternate 20180204 = fails

Lubuntu Alternate 20180205 = fails

Lubuntu Alternate 20180206 = fails

Lubuntu Alternate 20180208 = fails

Lubuntu Alternate 20180209 = fails

Lubuntu Alternate 20180210 = fails

Lubuntu Alternate 20180211 = fails

I'm not sure I can figure this one out... passing to Foundations, they know d-i better than I do.

Lubuntu Alternate 20180212 = fails

Lubuntu Alternate 20180213 = fails

Lubuntu Alternate 20180214 = fails

sysloggood.log = 20180131
syslogbad.log = 20180214

I stripped the date prefix from the two logs to simplify comparison.
e.g. cat syslog | sed "s/Feb 14 ..:..:..//" > sysloggood.log

sysloggood.log = 20180131
syslogbad.log = 20180214

I stripped the date prefix from the two logs to simplify comparison.
e.g. cat syslog | sed "s/Feb 14 ..:..:..//" > sysloggood.log

I noticed in a few places it has debconf entries for .. "load-cdrom, load-iso" .. but in the bad logs the "load-cdrom" part is missing.

This may be relevant as the cdrom source is "missing" later in the install:

e.g. GOOD:
 debconf: --> SUBST anna/choose_modules CHOICES-C choose-mirror, download-installer, fdisk-udeb, iso-scan, load-cdrom, load-iso, load-media, lowmem, mouse-modules-4.13.0-25-generic-di, multipath-modules-4.13.0-25-generic-di, network-console, oem-config-udeb, openssh-client-udeb, parted-udeb, reiserfsprogs-udeb, rescue-mode, vlan-modules-4.13.0-25-generic-di
 debconf: Adding [CHOICES-C] -> [choose-mirror, download-installer, fdisk-udeb, iso-scan, load-cdrom, load-iso, load-media, lowmem, mouse-modules-4.13.0-25-generic-di, multipath-modules-4.13.0-25-generic-di, network-console, oem-config-udeb, openssh-client-udeb, parted-udeb, reiserfsprogs-udeb, rescue-mode, vlan-modules-4.13.0-25-generic-di]

e.g. BAD:
 debconf: --> SUBST anna/choose_modules CHOICES-C choose-mirror, download-installer, fdisk-udeb, iso-scan, load-iso, load-media, lowmem, mouse-modules-4.13.0-32-generic-di, multipath-modules-4.13.0-32-generic-di, network-console, oem-config-udeb, openssh-client-udeb, parted-udeb, reiserfsprogs-udeb, rescue-mode, vlan-modules-4.13.0-32-generic-di
 debconf: Adding [CHOICES-C] -> [choose-mirror, download-installer, fdisk-udeb, iso-scan, load-iso, load-media, lowmem, mouse-modules-4.13.0-32-generic-di, multipath-modules-4.13.0-32-generic-di, network-console, oem-config-udeb, openssh-client-udeb, parted-udeb, reiserfsprogs-udeb, rescue-mode, vlan-modules-4.13.0-32-generic-di]

note: there is no "load-cdrom" option in the bad one...

---

Good:
 debconf-copydb: Adding [CHOICES-C] -> [choose-mirror, download-installer, fdisk-udeb, iso-scan, load-cdrom, load-iso, load-media, lowmem, mouse-modules-4.13.0-25-generic-di, multipath-modules-4.13.0-25-generic-di, network-console, oem-config-udeb, openssh-client-udeb, parted-udeb, reiserfsprogs-udeb, rescue-mode, vlan-modules-4.13.0-25-generic-di]
 debconf-copydb: Adding [CHOICES] -> [choose-mirror: Choose mirror to install from (menu item), download-installer: Download installer components, fdisk-udeb: Manually partition a hard drive (fdisk), iso-scan: Scan hard drives for an installer ISO image, load-cdrom: Load installer components from CD, load-iso: Load installer components from an installer ISO, load-media: Load installer components from removable media, lowmem: free memory for lowmem install, mouse-modules-4.13.0-25-generic-di: Mouse support, multipath-modules-4.13.0-25-generic-di: DM-Multipath support, network-console: Continue installation remotely using SSH, oem-config-udeb: Prepare for OEM configuration, openssh-client-udeb: secure shell client for the Debian installer, parted-udeb: Manually partition a hard drive (parted), reiserfsprogs-udeb: User-level tools for ReiserFS filesystems, rescue-mode: mount requested partition and start a rescue shell, vlan-modules-4.13.0-25-generic-di: vlan mo...

Lubuntu Alternate 20180215 = fails

From the logs:

 base-installer: This disc is called:
 base-installer: 'Lubuntu 18.04 LTS _Bionic Beaver_ - Alpha amd64 (20180213)'
 base-installer: Copying package lists...
 base-installer: gpgv: Signature made Tue Feb 13 16:36:04 2018 UTC
 base-installer:
 base-installer: gpgv: using RSA key
 base-installer:
 base-installer: gpgv: Good signature from "Ubuntu CD Image Automatic Signing Ke
y (2012) <email address hidden>"
 base-installer:
 base-installer: E
 base-installer: :
 base-installer: Stat failed for /media/cdrom/dists/bionic/main/dist-upgrader/binary-all/Packages - stat (2: No such file or directory)
 base-installer:
 base-installer: E
 base-installer: :
 base-installer: No CD-ROM could be auto-detected or found using the default mount point.
 base-installer:
 base-installer:
 base-installer: You may try the --cdrom option to set the CD-ROM mount point.

What is strange is that I *do* see a dists/bionic/main/dist-upgrader/binary-all/Packages file in the current daily amd64 image (20180215). I don't know why this file exists, given that this directory is not an apt source. I do not know why base-installer is trying to parse it, given that it's not referenced in the Releases file for the CD. I do not know why the stat is reported to fail given that the file clearly exists (and is empty).

Answering any one of these questions is likely to yield a solution for this bug.

Actually, it seems that it's not in the .list file: http://cdimage.ubuntu.com/lubuntu/daily/pending/bionic-alternate-amd64.list

Good question on why it wants this file...

I'll assign this to myself again and see if I can figure it out, but if anyone on Foundations knows the answer, you're welcome to assign it to yourself.

On reflection, I think this is the output from 'apt-cdrom add'. So reassigning to apt.

Are the different bad, good apt versions?

When the install was in progress on the good/at the point of failure on the bad I did:

control-f2 to activate another terminal
chroot /target
apt
apt-cdrom

good 20180130: apt 1.6~alpha 7 (amd64)
bad 20180216 : apt 1.6~alpha 5 (amd64)

So yes, the version of apt/apt-cdrom is different.

p.s. Simon, Steve, Julian thanks for your assistance.

TLDR: try rebuilding the iso downgrading the four packages /pool/main/a/apt/*.deb to 1.6~alpha 5?

Note: I pasted the info backwards...it's 1.6~alpha 5 that's the good version.

As a test...I tried manually downgrading apt*.deb on-the fly:

I created a virtual machine with a second (empty) cdrom drive. Once I got the missing kernel:

I switch to alternate terminal..
chroot /target

"apt-cdrom add" gives same error...

Mounted the GOOD 20180131 iso into the vm as the second cdrom.
mkdir /blah
mount /dev/sr1 /blah
cd /blah/pool/main/a/apt
dpkg -i *.deb
cd /
umount /blah

i.e. force load previous apt and apt-cdrom packages.

apt-cdrom

This no longer gives the error.

Just as a test...

cd /media/cdrom/pool/main/a/apt <-- original bad cdrom
dpkg -i *.deb

i.e. force load bad apt/apt-cdrom packages from latest 20180216

apt-cdrom

This gives the stat no such file or directory error.

So...downgrading the four packages /pool/main/a/apt/*.deb should work around the issue?

Only one package needs to be downgraded for apt-cdrom to work again: libapt-pkg5.0 to version: libapt-pkg5.0_1.6~alpha5

http://changelogs.ubuntu.com/changelogs/pool/main/a/apt/apt_1.6~alpha7ubuntu1/changelog

Links used for testing:
http://launchpadlibrarian.net/353920547/libapt-pkg5.0_1.6~alpha7_amd64.deb
http://launchpadlibrarian.net/353116448/libapt-pkg5.0_1.6~alpha6_amd64.deb
http://launchpadlibrarian.net/345537288/libapt-pkg5.0_1.6~alpha5_amd64.deb

For each ... I manually "dpkg -i" and then ran "apt-cdrom add".

Results:
libapt-pkg5.0_1.6~alpha7ubuntu1 = fails <-- stock Bionic 20180216
libapt-pkg5.0_1.6~alpha7ubuntu = fails
libapt-pkg5.0_1.6~alpha6 = fails
libapt-pkg5.0_1.6~alpha5 = works <-- Bionic 20180131

Lubuntu Alternate 20180217 = fails

I did a git bisect on the source and it identified this upstream commit as being the potential culprit: https://salsa.debian.org/apt-team/apt/commit/1adcf56bec7d2127d83aa423916639740fe8e586

I can confirm that this is the culprit by testing the output of `apt-cdrom add` before and after reverting that specific commit from the tip of master.

On further analysis, the following diff is the problematic one:
- for (std::vector<APT::Configuration::Compressor>::const_iterator c = compressor.begin();
- c != compressor.end(); ++c)
+ for (auto const &ext: APT::Configuration::getCompressorExtensions())
      {
- if (stat((file + c->Extension).c_str(), &Buf) != 0)
+ if (stat((file + ext).c_str(), &Buf) != 0)

Reverting that diff and that diff only fixes `apt-cdrom add`.

Julian, could you take a look at why this is problematic? I'm afraid I don't have the expertise to proceed.

(to be more specific, here's a link to the exact line: https://salsa.debian.org/apt-team/apt/commit/1adcf56bec7d2127d83aa423916639740fe8e586#41c542e857179bd3fe055e92c64389379fb23e75_65_65 )

getCompressorExtensions() ignores the empty extension, thus that change broke support for uncompressed index ifles on cdrom.

Simon / Julian, thanks! Sounds like the problem is sorted out?

Any idea when a fix will hit bionic daily?

Fixed in 1.6~alpha7ubuntu2, just uploaded.

This bug was fixed in the package apt - 1.6~alpha7ubuntu2

---------------
apt (1.6~alpha7ubuntu2) bionic; urgency=medium

  [ David Kalnischkies ]
  * allow the apt/lists/auxfiles/ directory to be missing (Closes: 887624)

  [ Julian Andres Klode ]
  * indexcopy: Copy uncompressed indices from cdrom again (LP: #1746807)

 -- Julian Andres Klode <email address hidden> Mon, 19 Feb 2018 14:43:47 +0100

I just manually triggered a rebuild for Lubuntu Alternate since this landed, and after that, things should be good.

Thanks everyone!

I tried 20180219.1 but that must before the rebuild as it still had 1.6~alpha7ubuntu1.

I'll re-test again when the version changes.

Glad this fix is in the pipeline. :)

Thank you.

Lubuntu Alternate 20180220 = works

Looks good! :)

\o/