fetch-url does not use --no-check-certificate on HTTP to HTTPS redirects

Testing performed with Disco, Cosmic, Bionic, Xenial, and Trusty.

The output is similar if not identical, so pasting just one test, from Disco.

(Web Server, HTTP/HTTPS redirect, setup not included, ping me if interested.)

$ wget http://archive.ubuntu.com/ubuntu/dists/disco/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/{linux,initrd.gz}

$ GUEST=disco

$ virt-install \
  --name $GUEST \
  --vcpus 2 \
  --memory 1024 \
  --disk $GUEST.qcow2,bus=virtio,format=qcow2,size=8 \
  --network bridge=virbr0,model=virtio \
  --graphics none \
  --import \
  --boot \
kernel=linux,\
initrd=initrd.gz,\
kernel_args='console=ttyS0 url=http://192.168.122.1/preseed debian-installer/allow_unauthenticated_ssl=true'

The installer hits an error when trying to get the preseed file:

     ┌──────────┤ [!!] Download debconf preconfiguration file ├──────────┐
     │ │
     │ Failed to retrieve the preconfiguration file │
     │ The file needed for preconfiguration could not be retrieved from │
     │ http://192.168.122.1/preseed. The installation will proceed in │
     │ non-automated mode. │
     │ │
     │ <Continue> │
     │ │
     └───────────────────────────────────────────────────────────────────┘

The synthetic tests with fetch-url:
===

~ # cat /proc/cmdline
console=ttyS0 url=http://192.168.122.1/preseed debian-installer/allow_unauthenticated_ssl=true

~ # cat /etc/default-release
disco

Without patch:
---

~ # fetch-url http://192.168.122.1/preseed preseed
ERROR: cannot verify 192.168.122.1's certificate, ...
...
To connect to 192.168.122.1 insecurely, use `--no-check-certificate'.

~ # echo $?
1

With patch:
---

~ # wget --no-check-certificate http://192.168.122.1/di-utils_1.124ubuntu2_amd64.udeb
~ # udpkg -i di-utils_1.124ubuntu2_amd64.udeb

~ # fetch-url http://192.168.122.1/preseed preseed
WARNING: cannot verify 192.168.122.1's certificate, ...
...
2018-11-14 13:17:03 URL:https://192.168.122.1//preseed [11/11] -> "./_fetch-url_preseed.1467" [1]

~ # echo $?
0

With patch and Without d-i/allow_unauthenticated_ssl=true: No Change
---

~ # cat /proc/cmdline
console=ttyS0 url=http://192.168.122.1/preseed

~ # wget --no-check-certificate http://192.168.122.1/di-utils_1.124ubuntu2_amd64.udeb
~ # udpkg -i di-utils_1.124ubuntu2_amd64.udeb

~ # fetch-url http://192.168.122.1/preseed preseed
ERROR: cannot verify 192.168.122.1's certificate, ...
...
To connect to 192.168.122.1 insecurely, use `--no-check-certificate'.

~ # echo $?
1

Oops, I'll add the Debina/Ubuntu-Bug: DEP-3 tags in the debdiffs.

Sponsor note:

* Substitute (LP: #NNNNNNN) by (LP: #1803385)
* Re-visit version for cosmic and bionic
  - In this case, if Disco is 1.124ubuntu2 | cosmic need to be 1.124ubuntu1.18.10.1 and bionic 1.124ubuntu1.18.04.1. Please double-check with dpkg --compare-version to make sure I didn't do typos.

- Eric

Er, I mean, the right LP bug numbers in changelog,
and fix the version suffixes for Bionic and Cosmic,
as reported by slashd in IRC (thanks!)

The attachment "disco_di-utils_httpsredir.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Eric,

Just updated the attached debdiffs w/ the fixes for the points listed.

Thanks for reviewing; nice catch in the Cosmic/Bionic version suffixes
(I have verified them with dpkg --compare-versions, all good this time).

cheers,
Mauricio

Closing this bug as Invalid.
The real solution is fix-released in LP#1807023.

This bug was a workaround for not having ca-certificates in d-i and use an HTTP mirror that redirected to HTTPS
(the resulting certificate validation error couldn't be ignored due to HTTP protocol not using the wget option.)

But this is no longer required with the ca-certificates shipped in debian-installer.

Sorry, I had lost track of this bug.
Mauricio

this was fixed a different way, in d-i and ca-certificates.

bug 1807023