Regression: images ship with modified configuration file

Could you provide which of the cloud-images? The build info doesn't let us know which clouds to triage, as each cloud has different configurations

Regression is also a bit of a misnomer -- based on the description, and bugs, I'd say this has been a problem for at least 7 years (based on the ticket ages, and when the bug was introduced). There hasn't been a change to that file since commit 928a79a

"Add hooks and includes."

So this has probably popped up in every release in some fashion, _unless_ ufc or another package changed that re-introduced a bug. ufc was called out in a few places as being a possible cause.

Looking at the specific config, for Focal forward, CPC should move the configuration from /etc/sshd/sshd_config to /etc/sshd/sshd_config.d/ files. that should prevent issues related

CPC will go through as many products as we can to ensure the upgrade path is viable and doesn't have any issues.

did the following test with ec2

launch ec2 Focal instance via ec2 launch wizard
created an instance that was _very_ old (will be opening a ticket on that)

cat /etc/cloud/build.info
build_name: server
serial: 20211129

sudo apt-get update && sudo apt-get dist-upgrade -y

and received no issues.

Sorry, I didn't realise the build info didn't provide what you needed.

It's reproducible with "lxc launch ubuntu:focal" and then "do-release-upgrade -d" (after a regular apt upgrade to ensure it's ready, etc).

A quicker way of verifying is to locate ucf's copy of what it left in /etc. This is in /var/lib/ucf/cache/. It should be identical to what is actually in /etc after booting a cloud image. This is presumably a hack, but should be a quick way to check for now.

> Regression is also a bit of a misnomer...

From a user's point of view, they were able to release upgrade without this issue before, and now they can't, so it's a regression even if it wasn't caused by a recent code change in livecd-rootfs. The bug in livecd-rootfs might have been latent and present for seven years, but it's now been brought to the surface by an otherwise-reasonable change in openssh-server.

is `do-release-upgrade` meant to be automatable? As an end-user that is a terrifying prospect. It seems improbable for us to know all possible end-user configurations and support an automated release upgrade.

i'm seeing a far more concerning issue related to adding `needsrestart`. Any service that gets updated after `needsrestart` is configured appears to add a splash. this affects not just do-release-upgrade , but apt upgrade and apt dist-upgrade

racb and I had a quick chat and here's our take away:

1. sshd configs need to move to /etc/sshd/sshd_config.d/ , as is appropriate

2. we on CPC are going to go through our images and ensure we have config files in appropriate places. If there are .d/ directories available, we need to use them. If there are not, we should open bugs against the packages we are configuring to get .d/ folders added as we have requirements to provided initial configuration, and support automated updates

3. CPC is going to check our package update (apt-get update && apt-get dist-upgrade -y) on default images on Focal and Jammy to see what pops up now, and document behaviour to end users. This is not a release blocked

4. CPC is going to check `do-release-upgrade` from Focal to Jammy and document default behaviour

note that our default behaviour documentation won't be able to cover all end-user experiences. it just gives a baseline for expectations for "base" starting points.

our goal is to make sure end-users know the best non-interactive paths on the cloud using the Ubuntu default experiences (meaning we won't add instructions for all IT automation possibilities, but will document apt, needrestart, etc)

Thank you for summarizing John.

To be clear, I don't think a non-interactive upgrade between releases is a particularly important user story in itself. I'm more concerned that the current issue in the upgrade from Focal to Jammy reveals that a latent issue exists, and this kind of thing can also impact automated upgrades _within_ a single release, and it's the latter which is a common thing for deployment automation to do. The fix is, as you say, unconnected with Jammy's release schedule. It also doesn't help already-deployed instances.

This bug was fixed in the package livecd-rootfs - 2.767

---------------
livecd-rootfs (2.767) kinetic; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc project
    builds. (LP: #1968873)

 -- Thomas Bechtold <email address hidden> Tue, 28 Jun 2022 13:29:05 +0200

Hello Robie, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Jammy SRU verification:

1) test build with:
./scripts/ubuntu-bartender/ubuntu-bartender --hook-extras-dir /home/tom/devel/canonical/cloudware/cpc_packaging.extra/ --livecd-rootfs-dir /home/tom/devel/livecd-rootfs/ -- --series jammy --project ubuntu-cpc --image-target ec2

2) checked that sshd_config is not modified inside of the image
3) checked /etc/ssh/sshd_config.d/10-cloudimg-settings.conf and looks good

So all good for Jammy

Please don't SRU! It breaks the way cloud-init sets password auth:

At least on Azure: if someone sets --authentication-type password, cloud-init does the following:

2022-07-07 15:40:54,037 - util.py[DEBUG]: Read 3254 bytes from /etc/ssh/sshd_config
2022-07-07 15:40:54,037 - ssh_util.py[DEBUG]: line 123: option PasswordAuthentication added with yes

And this option is overwritten by this new file "/etc/ssh/sshd_config.d/10-cloudimg-settings.conf", so users can't SSH into their VM!

I think we need this bug to get fixed in cloud-init before adding this to cloud-images.

Gauthier, thank you for pointing this out.

I think there are a couple of things that need fixing in cloud-init then:

1) The assumption that appending "PasswordAuthentication yes" to /etc/ssh/sshd_config will override all of sshd's configuration - false because /etc/ssh/sshd_config.d/ now exists (IIRC since Focal).

2) The editing of /etc/sshd_config instead of writing a new file to /etc/sshd_config.d/. This might only work for newer releases though; maybe for systems with older sshd the older behaviour is still the best option.

Unless we can find another way, it doesn't seem like this bug can be fixed until cloud-init is fixed, so I added it as a task for the same bug.

Hello Robie, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

This fix is reverted as of the accepted 2.765.5 (but still appears in .changes due to existing changelog entries).

Hello Robie, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted livecd-rootfs (2.765.6) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.765.6 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

This bug was fixed in the package livecd-rootfs - 2.765.6

---------------
livecd-rootfs (2.765.6) jammy; urgency=medium

  [ Alexandre Ghiti ]
  * Fix SiFive Unmatched build. While merging the VisionFive support, we
    removed the installation of u-boot-menu for the Unmatched by mistake
    (LP: #1982735).

livecd-rootfs (2.765.5) jammy; urgency=medium

  * Add support for the VisionFive and the Nezha boards (LP: #1980935)
  * Set FK_FORCE_CONTAINER for RISC-V images build to force flash-kernel
    to run in a container (LP: #1980929).

livecd-rootfs (2.765.4) jammy; urgency=medium

  * Revert "don't modify ssh_config for ubuntu-cpc projects (LP:1968873)"
    This change triggered a bug in cloud-init (see LP:1968873).

livecd-rootfs (2.765.3) jammy; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc
    project builds (LP: #1968873)

livecd-rootfs (2.765.2) jammy; urgency=medium

  * Switch the intel-iot images to use the linux-intel-iotg kernel instead.
    (LP: #1980065)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Mon, 25 Jul 2022 10:24:08 +0200

This was reverted so remains an issue.

Note, Alberto has started iterating on an upstream fix for this in cloud-init https://github.com/canonical/cloud-init/pull/1618

Upstream fix for cloud-init landed: https://github.com/canonical/cloud-init/commit/f4d5f73cfa82a72e3cd967ba4c8c8e7012d3f691

Expect this to be SRUd and available minimally in cloud-init version 22.3 scheduled for the week of August 16th https://discourse.ubuntu.com/t/cloud-init-2022-release-schedule/25413

This bug was fixed in the package cloud-init - 22.2-139-gb64ba456-0ubuntu1~22.10.1

---------------
cloud-init (22.2-139-gb64ba456-0ubuntu1~22.10.1) kinetic; urgency=medium

  * New upstream snapshot.
    + test: use fake filesystem to avoid file removal (#1647)
    + tox: Fix tip-flake8 and tip-mypy (#1635)
    + config: Add wireguard config module (#1570)
      [Fabian Lichtenegger-Lukas]
    + tests: can run without azure-cli, tests expect inactive ansible
      (#1643)
    + typing: Type UrlResponse.contents (#1633)
    + testing: fix references to `DEPRECATED.` (#1641)
    + ssh_util: Handle sshd_config.d folder (LP: #1968873)
    + schema: Enable deprecations in cc_update_etc_hosts (#1631)
    + Add Ansible Config Module (#1579)
    + util: Support Idle process state in get_proc_ppid() (#1637)
    + schema: Enable deprecations in cc_growpart (#1628)
    + schema: Enable deprecations in cc_users_groups (#1627)
    + util: Fix error path and parsing in get_proc_ppid()
    + main: avoid downloading full contents cmdline urls (#1606)
      (LP: #1937319)
    + schema: Enable deprecations in cc_scripts_vendor (#1629)
    + schema: Enable deprecations in cc_set_passwords (#1630)
    + sources/azure: add experimental support for preprovisioned os disks
      (#1622) [Chris Patterson]
    + Remove configobj a_to_u calls (#1632) [Stefano Rivera]
    + cc_debug: Drop this module (#1614)
    + schema: add aggregate descriptions in anyOf/oneOf (#1636)
    + testing: migrate test_sshutil to pytest (#1617)
    + testing: Fix test_ca_certs integration test (#1626)
    + testing: add support for pycloudlib's pro images (#1604)
    + testing: migrate test_cc_set_passwords to pytest (#1615)

 -- Alberto Contreras <email address hidden> Wed, 10 Aug 2022 22:23:47 +0200

A new cloud-init release (22.3) is being SRUed in https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1987318 for Focal and Jammy.

The bug status for livecd-rootfs devel needs to be set back to "in progress" because we reverted the fix for devel (see https://git.launchpad.net/livecd-rootfs/commit/?id=aa1be5eaaa1fdbb4c31104cc49e54b15f66c3343) due to a regression with cloud-init.

This bug was fixed in the package livecd-rootfs - 2.777

---------------
livecd-rootfs (2.777) kinetic; urgency=medium

  * Rename FK_FORCE_CONTAINER into FK_FORCE as its role was extended to also
    support chroot (LP: #1989194).

 -- Alexandre Ghiti <email address hidden> Thu, 08 Sep 2022 16:25:36 +0200

For the record it was actually version 2.776 which fixed this:

livecd-rootfs (2.776) kinetic; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc project
    builds. (LP: #1968873). The cloud-init bug (LP #1968873) got
    fixed so using a config snippet now.

The move to a sshd_config.d/ snippet seems to be incorrect. That snippet doesn't take the debconf value for openssh-server/password-authentication into account.
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2037718 is a follow up bug that will handle this case, too.