Regression: images ship with modified configuration file

Bug #1968873 reported by Robie Basak
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
livecd-rootfs
In Progress
High
Thomas Bechtold
cloud-init (Ubuntu)
Fix Released
High
Alberto Contreras
Focal
Fix Released
High
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Fix Released
High
Unassigned
livecd-rootfs (Ubuntu)
Fix Released
High
Thomas Bechtold
Focal
In Progress
High
Thomas Bechtold
Impish
Won't Fix
Undecided
Unassigned
Jammy
Fix Committed
High
Thomas Bechtold

Bug Description

This cloud image:

build_name: server
serial: 20220411.2

...ships with a "user-modified" /etc/ssh/sshd_config. This results in a confusing ucf prompt when upgrading from Focal to Jammy.

It looks like this is being done here: https://git.launchpad.net/ubuntu/+source/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/chroot/052-ssh_authentication.chroot

User impact: this messes up automation; it is a common use of cloud images for users to automate their server deployments which generally involves running a dist-upgrade and then bootstrapping some kind of configuration management system.

As well as during an upgrade between releases, this will happen if openssh-server needs to change its /etc/ssh/sshd_config in an SRU or security update. Even users who have not touched /etc/ssh/sshd_config will be told that they have and the default is to keep the user "modified" version, so the change will not be applied.

Workaround: users can use something like: apt-get update && UCF_FORCE_CONFOLD=1 apt-get -y dist-upgrade. Note that to avoid prompts in various other cases users should also set DEBIAN_FRONTEND=noninteractive, run apt-get with -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" and consider redirecting stdin from /dev/null.

Background:

This has happened multiple times before. Please put CI in place to avoid this regressing again. See bug 1485685, bug 1581044, bug 1581046, bug 1323772, bug 1747464.

This needs to be backported to >= Focal so upgrades from Focal to Jammy do not result in the ucf prompt.

[Test Plan]
1) build image (eg. with ubuntu-old-fashioned) with the changes from here
2) check that there is no modification for /etc/ssh/sshd_config via:
  $ ucfq openssh-server
3) check that the new configuration file under /etc/ssh/sshd_config.d/10-cloudimg-settings.conf is there and contains the correct setting

[Where problems could occur]
Password authentication could be enabled after that change because sshd does not read the new configuration for whatever reason. This can be checked with "sudo sshd -T|grep passwordauthentication"

Related branches

Robie Basak (racb)
summary: - Regression: images ship with modified conffiles
+ Regression: images ship with modified configuration file
Revision history for this message
John Chittum (jchittum) wrote :

Could you provide which of the cloud-images? The build info doesn't let us know which clouds to triage, as each cloud has different configurations

Regression is also a bit of a misnomer -- based on the description, and bugs, I'd say this has been a problem for at least 7 years (based on the ticket ages, and when the bug was introduced). There hasn't been a change to that file since commit 928a79a

"Add hooks and includes."

So this has probably popped up in every release in some fashion, _unless_ ufc or another package changed that re-introduced a bug. ufc was called out in a few places as being a possible cause.

Looking at the specific config, for Focal forward, CPC should move the configuration from /etc/sshd/sshd_config to /etc/sshd/sshd_config.d/ files. that should prevent issues related

CPC will go through as many products as we can to ensure the upgrade path is viable and doesn't have any issues.

Revision history for this message
John Chittum (jchittum) wrote (last edit ):

did the following test with ec2

launch ec2 Focal instance via ec2 launch wizard
created an instance that was _very_ old (will be opening a ticket on that)

cat /etc/cloud/build.info
build_name: server
serial: 20211129

sudo apt-get update && sudo apt-get dist-upgrade -y

and received no issues.

Revision history for this message
Robie Basak (racb) wrote :

Sorry, I didn't realise the build info didn't provide what you needed.

It's reproducible with "lxc launch ubuntu:focal" and then "do-release-upgrade -d" (after a regular apt upgrade to ensure it's ready, etc).

A quicker way of verifying is to locate ucf's copy of what it left in /etc. This is in /var/lib/ucf/cache/. It should be identical to what is actually in /etc after booting a cloud image. This is presumably a hack, but should be a quick way to check for now.

> Regression is also a bit of a misnomer...

From a user's point of view, they were able to release upgrade without this issue before, and now they can't, so it's a regression even if it wasn't caused by a recent code change in livecd-rootfs. The bug in livecd-rootfs might have been latent and present for seven years, but it's now been brought to the surface by an otherwise-reasonable change in openssh-server.

Revision history for this message
John Chittum (jchittum) wrote :

is `do-release-upgrade` meant to be automatable? As an end-user that is a terrifying prospect. It seems improbable for us to know all possible end-user configurations and support an automated release upgrade.

i'm seeing a far more concerning issue related to adding `needsrestart`. Any service that gets updated after `needsrestart` is configured appears to add a splash. this affects not just do-release-upgrade , but apt upgrade and apt dist-upgrade

Revision history for this message
John Chittum (jchittum) wrote :

racb and I had a quick chat and here's our take away:

1. sshd configs need to move to /etc/sshd/sshd_config.d/ , as is appropriate

2. we on CPC are going to go through our images and ensure we have config files in appropriate places. If there are .d/ directories available, we need to use them. If there are not, we should open bugs against the packages we are configuring to get .d/ folders added as we have requirements to provided initial configuration, and support automated updates

3. CPC is going to check our package update (apt-get update && apt-get dist-upgrade -y) on default images on Focal and Jammy to see what pops up now, and document behaviour to end users. This is not a release blocked

4. CPC is going to check `do-release-upgrade` from Focal to Jammy and document default behaviour

note that our default behaviour documentation won't be able to cover all end-user experiences. it just gives a baseline for expectations for "base" starting points.

our goal is to make sure end-users know the best non-interactive paths on the cloud using the Ubuntu default experiences (meaning we won't add instructions for all IT automation possibilities, but will document apt, needrestart, etc)

Revision history for this message
Robie Basak (racb) wrote :

Thank you for summarizing John.

To be clear, I don't think a non-interactive upgrade between releases is a particularly important user story in itself. I'm more concerned that the current issue in the upgrade from Focal to Jammy reveals that a latent issue exists, and this kind of thing can also impact automated upgrades _within_ a single release, and it's the latter which is a common thing for deployment automation to do. The fix is, as you say, unconnected with Jammy's release schedule. It also doesn't help already-deployed instances.

John Chittum (jchittum)
Changed in cloud-images:
status: New → Triaged
importance: Undecided → High
Changed in cloud-images:
assignee: nobody → Thomas Bechtold (toabctl)
status: Triaged → In Progress
no longer affects: cloud-images/focal
affects: cloud-images → livecd-rootfs
Utkarsh Gupta (utkarsh)
no longer affects: livecd-rootfs/trunk
Changed in livecd-rootfs (Ubuntu):
importance: Undecided → High
status: New → Fix Committed
assignee: nobody → Thomas Bechtold (toabctl)
Utkarsh Gupta (utkarsh)
no longer affects: livecd-rootfs (Ubuntu)
no longer affects: livecd-rootfs (Ubuntu Focal)
no longer affects: livecd-rootfs (Ubuntu Impish)
no longer affects: livecd-rootfs (Ubuntu Jammy)
Changed in livecd-rootfs (Ubuntu):
assignee: nobody → Thomas Bechtold (toabctl)
status: New → In Progress
importance: Undecided → High
Changed in livecd-rootfs (Ubuntu):
status: In Progress → Fix Committed
description: updated
Changed in livecd-rootfs (Ubuntu Jammy):
status: New → In Progress
assignee: nobody → Thomas Bechtold (toabctl)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.767

---------------
livecd-rootfs (2.767) kinetic; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc project
    builds. (LP: #1968873)

 -- Thomas Bechtold <email address hidden> Tue, 28 Jun 2022 13:29:05 +0200

Changed in livecd-rootfs (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Robie, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Thomas Bechtold (toabctl) wrote :

Jammy SRU verification:

1) test build with:
./scripts/ubuntu-bartender/ubuntu-bartender --hook-extras-dir /home/tom/devel/canonical/cloudware/cpc_packaging.extra/ --livecd-rootfs-dir /home/tom/devel/livecd-rootfs/ -- --series jammy --project ubuntu-cpc --image-target ec2

2) checked that sshd_config is not modified inside of the image
3) checked /etc/ssh/sshd_config.d/10-cloudimg-settings.conf and looks good

So all good for Jammy

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Gauthier Jolly (gjolly) wrote (last edit ):

Please don't SRU! It breaks the way cloud-init sets password auth:

At least on Azure: if someone sets --authentication-type password, cloud-init does the following:

2022-07-07 15:40:54,037 - util.py[DEBUG]: Read 3254 bytes from /etc/ssh/sshd_config
2022-07-07 15:40:54,037 - ssh_util.py[DEBUG]: line 123: option PasswordAuthentication added with yes

And this option is overwritten by this new file "/etc/ssh/sshd_config.d/10-cloudimg-settings.conf", so users can't SSH into their VM!

I think we need this bug to get fixed in cloud-init before adding this to cloud-images.

tags: added: verification-needed-jammy
removed: verification-done-jammy
Revision history for this message
Robie Basak (racb) wrote :

Gauthier, thank you for pointing this out.

I think there are a couple of things that need fixing in cloud-init then:

1) The assumption that appending "PasswordAuthentication yes" to /etc/ssh/sshd_config will override all of sshd's configuration - false because /etc/ssh/sshd_config.d/ now exists (IIRC since Focal).

2) The editing of /etc/sshd_config instead of writing a new file to /etc/sshd_config.d/. This might only work for newer releases though; maybe for systems with older sshd the older behaviour is still the best option.

Unless we can find another way, it doesn't seem like this bug can be fixed until cloud-init is fixed, so I added it as a task for the same bug.

tags: added: verification-failed verification-failed-jammy
removed: verification-needed verification-needed-jammy
tags: added: regression-proposed
James Falcon (falcojr)
Changed in cloud-init (Ubuntu):
importance: Undecided → High
Chad Smith (chad.smith)
Changed in cloud-init (Ubuntu):
assignee: nobody → Alberto Contreras (aciba)
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Robie, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-jammy
removed: verification-failed verification-failed-jammy
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

This fix is reverted as of the accepted 2.765.5 (but still appears in .changes due to existing changelog entries).

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Robie, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (livecd-rootfs/2.765.6)

All autopkgtests for the newly accepted livecd-rootfs (2.765.6) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.765.6 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.765.6

---------------
livecd-rootfs (2.765.6) jammy; urgency=medium

  [ Alexandre Ghiti ]
  * Fix SiFive Unmatched build. While merging the VisionFive support, we
    removed the installation of u-boot-menu for the Unmatched by mistake
    (LP: #1982735).

livecd-rootfs (2.765.5) jammy; urgency=medium

  * Add support for the VisionFive and the Nezha boards (LP: #1980935)
  * Set FK_FORCE_CONTAINER for RISC-V images build to force flash-kernel
    to run in a container (LP: #1980929).

livecd-rootfs (2.765.4) jammy; urgency=medium

  * Revert "don't modify ssh_config for ubuntu-cpc projects (LP:1968873)"
    This change triggered a bug in cloud-init (see LP:1968873).

livecd-rootfs (2.765.3) jammy; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc
    project builds (LP: #1968873)

livecd-rootfs (2.765.2) jammy; urgency=medium

  * Switch the intel-iot images to use the linux-intel-iotg kernel instead.
    (LP: #1980065)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Mon, 25 Jul 2022 10:24:08 +0200

Changed in livecd-rootfs (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote :

This was reverted so remains an issue.

Changed in livecd-rootfs (Ubuntu Jammy):
status: Fix Released → Triaged
Revision history for this message
Chad Smith (chad.smith) wrote :

Note, Alberto has started iterating on an upstream fix for this in cloud-init https://github.com/canonical/cloud-init/pull/1618

Brett Holman (holmanb)
Changed in cloud-init (Ubuntu):
status: New → In Progress
Revision history for this message
Chad Smith (chad.smith) wrote :

Upstream fix for cloud-init landed: https://github.com/canonical/cloud-init/commit/f4d5f73cfa82a72e3cd967ba4c8c8e7012d3f691

Expect this to be SRUd and available minimally in cloud-init version 22.3 scheduled for the week of August 16th https://discourse.ubuntu.com/t/cloud-init-2022-release-schedule/25413

Changed in cloud-init (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 22.2-139-gb64ba456-0ubuntu1~22.10.1

---------------
cloud-init (22.2-139-gb64ba456-0ubuntu1~22.10.1) kinetic; urgency=medium

  * New upstream snapshot.
    + test: use fake filesystem to avoid file removal (#1647)
    + tox: Fix tip-flake8 and tip-mypy (#1635)
    + config: Add wireguard config module (#1570)
      [Fabian Lichtenegger-Lukas]
    + tests: can run without azure-cli, tests expect inactive ansible
      (#1643)
    + typing: Type UrlResponse.contents (#1633)
    + testing: fix references to `DEPRECATED.` (#1641)
    + ssh_util: Handle sshd_config.d folder (LP: #1968873)
    + schema: Enable deprecations in cc_update_etc_hosts (#1631)
    + Add Ansible Config Module (#1579)
    + util: Support Idle process state in get_proc_ppid() (#1637)
    + schema: Enable deprecations in cc_growpart (#1628)
    + schema: Enable deprecations in cc_users_groups (#1627)
    + util: Fix error path and parsing in get_proc_ppid()
    + main: avoid downloading full contents cmdline urls (#1606)
      (LP: #1937319)
    + schema: Enable deprecations in cc_scripts_vendor (#1629)
    + schema: Enable deprecations in cc_set_passwords (#1630)
    + sources/azure: add experimental support for preprovisioned os disks
      (#1622) [Chris Patterson]
    + Remove configobj a_to_u calls (#1632) [Stefano Rivera]
    + cc_debug: Drop this module (#1614)
    + schema: add aggregate descriptions in anyOf/oneOf (#1636)
    + testing: migrate test_sshutil to pytest (#1617)
    + testing: Fix test_ca_certs integration test (#1626)
    + testing: add support for pycloudlib's pro images (#1604)
    + testing: migrate test_cc_set_passwords to pytest (#1615)

 -- Alberto Contreras <email address hidden> Wed, 10 Aug 2022 22:23:47 +0200

Changed in cloud-init (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Alberto Contreras (aciba) wrote :

A new cloud-init release (22.3) is being SRUed in https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1987318 for Focal and Jammy.

Revision history for this message
Thomas Bechtold (toabctl) wrote :

The bug status for livecd-rootfs devel needs to be set back to "in progress" because we reverted the fix for devel (see https://git.launchpad.net/livecd-rootfs/commit/?id=aa1be5eaaa1fdbb4c31104cc49e54b15f66c3343) due to a regression with cloud-init.

Utkarsh Gupta (utkarsh)
Changed in livecd-rootfs (Ubuntu):
status: Fix Released → In Progress
Changed in livecd-rootfs (Ubuntu Focal):
importance: Undecided → High
Changed in cloud-init (Ubuntu Impish):
status: New → Won't Fix
Changed in livecd-rootfs (Ubuntu Impish):
status: New → Won't Fix
Changed in livecd-rootfs (Ubuntu Jammy):
importance: Undecided → High
Changed in cloud-init (Ubuntu Focal):
importance: Undecided → High
Changed in cloud-init (Ubuntu Jammy):
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.777

---------------
livecd-rootfs (2.777) kinetic; urgency=medium

  * Rename FK_FORCE_CONTAINER into FK_FORCE as its role was extended to also
    support chroot (LP: #1989194).

 -- Alexandre Ghiti <email address hidden> Thu, 08 Sep 2022 16:25:36 +0200

Changed in livecd-rootfs (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote :

For the record it was actually version 2.776 which fixed this:

livecd-rootfs (2.776) kinetic; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc project
    builds. (LP: #1968873). The cloud-init bug (LP #1968873) got
    fixed so using a config snippet now.

Changed in livecd-rootfs (Ubuntu Jammy):
status: Triaged → Fix Committed
Changed in livecd-rootfs (Ubuntu Focal):
assignee: nobody → Thomas Bechtold (toabctl)
status: New → In Progress
Changed in cloud-init (Ubuntu Jammy):
status: New → Fix Released
Changed in cloud-init (Ubuntu Focal):
status: New → Fix Released
Revision history for this message
Thomas Bechtold (toabctl) wrote :

The move to a sshd_config.d/ snippet seems to be incorrect. That snippet doesn't take the debconf value for openssh-server/password-authentication into account.
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2037718 is a follow up bug that will handle this case, too.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.