cloud-init logs leak hashed passwords

Thanks Mike. We'll get this fixed ASAP.

Thanks Mike,

James, will these errors halt the install process entirely? Or will it continue on without expected users?

Will fixing this (adding aliases?) cause unexpected *success* in cloud-config use in deployments? If we add aliases, what happens if someone supplies both lock_passwd and lock-passwd keys in their user data?

I'm a bit surprised this is the first we're hearing a consequence of a change from two years ago.

Thanks

A little more background might be helpful here. Nothing has changed in the code handling of users or groups. The thing that changed is that we have added schema validation to our cloud-init cloud-config. In the users/groups module being used here, cloud-init has supported keys with dashes '-' and keys with underscores '_', though the documentation for the past 4 years or so have only shown underscores. When we introduced the schema to this module, we effectively deprecated the dashes, and in the future we'll only be supporting underscores. If a module fails schema validation, we currently warn but do not raise an exception.

The cloud-config from this bug contains 'lock-passwd' which is still supported but now deprecated. We logged the bit of the cloud-config that doesn't pass validation, but in doing so we have exposed sensitive user data.

To specifically answer you questions:
will these errors halt the install process entirely?
No.

Or will it continue on without expected users?
Neither. It will do the right thing. It is complaining about the supplied keys not matching the schema.

Will fixing this (adding aliases?) cause unexpected *success* in cloud-config use in deployments?
We won't be changing aliases for the fix. The fix is to write a more generic warning to the log telling the user to run a command as root to see why their config is invalid.

If we add aliases, what happens if someone supplies both lock_passwd and lock-passwd keys in their user data?
This is currently already valid. We're trying to deprecate 'lock-passwd'

"I'm a bit surprised this is the first we're hearing a consequence of a change from two years ago."
The change is in 22.2 and was SRUed last week.

Note that the warning message from the schema check is triggered by both the "lock-passwd" key and the representation of users_groups.user groups key as an array of strings rather than a single string with comma separated group names. I need to change both of those in the json file to make it accept the way that Ubuntu subiquity installer wrote the cloud config.

diff --git a/cloudinit/config/schemas/schema-cloud-config-v1.json b/cloudinit/config/schemas/schema-cloud-config-v1.json
index a5be310a..4e07c0c9 100644
--- a/cloudinit/config/schemas/schema-cloud-config-v1.json
+++ b/cloudinit/config/schemas/schema-cloud-config-v1.json
@@ -35,7 +35,8 @@
         },
         "groups": {
           "description": "Optional comma-separated string of groups to add the user to.",
- "type": "string"
+ "type": ["array", "string"],
+ "items": {"type": "string"}
         },
         "homedir": {
           "description": "Optional home dir for user. Default: ``/home/<username>``",
@@ -51,6 +52,11 @@
           "description": "Disable password login. Default: ``true``",
           "type": "boolean"
         },
+ "lock-passwd": {
+ "default": true,
+ "description": "Disable password login. Default: ``true``",
+ "type": "boolean"
+ },
         "no_create_home": {
           "default": false,
           "description": "Do not create home directory. Default: ``false``",

Thanks for the explanations James,

Please use CVE-2022-2084 for this issue.

Thanks

Thanks Mike for the context here and spending time understanding the symptoms.

In our deprecation of this key, we should have also added a DEPRECATED description to this in our schema. Looks like we covered that approach in other keys like `grub-dpkg` -> grub_dpkg[1] and `remove-defaults` -> remove_defaults[2] but not in this case. We have PRs in flight to fix this and will make sure subiquity is also onboard with our deprecation schedule for these keys too so we don't re-introduce this problem when the hyphenated keys are officially dropped in upstream.

References:
[1] https://github.com/canonical/cloud-init/blob/main/cloudinit/config/schemas/schema-cloud-config-v1.json#L760
[2] https://github.com/canonical/cloud-init/blob/main/cloudinit/config/schemas/schema-cloud-config-v1.json#L167

Hey Seth, what do you think about the severity for this bug?

Chad, James, I've added you both to https://launchpad.net/~ubuntu-security/+archive/ubuntu/ubuntu-security-collab -- I believe 'dput' should work to this ppa.

Thanks

Thanks Seth.
Email sent out on this Friday with the suggested fix, the packages have been uploaded to this private PPA and are available for security review and upload into the appropriate -security pockets for Bionic, Focal, Impish, and Jammy.

Proposed fix, generalize schema validation warnings messages to avoid reporting potentially sensitive user-data values in /var/log/cloud-init.log.

cloud-init.postinst fixes to redact historic sensitive logs.

diff --git a/debian/changelog b/debian/changelog
index 135671ae..baec8fd1 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+cloud-init (22.2-0ubuntu1~22.04.3) jammy; urgency=medium
+
+ * d/cloud-init.postinst: redact previously leaked schema errors from logs
+ * Remove schema errors from log (LP: #1978422) (CVE-2022-2084)
+
+ -- James Falcon <email address hidden> Tue, 14 Jun 2022 06:31:00 -0500
+
 cloud-init (22.2-0ubuntu1~22.04.2) jammy; urgency=medium

   * cherry-pick a2e62738: Fix cc_phone_home requiring 'tries' (#1500)
diff --git a/debian/cloud-init.postinst b/debian/cloud-init.postinst
index 683ba86d..85788a98 100644
--- a/debian/cloud-init.postinst
+++ b/debian/cloud-init.postinst
@@ -125,6 +125,27 @@ handle_preseed_local_cloud_config() {
    db_unregister "${debconf_name}" || :
 }

+fix_1978422_redact_sensitive_logs_on_invalid_userdata_schema() {
+ local oldver="$1" last_bad_ver="22.2-0ubuntu1~22.04.2"
+ dpkg --compare-versions "$oldver" le "$last_bad_ver" || return 0
+
+ MSG="Redacting sensitive logs due to invalid cloud-config user-data from"
+ INVALID_USERDATA_LOG="Invalid cloud-config provided:"
+ if grep -q "${INVALID_USERDATA_LOG}" /var/log/cloud-init.log; then
+ echo "${MSG} /var/log/cloud-init.log"
+ # Redact all schema warnings between
+ # 'Invalid cloud-config provided:' and the next timestamped log 2022-
+ sed -i '/Invalid cloud-config provided:/,/2022-/{/^[^2202-]/d};s/Invalid cloud-config provided:.*/Invalid cloud-config provided. To see errors, run: sudo cloud-init schema --system/' /var/log/cloud-init.log
+ fi
+ if grep -q "${INVALID_USERDATA_LOG}" /var/log/cloud-init-output.log; then
+ echo "${MSG} /var/log/cloud-init-output.log"
+ # Redact all schema warnings between
+ # 'Invalid cloud-config provided:' and the public/private key gen at
+ # 'Generating public/private rsa key pair' OR 'Cloud'
+ sed -i '/Invalid cloud-config provided:/,/Generating\|Cloud/{/Cloud/b; /^[^Generating]/d};s/Invalid cloud-config provided:.*/Invalid cloud-config provided. To see errors, run: sudo cloud-init schema --system\nGenerating public\/private rsa key pair./' /var/log/cloud-init-output.log
+ fi
+}
+
 fix_1336855() {
   ### Begin fix for LP: 1336855
   # fix issue where cloud-init misidentifies the location of grub and
@@ -375,6 +396,9 @@ EOF
    cleanup_ureadahead "$2"
    fix_lp1889555 "$2"
    change_cloud_init_output_log_permissions "$2"
+
+ # Redact schema sensitive warning logs on invalid user-data
+ fix_1978422_redact_sensitive_logs_on_invalid_userdata_schema "$2"
 fi

 #DEBHELPER#
diff --git a/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log b/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log
new file mode 100644
index 00000000..01e886d7
--- /dev/null
+++ b/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log
@@ -0,0 +1,148 @@
+From b0534cbf05221b141ebd2edb5a71e94742b...

functional debdiff for bionic between 22.2-0ubuntu1~18.04.2 (in bionic-updates pocket) and 22.2-0ubuntu1~18.04.3

functional debdiff for focal between 22.2-0ubuntu1~20.04.2 (in focal-updates pocket) and 22.2-0ubuntu1~20.04.3

functional debdiff for impish between 22.2-0ubuntu1~21.10.2 (in impish-updates pocket) and 22.2-0ubuntu1~21.10.3

functional debdiff for jammy between 22.2-0ubuntu1~22.04.2 (in jammy-updates pocket) and 22.2-0ubuntu1~22.04.3

Updated patches to suggest in messaging that admins may want to redact journalctl logs or remote log aggregators.

See https://github.com/canonical/cloud-init/pull/1551

Is Cloud-Init 21.4 affected?

The change landed in 22.2, so 21.4 should not be affected.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3987