cloud-init

Bug #1918303
Comment #4

Comment 4 for bug 1918303

Revision history for this message

Carl Pearson (wasp0x01) wrote on 2021-03-10: Re: [Bug 1918303] Re: Randomly set credentials written in cleartext to world-readable file

Hi Dan,
Thank you for the update and the detailed analysis. I'm not as familiar with some of the cloud-init use cases so good to know what some of the constraints are. Yep, restricting read access to cloud-init-output.log or redacting the messages when sent to the log both sound like workable strategies to me, although as you noted both have some caveats to work around.
Thanks and have a great day,
Carl
On Tuesday, March 9, 2021, 2:30:42 PM PST, Dan Watkins <email address hidden> wrote:

Hi Carl,

Thanks for this detailed and thoughtful bug report, we really appreciate
it. We've had some preliminary internal discussions, and we have a sync
with our security team scheduled for tomorrow afternoon to discuss
whether or not we think this issue warrants a coordinated release.
Below is my initial analysis.

In order to allow people to access systems using these randomly
generated passwords (without needing another access vector in order to
know the passwords), cloud-init emits them to the serial console. In
order to have log messages emitted to the console readily available
within the system also, it writes that same content to /var/log/cloud-
init-output.log. As a result, those passwords are written to that file,
which is world-readable.

In the first instance, we should consider whether or not emitting these
random passwords to the console is acceptable default behaviour. For
users who are using this functionality as the only way they access the
instance, it is: there's no other way for them to know what password to
use. The question then becomes, I think, what proportion of users of
this functionality use it in this way? If it's the majority, then
perhaps emitting them by default is justifiable. If not (or if we think
this behaviour is simply too dangerous to be the default), then we could
consider toggling it with a new configuration option, defaulting to
false: users are already passing in cloud-config to get into this
situation, so working around this change in behaviour should be
reasonably straightforward. (I don't think we can remove support for
the emission altogether: that will break existing workflows, and in a
way that is likely to move such folks to simply using hard-coded
passwords: not exactly a security fix.)

Given my (parenthetical) conclusion, we clearly do also need to address
how we are going to remove access to these passwords from unprivileged
users within the system. Two obvious options present themselves: (a) we
could make cloud-init-output.log only root-readable, or (b) we could
modify the code which emits these messages to either redact them when
going to cloud-init-output.log or omit them from there entirely. cloud-
init-output.log and the serial console are tied together very
tightly[0], so (b) could prove to be difficult: we'll perform some
further analysis. (It's possible that the default config in [0] could
be extended to omit the lines using grep, for example?)

(For (a), I also just noticed that the content in cloud-init-output.log
is also present in the journal for the various cloud-init units.
Whatever resolution we reach regarding cloud-init-output.log, we should
confirm that the permissions that journalctl enforces for such access
are acceptable to us.)

Thanks again for the report! Any comments, questions, or suggestions
(either from you, Carl, or the other cloud-init folks with access to
this report) are more than welcome!

Dan

[0] The default configuration should give you a sense of how thorough
that link is: "output: {all: '| tee -a /var/log/cloud-init-output.log'}"

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1918303

Title:
Randomly set credentials written in cleartext to world-readable file

Status in cloud-init:
New

Bug description:
## Summary

  cloud-init allows administrators to set passwords for user accounts
  via the chpasswd configuration module. Administrators can instruct
  cloud-init to set a random password generated at runtime using the 'R'
  or 'RANDOM' keywords.

  However, cloud-init appears to write all randomly generated passwords
  in cleartext to stderr. Cloud-init's default logging configuration, in
  file /etc/cloud/cloud.cfg.d/05_logging.cfg, redirects both stdout and
  stderr to the log file /var/log/cloud-init-output.log. The file
  /var/log/cloud-init-output.log is world readable. Thus, any
  unprivileged account on the system can view the cleartext password for
  any account which had a random password generated at runtime. The
  credentials are not redacted in the log.

## Reproduction

  Pre-requisites: A device with Ubuntu Server 20.04 installed. Ubuntu
  Server comes with cloud-init pre-installed out of the box, but the
  latest release of cloud-init as of this report (21.1) is not available
  in 20.04's apt repositories. You may need to install v21.1 manually.
  You will also need an exsiting admin account with root privileges.

  1. Login as admin.
  2. Create an unprivileged user account, bob, and set a password. We will use this account to demonstrate unprivileged account access to generated passwords.
  sudo adduser bob
  3. Create another unprivileged user account, alice, and set a password. We will change this account's password with cloud-init.
  sudo adduser alice
  4. Create and open configuration file /etc/cloud/cloud.cfg.d/95_chpasswd.cfg using vim or other editor of your choice.
  sudo vim /etc/cloud/cloud.cfg.d/95_chpasswd.cfg
  5. Add the following chpasswd configuration content to the file then save and exit.
  chpasswd:
    list: |
      alice:RANDOM
  6. cloud-init only runs the chpasswd function on first boot of the OS that cloud-init knows about. For proof of concept purposes, we need to simulate a new instance. Run:
  sudo cloud-init clean
  to reset cloud-init's state.
  7. Reboot the system.
  sudo reboot
  8. Login as unprivileged user bob.
  9. View the password by runnnig
  cat /var/log/cloud-init-output.log | grep alice
  10. Alice's temporary password should appear on terminal in the form alice:<password>
  11. Logout and log back in to the system as alice using the temporary password. You should get access and prompted to set a new password, which confirms the password bob retrieved from the logs is the actual password for alice's account.

## Impact

  Any unprivileged user on the system can retrieve all cloud-init
  randomly set credentials. These could potentially be used to access
  other accounts.

# Notes

  If 'expire: false' is added to the chpasswd config, then leaked
  passwords remain valid until manually changed and increases the risk
  of unauthorized account access. Otherwise, the default behaviour
  prompts accounts to set a new password at next login, reducing the
  time window for unauthorized access.

  Accounts not used for interactive login might not get passwords
  changed or accounts might get a password set but then not authenticate
  for some time. The precise impact and duration of valid exposed
  credentials appears dependent somewhat on each cloud-init customer's
  environment and how they use cloud-init to set credentials.

  I'm not sure the best approach to patch this but perhaps the
  credentials could be written to cloud-init's protected directories or
  files which restrict access to root users only, such as /var/run
  /cloud-init/instance-data-sensitive.json?

  Line 214 of https://github.com/canonical/cloud-
  init/blob/master/cloudinit/config/cc_set_passwords.py checks if any
  random passwords were set and if so prints each one to stderror. This
  might be the root cause.

Tested on Ubuntu Server 20.04.02, cloud-init latest release 21.1 as of report time. If I can provide any further information please let me know. Thanks!
-Carl

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1918303/+subscriptions

Hi Dan,
Thank you for the update and the detailed analysis. I'm not as familiar with some of the cloud-init use cases so good to know what some of the constraints are. Yep, restricting read access to cloud-init-output.log or redacting the messages when sent to the log both sound like workable strategies to me, although as you noted both have some caveats to work around. 
Thanks and have a great day,
Carl
   On Tuesday, March 9, 2021, 2:30:42 PM PST, Dan Watkins <1918303@bugs.launchpad.net> wrote:  
 
 Hi Carl,

Thanks for this detailed and thoughtful bug report, we really appreciate
it.  We've had some preliminary internal discussions, and we have a sync
with our security team scheduled for tomorrow afternoon to discuss
whether or not we think this issue warrants a coordinated release.
Below is my initial analysis.

In order to allow people to access systems using these randomly
generated passwords (without needing another access vector in order to
know the passwords), cloud-init emits them to the serial console.  In
order to have log messages emitted to the console readily available
within the system also, it writes that same content to /var/log/cloud-
init-output.log.  As a result, those passwords are written to that file,
which is world-readable.

In the first instance, we should consider whether or not emitting these
random passwords to the console is acceptable default behaviour.  For
users who are using this functionality as the only way they access the
instance, it is: there's no other way for them to know what password to
use.  The question then becomes, I think, what proportion of users of
this functionality use it in this way?  If it's the majority, then
perhaps emitting them by default is justifiable.  If not (or if we think
this behaviour is simply too dangerous to be the default), then we could
consider toggling it with a new configuration option, defaulting to
false: users are already passing in cloud-config to get into this
situation, so working around this change in behaviour should be
reasonably straightforward.  (I don't think we can remove support for
the emission altogether: that will break existing workflows, and in a
way that is likely to move such folks to simply using hard-coded
passwords: not exactly a security fix.)

Given my (parenthetical) conclusion, we clearly do also need to address
how we are going to remove access to these passwords from unprivileged
users within the system.  Two obvious options present themselves: (a) we
could make cloud-init-output.log only root-readable, or (b) we could
modify the code which emits these messages to either redact them when
going to cloud-init-output.log or omit them from there entirely.  cloud-
init-output.log and the serial console are tied together very
tightly[0], so (b) could prove to be difficult: we'll perform some
further analysis.  (It's possible that the default config in [0] could
be extended to omit the lines using grep, for example?)

Thanks again for the report!  Any comments, questions, or suggestions
(either from you, Carl, or the other cloud-init folks with access to
this report) are more than welcome!

Dan

[0] The default configuration should give you a sense of how thorough
that link is: "output: {all: '| tee -a /var/log/cloud-init-output.log'}"

-- 
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1918303

Title:
  Randomly set credentials written in cleartext to world-readable file

Status in cloud-init:
  New

Bug description:
  ## Summary

cloud-init allows administrators to set passwords for user accounts
  via the chpasswd configuration module. Administrators can instruct
  cloud-init to set a random password generated at runtime using the 'R'
  or 'RANDOM' keywords.

However, cloud-init appears to write all randomly generated passwords
  in cleartext to stderr. Cloud-init's default logging configuration, in
  file /etc/cloud/cloud.cfg.d/05_logging.cfg, redirects both stdout and
  stderr to the log file /var/log/cloud-init-output.log. The file
  /var/log/cloud-init-output.log is world readable. Thus, any
  unprivileged account on the system can view the cleartext password for
  any account which had a random password generated at runtime. The
  credentials are not redacted in the log.

## Reproduction

Pre-requisites: A device with Ubuntu Server 20.04 installed. Ubuntu
  Server comes with cloud-init pre-installed out of the box, but the
  latest release of cloud-init as of this report (21.1) is not available
  in 20.04's apt repositories. You may need to install v21.1 manually.
  You will also need an exsiting admin account with root privileges.

1. Login as admin.
  2. Create an unprivileged user account, bob, and set a password. We will use this account to demonstrate unprivileged account access to generated passwords.
  sudo adduser bob
  3. Create another unprivileged user account, alice, and set a password. We will change this account's password with cloud-init.
  sudo adduser alice
  4. Create and open configuration file /etc/cloud/cloud.cfg.d/95_chpasswd.cfg using vim or other editor of your choice.
  sudo vim /etc/cloud/cloud.cfg.d/95_chpasswd.cfg
  5. Add the following chpasswd configuration content to the file then save and exit.
  chpasswd:
    list: |
      alice:RANDOM
  6. cloud-init only runs the chpasswd function on first boot of the OS that cloud-init knows about. For proof of concept purposes, we need to simulate a new instance. Run:
  sudo cloud-init clean
  to reset cloud-init's state.
  7. Reboot the system.
  sudo reboot
  8. Login as unprivileged user bob.
  9. View the password by runnnig
  cat /var/log/cloud-init-output.log | grep alice
  10. Alice's temporary password should appear on terminal in the form alice:<password>
  11. Logout and log back in to the system as alice using the temporary password. You should get access and prompted to set a new password, which confirms the password bob retrieved from the logs is the actual password for alice's account.

## Impact

Any unprivileged user on the system can retrieve all cloud-init
  randomly set credentials. These could potentially be used to access
  other accounts.

# Notes

If 'expire: false' is added to the chpasswd config, then leaked
  passwords remain valid until manually changed and increases the risk
  of unauthorized account access. Otherwise, the default behaviour
  prompts accounts to set a new password at next login, reducing the
  time window for unauthorized access.

Accounts not used for interactive login might not get passwords
  changed or accounts might get a password set but then not authenticate
  for some time. The precise impact and duration of valid exposed
  credentials appears dependent somewhat on each cloud-init customer's
  environment and how they use cloud-init to set credentials.

I'm not sure the best approach to patch this but perhaps the
  credentials could be written to cloud-init's protected directories or
  files which restrict access to root users only, such as /var/run
  /cloud-init/instance-data-sensitive.json?

Line 214 of https://github.com/canonical/cloud-
  init/blob/master/cloudinit/config/cc_set_passwords.py checks if any
  random passwords were set and if so prints each one to stderror. This
  might be the root cause.

Tested on Ubuntu Server 20.04.02, cloud-init latest release 21.1 as of report time. If I can provide any further information please let me know. Thanks!
  -Carl

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1918303/+subscriptions