Hi Dan,
Thank you for the update and the detailed analysis. I'm not as familiar with some of the cloud-init use cases so good to know what some of the constraints are. Yep, restricting read access to cloud-init-output.log or redacting the messages when sent to the log both sound like workable strategies to me, although as you noted both have some caveats to work around.
Thanks and have a great day,
Carl
On Tuesday, March 9, 2021, 2:30:42 PM PST, Dan Watkins <email address hidden> wrote:
Hi Carl,
Thanks for this detailed and thoughtful bug report, we really appreciate
it. We've had some preliminary internal discussions, and we have a sync
with our security team scheduled for tomorrow afternoon to discuss
whether or not we think this issue warrants a coordinated release.
Below is my initial analysis.
In order to allow people to access systems using these randomly
generated passwords (without needing another access vector in order to
know the passwords), cloud-init emits them to the serial console. In
order to have log messages emitted to the console readily available
within the system also, it writes that same content to /var/log/cloud-
init-output.log. As a result, those passwords are written to that file,
which is world-readable.
In the first instance, we should consider whether or not emitting these
random passwords to the console is acceptable default behaviour. For
users who are using this functionality as the only way they access the
instance, it is: there's no other way for them to know what password to
use. The question then becomes, I think, what proportion of users of
this functionality use it in this way? If it's the majority, then
perhaps emitting them by default is justifiable. If not (or if we think
this behaviour is simply too dangerous to be the default), then we could
consider toggling it with a new configuration option, defaulting to
false: users are already passing in cloud-config to get into this
situation, so working around this change in behaviour should be
reasonably straightforward. (I don't think we can remove support for
the emission altogether: that will break existing workflows, and in a
way that is likely to move such folks to simply using hard-coded
passwords: not exactly a security fix.)
Given my (parenthetical) conclusion, we clearly do also need to address
how we are going to remove access to these passwords from unprivileged
users within the system. Two obvious options present themselves: (a) we
could make cloud-init-output.log only root-readable, or (b) we could
modify the code which emits these messages to either redact them when
going to cloud-init-output.log or omit them from there entirely. cloud-
init-output.log and the serial console are tied together very
tightly[0], so (b) could prove to be difficult: we'll perform some
further analysis. (It's possible that the default config in [0] could
be extended to omit the lines using grep, for example?)
(For (a), I also just noticed that the content in cloud-init-output.log
is also present in the journal for the various cloud-init units.
Whatever resolution we reach regarding cloud-init-output.log, we should
confirm that the permissions that journalctl enforces for such access
are acceptable to us.)
Thanks again for the report! Any comments, questions, or suggestions
(either from you, Carl, or the other cloud-init folks with access to
this report) are more than welcome!
Dan
[0] The default configuration should give you a sense of how thorough
that link is: "output: {all: '| tee -a /var/log/cloud-init-output.log'}"
Title:
Randomly set credentials written in cleartext to world-readable file
Status in cloud-init:
New
Bug description:
## Summary
cloud-init allows administrators to set passwords for user accounts
via the chpasswd configuration module. Administrators can instruct
cloud-init to set a random password generated at runtime using the 'R'
or 'RANDOM' keywords.
However, cloud-init appears to write all randomly generated passwords
in cleartext to stderr. Cloud-init's default logging configuration, in
file /etc/cloud/cloud.cfg.d/05_logging.cfg, redirects both stdout and
stderr to the log file /var/log/cloud-init-output.log. The file
/var/log/cloud-init-output.log is world readable. Thus, any
unprivileged account on the system can view the cleartext password for
any account which had a random password generated at runtime. The
credentials are not redacted in the log.
## Reproduction
Pre-requisites: A device with Ubuntu Server 20.04 installed. Ubuntu
Server comes with cloud-init pre-installed out of the box, but the
latest release of cloud-init as of this report (21.1) is not available
in 20.04's apt repositories. You may need to install v21.1 manually.
You will also need an exsiting admin account with root privileges.
1. Login as admin.
2. Create an unprivileged user account, bob, and set a password. We will use this account to demonstrate unprivileged account access to generated passwords.
sudo adduser bob
3. Create another unprivileged user account, alice, and set a password. We will change this account's password with cloud-init.
sudo adduser alice
4. Create and open configuration file /etc/cloud/cloud.cfg.d/95_chpasswd.cfg using vim or other editor of your choice.
sudo vim /etc/cloud/cloud.cfg.d/95_chpasswd.cfg
5. Add the following chpasswd configuration content to the file then save and exit.
chpasswd:
list: |
alice:RANDOM
6. cloud-init only runs the chpasswd function on first boot of the OS that cloud-init knows about. For proof of concept purposes, we need to simulate a new instance. Run:
sudo cloud-init clean
to reset cloud-init's state.
7. Reboot the system.
sudo reboot
8. Login as unprivileged user bob.
9. View the password by runnnig
cat /var/log/cloud-init-output.log | grep alice
10. Alice's temporary password should appear on terminal in the form alice:<password>
11. Logout and log back in to the system as alice using the temporary password. You should get access and prompted to set a new password, which confirms the password bob retrieved from the logs is the actual password for alice's account.
## Impact
Any unprivileged user on the system can retrieve all cloud-init
randomly set credentials. These could potentially be used to access
other accounts.
# Notes
If 'expire: false' is added to the chpasswd config, then leaked
passwords remain valid until manually changed and increases the risk
of unauthorized account access. Otherwise, the default behaviour
prompts accounts to set a new password at next login, reducing the
time window for unauthorized access.
Accounts not used for interactive login might not get passwords
changed or accounts might get a password set but then not authenticate
for some time. The precise impact and duration of valid exposed
credentials appears dependent somewhat on each cloud-init customer's
environment and how they use cloud-init to set credentials.
I'm not sure the best approach to patch this but perhaps the
credentials could be written to cloud-init's protected directories or
files which restrict access to root users only, such as /var/run
/cloud-init/instance-data-sensitive.json?
Line 214 of https://github.com/canonical/cloud-
init/blob/master/cloudinit/config/cc_set_passwords.py checks if any
random passwords were set and if so prints each one to stderror. This
might be the root cause.
Tested on Ubuntu Server 20.04.02, cloud-init latest release 21.1 as of report time. If I can provide any further information please let me know. Thanks!
-Carl
Hi Dan, output. log or redacting the messages when sent to the log both sound like workable strategies to me, although as you noted both have some caveats to work around.
Thank you for the update and the detailed analysis. I'm not as familiar with some of the cloud-init use cases so good to know what some of the constraints are. Yep, restricting read access to cloud-init-
Thanks and have a great day,
Carl
On Tuesday, March 9, 2021, 2:30:42 PM PST, Dan Watkins <email address hidden> wrote:
Hi Carl,
Thanks for this detailed and thoughtful bug report, we really appreciate
it. We've had some preliminary internal discussions, and we have a sync
with our security team scheduled for tomorrow afternoon to discuss
whether or not we think this issue warrants a coordinated release.
Below is my initial analysis.
In order to allow people to access systems using these randomly
generated passwords (without needing another access vector in order to
know the passwords), cloud-init emits them to the serial console. In
order to have log messages emitted to the console readily available
within the system also, it writes that same content to /var/log/cloud-
init-output.log. As a result, those passwords are written to that file,
which is world-readable.
In the first instance, we should consider whether or not emitting these
random passwords to the console is acceptable default behaviour. For
users who are using this functionality as the only way they access the
instance, it is: there's no other way for them to know what password to
use. The question then becomes, I think, what proportion of users of
this functionality use it in this way? If it's the majority, then
perhaps emitting them by default is justifiable. If not (or if we think
this behaviour is simply too dangerous to be the default), then we could
consider toggling it with a new configuration option, defaulting to
false: users are already passing in cloud-config to get into this
situation, so working around this change in behaviour should be
reasonably straightforward. (I don't think we can remove support for
the emission altogether: that will break existing workflows, and in a
way that is likely to move such folks to simply using hard-coded
passwords: not exactly a security fix.)
Given my (parenthetical) conclusion, we clearly do also need to address output. log only root-readable, or (b) we could output. log or omit them from there entirely. cloud-
how we are going to remove access to these passwords from unprivileged
users within the system. Two obvious options present themselves: (a) we
could make cloud-init-
modify the code which emits these messages to either redact them when
going to cloud-init-
init-output.log and the serial console are tied together very
tightly[0], so (b) could prove to be difficult: we'll perform some
further analysis. (It's possible that the default config in [0] could
be extended to omit the lines using grep, for example?)
(For (a), I also just noticed that the content in cloud-init- output. log output. log, we should
is also present in the journal for the various cloud-init units.
Whatever resolution we reach regarding cloud-init-
confirm that the permissions that journalctl enforces for such access
are acceptable to us.)
Thanks again for the report! Any comments, questions, or suggestions
(either from you, Carl, or the other cloud-init folks with access to
this report) are more than welcome!
Dan
[0] The default configuration should give you a sense of how thorough cloud-init- output. log'}"
that link is: "output: {all: '| tee -a /var/log/
-- /bugs.launchpad .net/bugs/ 1918303
You received this bug notification because you are subscribed to the bug
report.
https:/
Title:
Randomly set credentials written in cleartext to world-readable file
Status in cloud-init:
New
Bug description:
## Summary
cloud-init allows administrators to set passwords for user accounts
via the chpasswd configuration module. Administrators can instruct
cloud-init to set a random password generated at runtime using the 'R'
or 'RANDOM' keywords.
However, cloud-init appears to write all randomly generated passwords cloud.cfg. d/05_logging. cfg, redirects both stdout and cloud-init- output. log. The file log/cloud- init-output. log is world readable. Thus, any
in cleartext to stderr. Cloud-init's default logging configuration, in
file /etc/cloud/
stderr to the log file /var/log/
/var/
unprivileged account on the system can view the cleartext password for
any account which had a random password generated at runtime. The
credentials are not redacted in the log.
## Reproduction
Pre-requisites: A device with Ubuntu Server 20.04 installed. Ubuntu
Server comes with cloud-init pre-installed out of the box, but the
latest release of cloud-init as of this report (21.1) is not available
in 20.04's apt repositories. You may need to install v21.1 manually.
You will also need an exsiting admin account with root privileges.
1. Login as admin. cloud.cfg. d/95_chpasswd. cfg using vim or other editor of your choice. cloud.cfg. d/95_chpasswd. cfg cloud-init- output. log | grep alice
2. Create an unprivileged user account, bob, and set a password. We will use this account to demonstrate unprivileged account access to generated passwords.
sudo adduser bob
3. Create another unprivileged user account, alice, and set a password. We will change this account's password with cloud-init.
sudo adduser alice
4. Create and open configuration file /etc/cloud/
sudo vim /etc/cloud/
5. Add the following chpasswd configuration content to the file then save and exit.
chpasswd:
list: |
alice:RANDOM
6. cloud-init only runs the chpasswd function on first boot of the OS that cloud-init knows about. For proof of concept purposes, we need to simulate a new instance. Run:
sudo cloud-init clean
to reset cloud-init's state.
7. Reboot the system.
sudo reboot
8. Login as unprivileged user bob.
9. View the password by runnnig
cat /var/log/
10. Alice's temporary password should appear on terminal in the form alice:<password>
11. Logout and log back in to the system as alice using the temporary password. You should get access and prompted to set a new password, which confirms the password bob retrieved from the logs is the actual password for alice's account.
## Impact
Any unprivileged user on the system can retrieve all cloud-init
randomly set credentials. These could potentially be used to access
other accounts.
# Notes
If 'expire: false' is added to the chpasswd config, then leaked
passwords remain valid until manually changed and increases the risk
of unauthorized account access. Otherwise, the default behaviour
prompts accounts to set a new password at next login, reducing the
time window for unauthorized access.
Accounts not used for interactive login might not get passwords
changed or accounts might get a password set but then not authenticate
for some time. The precise impact and duration of valid exposed
credentials appears dependent somewhat on each cloud-init customer's
environment and how they use cloud-init to set credentials.
I'm not sure the best approach to patch this but perhaps the init/instance- data-sensitive. json?
credentials could be written to cloud-init's protected directories or
files which restrict access to root users only, such as /var/run
/cloud-
Line 214 of https:/ /github. com/canonical/ cloud- blob/master/ cloudinit/ config/ cc_set_ passwords. py checks if any
init/
random passwords were set and if so prints each one to stderror. This
might be the root cause.
Tested on Ubuntu Server 20.04.02, cloud-init latest release 21.1 as of report time. If I can provide any further information please let me know. Thanks!
-Carl
To manage notifications about this bug go to: /bugs.launchpad .net/cloud- init/+bug/ 1918303/ +subscriptions
https:/