| 2016-09-15 09:42:31 |
Martin Pitt |
bug |
|
|
added bug |
| 2016-09-15 09:42:44 |
Martin Pitt |
tags |
|
regression-release yakkety |
|
| 2016-09-15 09:42:46 |
Martin Pitt |
cloud-init (Ubuntu): importance |
Undecided |
High |
|
| 2016-09-15 09:54:00 |
Launchpad Janitor |
cloud-init (Ubuntu): status |
New |
Confirmed |
|
| 2016-09-15 13:14:46 |
Scott Moser |
attachment added |
|
journalctl of vm where cloud-final was deleted https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1623868/+attachment/4741407/+files/journalctl-break-cloud-final.log |
|
| 2016-09-15 13:15:10 |
Scott Moser |
attachment added |
|
journalctl of vm where cloud-init.target was deleted https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1623868/+attachment/4741408/+files/journalctl-break-cloud-init-target.log |
|
| 2016-09-15 14:32:04 |
Martin Pitt |
cloud-init (Ubuntu Xenial): status |
New |
Fix Committed |
|
| 2016-09-15 14:32:07 |
Martin Pitt |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-09-15 14:32:09 |
Martin Pitt |
bug |
|
|
added subscriber SRU Verification |
| 2016-09-15 14:32:13 |
Martin Pitt |
tags |
regression-release yakkety |
regression-release verification-needed yakkety |
|
| 2016-09-15 14:37:02 |
Scott Moser |
description |
With current yakkety cloud images (at least in Scalingstack), I often run into this dependency cycle at boot:
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found ordering cycle on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on multi-user.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Breaking ordering cycle by deleting job cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-final.service: Job cloud-final.service/start deleted to break ordering cycle starting with cloud-init.target/start
● cloud-final.service - Execute cloud user/final scripts
Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Thus /var/lib/cloud/instance/boot-finished never gets written and thus waiting for an instance to init just times out.
This is with the most recent https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1 |
==== Begin SRU Template ====
[Impact]
As part of the change in bug 1623868, we made cloud-final.target
run After multi-user.target. That created a dependency loop between
cloud-init.target and multi-user.target and cloud-final.target.
Most of the time systemd would break that loop by dropping
cloud-init.target. But sometimes, it would break the loop by dropping
cloud-final.target, which would mean that user scripts do not run
and generally cloud-init doesn't finish.
[Test Case]
## Failure in a xenial image can only be reproduced by
## patching an image up to the previous xenial-proposed upload
## (0.7.7-31-g65ace7b-0ubuntu1~16.04.1), then cleaning it
## and then restarting. We will focus on verifying there is not
## a problem.
## Launch an instance and patch it up to xenial-proposed
$ release=xenial
$ name=x1
$ lxc launch ubuntu-daily:$release $name
# wait for it to boot
$ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done
## Now update container, clean and reboot to show first boot
$ lxc exec $name -- sh -c '
p=/etc/apt/sources.list.d/proposed.list
echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
apt-get update -q && apt-get -qy install cloud-init'
$ lxc exec $name -- sh -c '
cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
rm -Rf /var/log/cloud-init*'
## This is like first boot now.
$ lxc exec $name reboot
$ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done
## The services should show active
$ lxc exec $name -- journalctl | grep Break || echo Good, no breaks
$ lxc exec $name -- systemctl --no-pager status cloud-final.service
$ lxc exec $name -- systemctl --no-pager status cloud-init.target
[Regression Potential]
Playing with boot order can cause problems. Regression would be around
some targets not running. On a booted system this would show itself inx
journalctl | grep -i Break
or
journalctl | grep -i ordering
==== End SRU Template ====
With current yakkety cloud images (at least in Scalingstack), I often run into this dependency cycle at boot:
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found ordering cycle on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on multi-user.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Breaking ordering cycle by deleting job cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-final.service: Job cloud-final.service/start deleted to break ordering cycle starting with cloud-init.target/start
● cloud-final.service - Execute cloud user/final scripts
Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Thus /var/lib/cloud/instance/boot-finished never gets written and thus waiting for an instance to init just times out.
This is with the most recent https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1 |
|
| 2016-09-15 14:39:17 |
Scott Moser |
description |
==== Begin SRU Template ====
[Impact]
As part of the change in bug 1623868, we made cloud-final.target
run After multi-user.target. That created a dependency loop between
cloud-init.target and multi-user.target and cloud-final.target.
Most of the time systemd would break that loop by dropping
cloud-init.target. But sometimes, it would break the loop by dropping
cloud-final.target, which would mean that user scripts do not run
and generally cloud-init doesn't finish.
[Test Case]
## Failure in a xenial image can only be reproduced by
## patching an image up to the previous xenial-proposed upload
## (0.7.7-31-g65ace7b-0ubuntu1~16.04.1), then cleaning it
## and then restarting. We will focus on verifying there is not
## a problem.
## Launch an instance and patch it up to xenial-proposed
$ release=xenial
$ name=x1
$ lxc launch ubuntu-daily:$release $name
# wait for it to boot
$ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done
## Now update container, clean and reboot to show first boot
$ lxc exec $name -- sh -c '
p=/etc/apt/sources.list.d/proposed.list
echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
apt-get update -q && apt-get -qy install cloud-init'
$ lxc exec $name -- sh -c '
cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
rm -Rf /var/log/cloud-init*'
## This is like first boot now.
$ lxc exec $name reboot
$ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done
## The services should show active
$ lxc exec $name -- journalctl | grep Break || echo Good, no breaks
$ lxc exec $name -- systemctl --no-pager status cloud-final.service
$ lxc exec $name -- systemctl --no-pager status cloud-init.target
[Regression Potential]
Playing with boot order can cause problems. Regression would be around
some targets not running. On a booted system this would show itself inx
journalctl | grep -i Break
or
journalctl | grep -i ordering
==== End SRU Template ====
With current yakkety cloud images (at least in Scalingstack), I often run into this dependency cycle at boot:
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found ordering cycle on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on multi-user.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Breaking ordering cycle by deleting job cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-final.service: Job cloud-final.service/start deleted to break ordering cycle starting with cloud-init.target/start
● cloud-final.service - Execute cloud user/final scripts
Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Thus /var/lib/cloud/instance/boot-finished never gets written and thus waiting for an instance to init just times out.
This is with the most recent https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1 |
==== Begin SRU Template ====
[Impact]
As part of the change in bug 1576692, we made cloud-final.target
run After multi-user.target. That created a dependency loop between
cloud-init.target and multi-user.target and cloud-final.target.
Most of the time systemd would break that loop by dropping
cloud-init.target. But sometimes, it would break the loop by dropping
cloud-final.target, which would mean that user scripts do not run
and generally cloud-init doesn't finish.
[Test Case]
## Failure in a xenial image can only be reproduced by
## patching an image up to the previous xenial-proposed upload
## (0.7.7-31-g65ace7b-0ubuntu1~16.04.1), then cleaning it
## and then restarting. We will focus on verifying there is not
## a problem.
## Launch an instance and patch it up to xenial-proposed
$ release=xenial
$ name=x1
$ lxc launch ubuntu-daily:$release $name
# wait for it to boot
$ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done
## Now update container, clean and reboot to show first boot
$ lxc exec $name -- sh -c '
p=/etc/apt/sources.list.d/proposed.list
echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
apt-get update -q && apt-get -qy install cloud-init'
$ lxc exec $name -- sh -c '
cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
rm -Rf /var/log/cloud-init*'
## This is like first boot now.
$ lxc exec $name reboot
$ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done
## The services should show active
$ lxc exec $name -- journalctl | grep Break || echo Good, no breaks
$ lxc exec $name -- systemctl --no-pager status cloud-final.service
$ lxc exec $name -- systemctl --no-pager status cloud-init.target
[Regression Potential]
Playing with boot order can cause problems. Regression would be around
some targets not running. On a booted system this would show itself inx
journalctl | grep -i Break
or
journalctl | grep -i ordering
==== End SRU Template ====
With current yakkety cloud images (at least in Scalingstack), I often run into this dependency cycle at boot:
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found ordering cycle on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on multi-user.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-init.target/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Breaking ordering cycle by deleting job cloud-final.service/start
Sep 15 09:28:51 ubuntu systemd[1]: cloud-final.service: Job cloud-final.service/start deleted to break ordering cycle starting with cloud-init.target/start
● cloud-final.service - Execute cloud user/final scripts
Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Thus /var/lib/cloud/instance/boot-finished never gets written and thus waiting for an instance to init just times out.
This is with the most recent https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1
Related bugs:
* bug 1576692: fully support package installation in systemd |
|
| 2016-09-15 15:14:33 |
Launchpad Janitor |
cloud-init (Ubuntu): status |
Confirmed |
Fix Released |
|
| 2016-09-17 02:38:26 |
Scott Moser |
tags |
regression-release verification-needed yakkety |
regression-release verification-done yakkety |
|
| 2016-09-22 11:28:07 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2016-09-22 17:34:08 |
Launchpad Janitor |
cloud-init (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2016-09-22 17:36:05 |
Chris J Arges |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2016-12-23 17:42:17 |
Scott Moser |
bug task added |
|
cloud-init |
|
| 2016-12-23 17:42:33 |
Scott Moser |
cloud-init: status |
New |
Fix Released |
|
| 2023-05-10 17:00:58 |
James Falcon |
bug watch added |
|
https://github.com/canonical/cloud-init/issues/2728 |
|
