fully support package installation in systemd

Would it be possible to move package installation into a cloud-init-*.service that is "Type=idle", i. e. runs after booting is complete? This would avoid all these corner cases and breakage when trying to install/start things while booting is not complete yet.

Martin,

Probably worth noting that this impacts upon the configuration systems as well. I'm using the PostgreSQL puppet configuration system, and that will sit in a loop waiting for PostgresQL to come up before moving onto the next stage of the configuration.

So if you are using puppet within cloud-init, and cloud events delay the start event until the boot is complete, then the configurator that expects things to happen in sequence will break.

It looks to me that large chunks of cloud-init need to be moved so it runs after 'multi-user.target' has been reached, not just package installation.

An update to this, I think for the moment the plan is to move many of the config modules that run in 'config_modules' to 'final_modules' and to move final_modules to run as idle.

I dont love it, but it seems like the only actual path to package installation to work.

It's worth mentioning the scope of the package upgrade/install issue related to systemd.

For packages like apache2 which do not use dependent systemd service files, those service packages install and start properly.

For packages with dependent service files, like postgresql (it has both a postgresql.service (a dummy) and systemd generator service which creates a postgresql@<version>-<dbname> service; they currently do not start automatically due to being installed during cloud-init's cloud-config.service unit execution.

W.r.t package upgrades, the issue is scoped to service packages in the image which have an update that's not in the current image and also require an update to systemd services.

I have a branch at https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/bug/1576692 which:
a.) moves cloud-init-final to be Type=idle
b.) moves config modules such as package-update-upgrade-install to run in the final_modules

I've patched a lxc container with that, and then launched several instances. My experience is that out of 5 containers 2 or 3 of them will have a running postgres at the end (per systemctl status postgresql@9.5-main).

The user-data I'm providing is just:
 #cloud-config
 packages: [postgresql]
 runcmd:
   - "systemctl status postgresql@9.5-main > /run/my-status || true"

Then you can just look at /run/my-status.

To start a patched image what you can do is:
 n=y1
 lxc init ubuntu-daily:yakkety $n "--config=user.user-data=$(cat my.user-data)"
 lxc-chroot $n -- sh -ec 'd=/tmp/my.deb; trap "rm -f $d" EXIT; cat > $d && dpkg -i $d' < "$deb"
 lxc start $n

I cannot see failed containers in your cloud instance, nor reproduce the failure by starting new ones.

I have also created and run http://people.canonical.com/~pitti/tmp/psql-idle.sh on my laptop and your cloud instance for 40 iterations, and I couldn't reproduce a failure. This takes cloud-init out of the equation and just tests running apt install postgresq in a Type=idle unit (plus some glue around it to wait for booting and iterate). So I'm fairly sure that this approach works in principle -- but of course with more moving parts there's more that can go wrong.

The bit that I have doubts about in https://git.launchpad.net/~smoser/cloud-init/commit/?h=bug/1576692&id=6a249689a179f is why "runcmd" still runs in cloud_config_modules -- it's arbitrary code which might (and often does) run package installs, so it should really live in cloud_final_modules as well?

Scott and I debugged this further, and the best hint so far is bug 1620780. In Scott's local instances he gets "systemctl is-system-running" == "starting" with

JOB UNIT TYPE STATE
  2 dev-sda2.device start running

postgresql-9.5.postinst calls "invoke-rc.d postgresql start", and since the system is not booted yet (according to is-system-running) it starts the postgresql.service wrapper job with --job-mode=ignore-dependencies, and thus it never starts the @9.5-main instance.

I suggest to handle this bit (which is LXD specific) in bug 1620780, and keep this bug for the cloud-init change.

Type=idle not waiting for running *.device jobs is related, but not identical to bug 1620780. I filed that as bug 1621846. Those are both on the systemd side, and for the cases where bug 1620780 does not hit (and thus bug 1621846 does not happen), it has been demonstrated that moving these units after the boot process works in principle.

Pre-weekend braindump: I've had success with modifying a xenial image like that:

/usr/sbin/invoke-rc.d:

               if ! systemctl --quiet is-active default.target; then
                    sctl_args="--job-mode=ignore-dependencies"
               fi

Add multi-user.target to cloud-{config,final}.service

Then this works:

  lxc launch ubuntu-xenial-mod --config=user.user-data="$(printf "#cloud-config\npackages: [postgresql, samba, postfix]")" x1

Services start:

# systemctl list-units --no-legend postg* samb* postfix*
postfix.service loaded active running LSB: Postfix Mail Transport Agent
postgresql.service loaded active exited PostgreSQL RDBMS
postgresql@9.5-main.service loaded active running PostgreSQL Cluster 9.5-main
samba-ad-dc.service loaded active exited LSB: start Samba daemons for the AD DC

and reboot works fine.

> Add multi-user.target to cloud-{config,final}.service

Sorry, I meant to add After=multi-user.target

Change invoke-rc.d to check is-active multi-user.target instead of is-system-running, to match After=multi-user.target.
Also, always use --no-block for reload as an additional line of defence for if-up.d/ scripts, as reload has never been synchronous.

For invoke-rc.d: original commit https://anonscm.debian.org/cgit/collab-maint/sysvinit.git/commit/?id=38e2b9fca

Try to reproduce the hangs in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=777113 when completely removing the hack, then ensure that they go away again with is-active m-u.target

This bug was fixed in the package cloud-init - 0.7.7-28-g34a26f7-0ubuntu1

---------------
cloud-init (0.7.7-28-g34a26f7-0ubuntu1) yakkety; urgency=medium

  * New upstream snapshot.
    - systemd: Better support package and upgrade.
      (LP: #1576692, #1621336)
    - tests: cleanup tempdirs in apt_source tests

 -- Scott Moser <email address hidden> Fri, 09 Sep 2016 16:01:13 -0400

Just a comment / status here.
cloud-init is now running the modules that do package installation after multi-user.target. the plan is to change init-system-helpers as pitti described in comment 10. Until that is fixed, the problem isn't really fixed.

@Scott: Oh, does that actually work with adding the After= to just cloud-final.service? I thought the thing that *actually* does the package installs is cloud-config.service, and this needed that After= as well?

https://anonscm.debian.org/cgit/collab-maint/init-system-helpers.git/commit/?id=1460d6a02

I also committed https://anonscm.debian.org/cgit/collab-maint/init-system-helpers.git/commit/?id=9cfb6dfed to further robustify the behaviour, but it is not required to fix this bug.

i-s-h SRU uploaded.

Martin,
I moved all things that do package installation into final. The user could still manage to have some things run at 'config' point in boot that would install packages, but anything that does it in cloud-init directly is now part of final.

fixed in 0.7.8.

@smoser

Did you commit your changes to the xenial cloud-init as well? I'm not sure where xenial images grab cloud init for themselves, but I assume out of the xenial archives. Am I missing something here?

We use what's in the archive for what we include in cloud images. So once cloud-init lands, it will makes it way to an image. I would expect that Scott working his way through the SRU process.

This bug was fixed in the package init-system-helpers - 1.44

---------------
init-system-helpers (1.44) unstable; urgency=medium

  * invoke-rc.d, service: Check for multi-user.target instead of graphical.target.
    There is a curious bug which sometimes causes "systemctl is-active
    default.target" to say inactive until "show" or "status" gets called on
    the unit. This needs to be investigated. Until then, check for
    multi-user.target which by and large does the same job, but seems to work
    reliably.

 -- Martin Pitt <email address hidden> Mon, 12 Sep 2016 22:52:23 +0200

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Have we back ported the init-system-helpers changes to Xenial?

I'm only seeing 1.29ubuntu2 this morning.

init-system-helpers is still sitting in the SRU queue and needs to be reviewed/accepted.

Hello Scott, or anyone else affected,

Accepted init-system-helpers into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/init-system-helpers/1.29ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Added both cloud-ini t and init-system-helpers from proposed to the standard Xenial cloud image (com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:16.04:amd64/20160907.1/disk1.img) on a suitably sized server.

Reset the cloud init with rm -rf /var/lib/cloud/instances/*, shutdown the server and snapshotted the image.

Rebuilt a new server from the snapshotted image using the previously failing postgresql user data and all is well. The new packages correct my problem - bug 1611973

Thank you Neil!

I've been going through my testing here, and found:
* bug 1623570: Azure: cannot start walinux agent (Transaction order is cyclic.)

That will require us to get that fix in and through proposed or we will break Azure boot. Its fallout of the systemd ordering.

I just filed bug 1623868 which is fallout from this change, so blocking this SRU for now.

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-1-g3705bb5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

verified with:
printf "#cloud-config\npackages: [postgresql, samba, postfix]\n" > user-data
n=x1
lxc launch ubuntu-daily:xenial $n
sleep 10
lxc exec $n -- sh -c '
    p=/etc/apt/sources.list.d/proposed.list
    echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main > "$p" &&
    apt-get update -q && apt-get -qy install cloud-init'

lxc file push - $n/etc/cloud/cloud.cfg.d/update.cfg < user-data

## clean it out so next is first boot.
lxc exec $n -- sh -c '
  cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
  rm -Rf /var/log/cloud-init*'

lxc exec $n reboot
lxc exec $n -- tail -f /var/log/cloud-init-output.log

This bug was fixed in the package cloud-init - 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

---------------
cloud-init (0.7.8-1-g3705bb5-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream release 0.7.8.
  * New upstream snapshot.
    - systemd: put cloud-init.target After multi-user.target (LP: #1623868)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * debian/control: add Breaks of older versions of walinuxagent (LP: #1623570)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/control: fix missing dependency on python3-serial,
    and make SmartOS datasource work.
  * debian/cloud-init.templates fix capitalisation in template so
    dpkg-reconfigure works to select OpenStack. (LP: #1575727)
  * d/README.source, d/control, d/new-upstream-snapshot, d/rules: sync
    with yakkety for changes due to move to git.
  * d/rules: change PYVER=python3 to PYVER=3 to adjust to upstream change.
  * debian/rules, debian/cloud-init.install: remove install file
    to ensure expected files are collected into cloud-init deb.
    (LP: #1615745)
  * debian/dirs: remove obsolete / unused file.
  * upstream move from bzr to git.
  * New upstream snapshot.
    - Allow link type of null in network_data.json [Jon Grimm] (LP: #1621968)
    - DataSourceOVF: fix user-data as base64 with python3 (LP: #1619394)
    - remove obsolete .bzrignore
    - systemd: Better support package and upgrade. (LP: #1576692, #1621336)
    - tests: cleanup tempdirs in apt_source tests
    - apt config conversion: treat empty string as not provided. (LP: #1621180)
    - Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
    - salt minion: update default pki directory for newer salt minion.
      (LP: #1609899)
    - bddeb: add --release flag to specify the release in changelog.
    - apt-config: allow both old and new format to be present.
      [Christian Ehrhardt] (LP: #1616831)
    - python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
    - Add a module that can configure spacewalk. [Joshua Harlow]
    - add install option for openrc [Matthew Thode]
    - Generate a dummy bond name for OpenStack (LP: #1605749)
    - network: fix get_interface_mac for bond slave, read_sys_net for ENOTDIR
    - azure dhclient-hook cleanups
    - Minor cleanups to atomic_helper and add unit tests.
    - Fix Gentoo net config generation [Matthew Thode]
    - distros: fix get_primary_arch method use of os.uname [Andrew Jorgensen]
    - Apt: add new apt configuration format [Christian Ehrhardt]
    - Get Azure endpoint server from DHCP client [Brent Baude]
    - DigitalOcean: use the v1.json endpoint [Ben Howard]
    - MAAS: add vendor-data support (LP: #1612313)
    - Upgrade to a configobj package new enough to work [Joshua Harlow]
    - ConfigDrive: recognize 'tap' as a link type. (LP: #1610784)
    - NoCloud: fix bug providing network-interfaces via meta-data.
      (LP: 1577982)
    - Add distro tags on config modules that should have it [Joshua Harlow]
    - ChangeLog: update changelog for previous commit.
    - add ntp config module [Ryan Harper]
    - SmartOS: more improvement...

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package init-system-helpers - 1.29ubuntu3

---------------
init-system-helpers (1.29ubuntu3) xenial-proposed; urgency=medium

  * invoke-rc.d, service: Only ignore systemd unit dependencies before
    multi-user.target. "systemctl is-system-running" might still be false in
    case of running jobs for device/mount/hotplug/dynamic actions units. But
    in those cases we already do want to respect unit dependencies, as the
    system is booted up sufficiently to avoid dependency loops. Thus weaken
    the condition to "multi-user.target is active".

    This does not change the behaviour for single-user: is-system-running has
    always been false there, so dependencies continue to be ignored.

    Fixes installation of packages like PostgreSQL under cloud-init or when
    manually installing packages right after booting.

    LP: #1576692

 -- Martin Pitt <email address hidden> Mon, 12 Sep 2016 10:57:57 +0200

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2662