shutdown-scripts service is restarted on package upgrade - causing any configured shutdown scripts to be run

Bug #2019089 reported by Philip Roche
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
google-guest-agent (Ubuntu)
Status tracked in Mantic
Bionic
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
Kinetic
Fix Released
Undecided
Unassigned
Lunar
Fix Released
Undecided
Unassigned
Mantic
Fix Released
Undecided
Philip Roche

Bug Description

[Impact]
========

Initially reported @ https://github.com/GoogleCloudPlatform/guest-agent/issues/119

Basically anytime the google-guest-agent package is updated, these services are restarted which means the customer scripts are run at a time they are not supposed to be. This caused an outage for one customer who had some logic in their shutdown script that basically turned off their services.

The issue is most certainly in the autogenerated preinst script which should not be there. Stopping the shutdown service will invoke the shutdown script. This wasn't there in previous Ubuntu versions.

Ubuntu 22.04 preinst script
```
#!/bin/sh
set -e
# Automatically added by dh_installdeb/13.6ubuntu1
dpkg-maintscript-helper rm_conffile /etc/defaults/instance_configs.cfg 20200617.00-0ubuntu2\~ -- "$@"
# End automatically added section
# Automatically added by dh_installsystemd/13.6ubuntu1
if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = upgrade ] && [ -d /run/systemd/system ] ; then
deb-systemd-invoke stop 'google-shutdown-scripts.service' 'google-startup-scripts.service' >/dev/null || true
fi
# End automatically added section
```

Ubuntu 20.04 preinst script

```
root@ubu2004:/tmp/DEBIAN# cat preinst
#!/bin/sh
set -e
# Automatically added by dh_installdeb/12.10ubuntu1
dpkg-maintscript-helper rm_conffile /etc/defaults/instance_configs.cfg 20200617.00-0ubuntu2\~ -- "$@"
# End automatically added section
```

One reason this issue was of concern as the latest google-guest-agent update was a rebuild promoted by golang-1.18 being updated to address CVEs prompting a rebuild of all packages dependant on golang-1.18 and publication of those rebuilds to the -security pocket. unattended-upgrades then upgraded google-guest-agent in the background which restarted the shutdown-scripts.

We do want the services to be enabled on initial installation and to be started on instance start but they should not be restarted/stopped during the lifetime of the instance.

[Test Plan]
===========

After having the fixes in, we'll run the reproducer and the test script mentioned in the first comment: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2019089/comments/1.

This will help us validate that the fix is indeed working as it should.

A demonstration of that reproducer script is shown in the 2nd comment:
https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2019089/comments/2.

Further, we'll also build the images and run the CTF tests (internal tests) on them on our side and then ask Google to validate these images to make sure they also confirm that the bug is indeed fix and there's no regression or fallout with this update.

Once Google comes back with the validation, we'll assume enough testing and verification has been done and will mark the bug with the same.

[Where Problems Could Occur]
============================

The patch just adds "--no-stop-on-upgrade" to the already existing "--no-start --no-restart-after-upgrade". So if somebody is relying on that behavior or has a workaround that somehow, it'll break the setup for them. But this update should bring them the right fix. I don't expect a lot of users to regress because of that, though.

Related branches

Revision history for this message
Philip Roche (philroche) wrote :
Revision history for this message
Philip Roche (philroche) wrote (last edit ):

I have a patched google-guest-agent for Ubuntu 22.04 in PPA https://launchpad.net/~philroche/+archive/ubuntu/google-guest-agent/

And reproducer script and logs @ https://people.canonical.com/~philroche/20230508-google-guest-agent-regression/

The reproducer confirms that the shutdown script is no longer run on upgrade.

The patch was as follows:

```
--- control/preinst
+++ control/preinst
@@ -3,8 +3,3 @@
# Automatically added by dh_installdeb/13.6ubuntu1
dpkg-maintscript-helper rm_conffile /etc/defaults/instance_configs.cfg 20200617.00-0ubuntu2\~ -- "$@"
# End automatically added section
-# Automatically added by dh_installsystemd/13.6ubuntu1
-if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = upgrade ] && [ -d /run/systemd/system ] ; then
- deb-systemd-invoke stop 'google-shutdown-scripts.service' 'google-startup-scripts.service' >/dev/null || true
-fi
-# End automatically added section
```

This was achieved by updating the `debian/rules` files

```
--- debian/rules
+++ debian/rules
@@ -17,7 +17,7 @@

override_dh_installsystemd:
dh_installsystemd google-guest-agent.service
- dh_installsystemd --no-start --no-restart-after-upgrade google-startup-scripts.service google-shutdown-scripts.service
+ dh_installsystemd --no-start --no-restart-after-upgrade --no-restart-on-upgrade google-startup-scripts.service google-shutdown-scripts.service

override_dh_clean:
dh_clean vendor/

```

Changed in google-guest-agent (Ubuntu):
assignee: nobody → Philip Roche (philroche)
status: New → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Phil,
there has been a lot of confusion in those options.

You suggest:
1.
 --no-restart-after-upgrade - which actually means "no ... after" which translates to "stop it early and start it late"
2.
 --no-restart-on-upgrade - which is known to be confusing, see the man page "Note that the --no-restart-on-upgrade alias is deprecated and will be removed in compat 14. This is to avoid confusion with the --no-restart-after-upgrade option."

While you touch it you should change it in the way it hopefully can stay, which would IMHO be:

dh_installsystemd --no-start --no-stop-on-upgrade google-startup-scripts.service google-shutdown-scripts.service

I see this only goes back to Jammy, so the options would be there already (no backport pain)
WDYT?

Revision history for this message
Philip Roche (philroche) wrote :

@paelzer Thanks

> I see this only goes back to Jammy

Yes this affects jammy+ only but the patch would likely go back to previous releases too as part of the SRU exception and future bug and feature releases.

I have verified your proposed change too and the resulting `preinst` remains the same

```
dh_installsystemd --no-start --no-stop-on-upgrade google-startup-scripts.service google-shutdown-scripts.service
```

I see from man pages [bionic](https://manpages.ubuntu.com/manpages/bionic/en/man1/dh_installsystemd.1.html),[focal]( https://manpages.ubuntu.com/manpages/focal/en/man1/dh_installsystemd.1.html), [jammy](https://manpages.ubuntu.com/manpages/jammy/en/man1/dh_installsystemd.1.html), [kinetic+](https://manpages.ubuntu.com/manpages/kinetic/en/man1/dh_installsystemd.1.html) that --no-stop-on-upgrade` is available so, as you say, likely no issues with backporting.

Revision history for this message
Philip Roche (philroche) wrote :

As per @paelzer's suggesting the proposed patch is now

```
--- debian/rules
+++ debian/rules
@@ -17,7 +17,7 @@

 override_dh_installsystemd:
  dh_installsystemd google-guest-agent.service
- dh_installsystemd --no-start --no-restart-after-upgrade google-startup-scripts.service google-shutdown-scripts.service
-
+ dh_installsystemd --no-start --no-stop-on-upgrade google-startup-scripts.service google-shutdown-scripts.service
+
 override_dh_clean:
  dh_clean vendor/

```

Revision history for this message
Philip Roche (philroche) wrote :

https://launchpad.net/~philroche/+archive/ubuntu/google-guest-agent/ PPA now has patched versions of google-guest-agent and I have verified that it addresses the problem for all releases.

I will now work to create merge proposals for each release and seek sponsorship of upload to -proposed.

Revision history for this message
Philip Roche (philroche) wrote :
Revision history for this message
Philip Roche (philroche) wrote :

I have rebuilt the unpatched focal packages and published to https://launchpad.net/~philroche/+archive/ubuntu/google-guest-agent/ but this does not result in the troublesome `preinst` section like we had in jammy+

```
# Automatically added by dh_installsystemd/13.6ubuntu1
if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = upgrade ] && [ -d /run/systemd/system ] ; then
 deb-systemd-invoke stop 'google-shutdown-scripts.service' 'google-startup-scripts.service' >/dev/null || true
fi
# End automatically added section
```

Could this be as a result of `dh_installdeb/12.10ubuntu1` being used on focal vs `dh_installdeb/13.6ubuntu1` on jammy and the default changing between these versions?

description: updated
Revision history for this message
Bonka (bonkazx) wrote :

Hello @philroche, I am affected by this issue also. Is there an ETA when this fix will be deployed across please? Thanks.

Revision history for this message
Philip Roche (philroche) wrote (last edit ):

Following much testing we now have a proposed solution all options `--no-start --no-stop-on-upgrade --no-restart-after-upgrade` which passes our reproduction tests for all releases 23.10, 23.04, 22.10, 22.04, 20.04 and 18.04.

Ubuntu 23.10 has proposed package version `20230426.00-0ubuntu2` uploaded to the -proposed pocket of the archive. This will migrate to the -release pocket shortly.

We are still working on finding a solution for 16.04.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2

---------------
google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Tue, 16 May 2023 11:11:11 +0530

Changed in google-guest-agent (Ubuntu Mantic):
status: In Progress → Fix Released
Revision history for this message
Philip Roche (philroche) wrote :

test-google-guest-agent-upgrade-path.sh attached to help validate upgrade path. Running this with Ubuntu 23.10 mantic image daily-ubuntu-2310-mantic-amd64-v20230508 that the install of proposed google-guest-agent `20230426.00-0ubuntu2` does not trigger the bug of the shutdown script services being restarted.

Revision history for this message
Bonka (bonkazx) wrote :

Hello Team, please is there any ETA when this update could be available generally on latest GCP images running on 22.04.2 LTS (Jammy Jellyfish)?
My GCP instances are still running this version: 'google-guest-agent/jammy,now 20220622.00-0ubuntu2~22.04.4 amd64 [installed]'; even after running: 'sudo apt install google-guest-agent'. Thanks.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in google-guest-agent (Ubuntu Jammy):
status: New → Confirmed
Changed in google-guest-agent (Ubuntu Kinetic):
status: New → Confirmed
Changed in google-guest-agent (Ubuntu Lunar):
status: New → Confirmed
Revision history for this message
Andrew Cloke (andrew-cloke) wrote :

@bonkazx we're working to roll this fix out as quickly as possible, but the change will need to go through the SRU (Stable Release Update) process, so it is hard to communicate an ETA at this time.

Utkarsh Gupta (utkarsh)
description: updated
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Philip, or anyone else affected,

Accepted google-guest-agent into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~23.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in google-guest-agent (Ubuntu Lunar):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-lunar
Changed in google-guest-agent (Ubuntu Kinetic):
status: Confirmed → Fix Committed
tags: added: verification-needed-kinetic
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Philip, or anyone else affected,

Accepted google-guest-agent into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~22.10.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in google-guest-agent (Ubuntu Jammy):
status: Confirmed → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Philip, or anyone else affected,

Accepted google-guest-agent into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~22.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in google-guest-agent (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Philip, or anyone else affected,

Accepted google-guest-agent into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~20.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in google-guest-agent (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Philip, or anyone else affected,

Accepted google-guest-agent into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~18.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Andrew Cloke (andrew-cloke) wrote :

Testing
=======
The following testing has been done on the -proposed packages:

* The restart regression has been successfully tested
* Images using the -proposed packages have been built, and the CTF tests have passed on those images
* Those images have also been shared with Google and they have successfully validated them

Adjusting verification tags accordingly.

tags: added: verification-done verification-done-bionic verification-done-focal verification-done-jammy verification-done-kinetic verification-done-lunar
removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-jammy verification-needed-kinetic verification-needed-lunar
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~23.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~23.04.0) lunar; urgency=medium

  * No-change rebuild for Lunar.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:07:36 +0530

Changed in google-guest-agent (Ubuntu Lunar):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for google-guest-agent has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~22.10.0

---------------
google-guest-agent (20230426.00-0ubuntu2~22.10.0) kinetic; urgency=medium

  * No-change rebuild for Kinetic

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:11:33 +0530

Changed in google-guest-agent (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~22.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~22.04.0) jammy; urgency=medium

  * No-change rebuild for Jammy.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:13:37 +0530

Changed in google-guest-agent (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~20.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~20.04.0) focal; urgency=medium

  * No-change rebuild for Focal.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:16:13 +0530

Changed in google-guest-agent (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~18.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~18.04.0) bionic; urgency=medium

  * No-change rebuild for Bionic.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:22:23 +0530

Changed in google-guest-agent (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.