Anacron service and timer disabled after upgrading from 2.3-33

Status changed to 'Confirmed' because the bug affects multiple users.

I'm attaching a patch from 2.3.26ubuntu1 (kinetic's) to 2.3-36ubuntu2(~ppa2) which uses a heuristic to re-enable the service and/or the timer if they were previously enabled. There are comments inside the postinst/preinst scripts.

A typical test I used on my machine looked like the following:

  sudo dpkg -i ../lp/anacron_2.3-33ubuntu1_amd64.deb
  sudo systemctl enable anacron.timer
  sudo systemctl disable anacron.service
  sudo dpkg -i ../anacron_2.3-36ubuntu2\~ppa2_amd64.deb

The following displays the enabled/disabled status:

  systemctl show --property=UnitFileState anacron.timer
  systemctl show --property=UnitFileState anacron.service

As I mention in my changes, systemctl show/status have side-effects and I have no idea why.

The attachment "anacron_2.3-36ubuntu1-to-2.3-36ubuntu2~ppa2.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Target should be lunar instead of kinetic.

If we end up making a new version for kinetic, we would have to include these change

Ugh what a mess. Well done on fighting to a solution! I have some peanut gallery level observations:

1) can you explain in a bit more detail what happens without the systemctl show calls in preinst? "goes wrong" is a bit ambiguous

2) Did you consider recording the results of systemctl show --property=UnitFileState anacron.{timer,service} in preinst (guarded by checks on $1 and $2) into /run or something and then acting on that in postinst? It feels like it might be a bit less delicate than the solution you have here. But maybe I'm missing something.

1) Without the "systemctl show" in preinst, the "systemctl show" in postinst is wrong. Yes, that makes no sense.
Basically, the steps would be:
- systemctl enable anacron.service
- dpkg -i anacron 2.3-33
- (magic step)
- dpkg -i anacron 2.3-36 + my changes
  - in postinst, systemctl show anacron.service returns disabled unless "systemctl show anacron.service" is called during (magic step), which could also be in the new anacron's postinst.

TBH, I'm thinking "select isn't broken" but I also feel like I've tried everything and this is the only change. I might also describe this improperly. Maybe there's some cache effect, especially since the whole issue here is related to changing configuration behind systemd's back.

2) I didn't consider doing that, mostly because I changed postinst late. Basically I was debugging and somehow noticed the pattern that results in postinst were right when I was debugging and only when I was debugging. Probing only from preinst could be cleaner and it could also make more sense overall since there would be less opportunities for changes to have happened to the system. I need to look at that and review the exact order in which the package scripts are being run.

I'll see what I can do following your idea in 2). Thanks.

+ if [ -n "${units_to_enable}" ]; then
+ echo "Some systemd units were probably just disabled mistakenly by anacron 2.3-33; re-enabling ${units_to_enable}"
+ for unit in ${units_to_enable}; do
+ systemctl enable "${unit}" || true
+ done
+ fi
+fi

Is there any deb-systemd-helper state to worry about here in addition to the systemd state?

WRT recording results in preinst and acting on them in postinst, if this is necessary, please use /var/lib/ not /run. It is possible under undesirable circumstances that a reboot happens between the preinst and the postinst being executed.

I did review the order the maintainer scripts run before making my preinst / postinst suggestion but of course checking that carefully again is a good idea here! Steve makes a good point about them needing to persist.

I'm attaching another patch.

It backups the symlinks in preinst and restores them in postinst. Overall it feels cleaner than my previous changes and the symlinks should be unaltered. I output warnings (although on stdout now that I think of it) from both preinst and postinst in order to warn about what will be done and to warn about what has been done, along with the symlink targets for debug (and now that I think of it, I shouldn't assume they are symlinks).

The changelog isn't cleaned up; I'll do that once the review is (mostly) cleared.

I had an issue with my mail sync and didn't notice your comments here.

The issue with persisting changes is that there is a window of time during which we could end up overwriting user changes. The more time between preinst and postinst, the more likely we are to un-do something done by the user. It might be possible to protect more against that by re-using my logic from the first iteration of the patch and check whether the on-disk and in-memory states differ. There would still be a window of time during which we would be blind (namely a reboot and/or another systemctl enable/disable anacron.{timer,service}) but it would be smaller.

That being said, I don't think we can have a perfect solution without overwriting 2.3-33's postrm file on disk (I've been tempted to do that more than once) and there is almost certainly a decision to make between possibly being blind to user changes, and not restoring the state in all situations.

About deb-systemd-helper, I don't think this is an issue with it. I need to audit this again in order to be sure.

It's a bit difficult to understand deb-systemd-helper fully well but this doesn't matter much.

Luckily, we don't need to look at the whole code. No script is called between the new preinst, the old postrm and the new postinst. As such, the only code we have to look at is what is inserted through the #DEBHELPER placeholder. The only cases during which deb-systemd-helper can be called are on "purge" and "remove" events which are not a concern here.

As for the configuration step being interrupted, I think I prefer to keep this simpler and not try to take into account voluntary changes between preinst and postinst, even if the two are separate by a month. Having anacron installed but disabled makes fairly little sense and disabling it in the middle of an upgrades makes even less sense. What do you think about this?

I think your change looks good.

On the topic of where the units/symlinks are persisted, I don't really understand quite what you are trying to say (possibly my problem).

I guess the situations where there is an unexpected delay between new-preinst and new-postinst are when the system crashes or a maintainer script in a different package fails (I think?). The risk here is that an explicit systemctl disable anacron.service before "apt install -f" or whatever gets run will be ignored. I think given the general edge-case-y-ness of all this I'm OK with this?

The issue of persisting to /run or /var/lib is that the behaviour depends on if the system is rebooted between preinst and postinst. This frankly seems a bit surprising -- so I'd prefer you to change to persisting in /var/lib.

Thanks. I'll post an updated patch on tuesday or wednesday.

What I meant for with the persistence change is that it lengthen the time during which extra things can happen but we'll still be trying to restore the configuration. Obviously tt's not the only way configuration changes happen but in my mind it changes that timeframe from "a couple seconds at most" to "maybe hours or days".

Anyway, I also think it's an edge-case and you have answered my concern. Thanks. :)

It's Thursday, here's my latest patch. The debian/changelog is not clean but I've edited it so that it's only a matter of dropping all the ~ppa versions.

Attached is patch from 2.3-36ubuntu1 to 2.3-36-ubuntu2~ppa5: anacron_2.3-36ubuntu1.dsc-to-anacron_2.3-36ubuntu2~ppa5.dsc.diff

And here is the diff from the previous patch that I posted, from 2.3-36ubuntu2~ppa4 to 2.3-36ubuntu2~ppa5: anacron_2.3-36ubuntu2~ppa4.dsc-to-anacron_2.3-36ubuntu2~ppa5.dsc.diff .

I think this looks pretty good. Only one quibble (apart from the indentation) is that you have a message

    echo "Warning: found a backup for ${name} but original file still exist; please check its status and configuration; backup is available at \`${backup}'"

but it looks to me as if the backup file will be deleted shortly after this. I don't know how much it is worth complicating things to handle this edge case on an edge case but this contradiction should be resolved somehow I think (maybe just don't do anything silently if the original file is still there?)

I'm attaching an updated debdiff. I can't (easily) produce an debdiff between patch revisions unfortunately.

Changes:
- fixed the indentation
- improved messages a bit
- made it so that the backup directory is not removed anymore in case a file exists at the original location of the backup: replace "rm -rf $dir" with "rmdir --ignore-fail-on-non-empty" in order to not fail configuration but maybe we prefer it to fail.

For reference, the (updated) message in case a file exists at the original location of the backup:
> Warning: found a backup for anacron.timer but a file exists at the original location; please check its status and configuration; backup is available at `/var/lib/dpkg/anacron-service-timer-status/anacron-timer-backup'

The changelog still has all the clutter from the several revisions here but it should be simple to remove all the ppa entries before applying.

OK, this version looks good and I've just uploaded it:

Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading anacron_2.3-36ubuntu2.dsc: done.
  Uploading anacron_2.3-36ubuntu2.debian.tar.xz: done.
  Uploading anacron_2.3-36ubuntu2_source.buildinfo: done.
  Uploading anacron_2.3-36ubuntu2_source.changes: done.
Successfully uploaded packages.

Thanks for the perseverance!

This bug was fixed in the package anacron - 2.3-36ubuntu2

---------------
anacron (2.3-36ubuntu2) lunar; urgency=medium

  * Work around upgrades from 2.3-33 disabling service and timer (LP: #2006589)

 -- Adrien Nader <email address hidden> Wed, 08 Feb 2023 09:53:26 +0100

[Removed wrong patch]

Here's the patch for the SRU

+anacron (2.3-33ubuntu2) kinetic; urgency=low
+
+ If you have installed Ubuntu Kinetic Kudu and then upgraded to Lunar
+ Lobster during its development cycle before mid March 2023, then anacron
+ might no longer be enabled and the daily/weekly/monthly cron jobs might
+ not be run until it is.
+
[...]

This doesn't seem correct to include in the SRU. (I don't know that anyone will ever *see* this NEWS file, but let's not include a larger than necessary delta that contains irrelevant information, shall we?)

The SRU also includes no changes to the postrm. Isn't this the part in 33ubuntu1 where the bug lies? (Seen in delta between the proposed SRU and current lunar)

-# Close bug #993348, remove dangling symlinks if they exist
-if [ -f /etc/systemd/system/multi-user.target.wants/anacron.service ]; then
- rm /etc/systemd/system/multi-user.target.wants/anacron.service
-fi
-
-if [ -f /etc/systemd/system/timers.target.wants/anacron.timer ]; then
- rm /etc/systemd/system/timers.target.wants/anacron.timer
-fi
-

Thanks a lot for the review.

I wasn't sure about the NEWS entry. I tweaked it compared to the one in Debian. In hindsight, I think I had two releases in mind at the same time and the result doesn't make much sense indeed. There's indeed no version path that would lead to what I described there. I'll remove the NEWS file.

About the postrm, I'm not sure I understand what you mean. The diff you've posted is included in my patch. I have posted two patches, the older of the two being wrong/obsolete and I've removed it now. Did you look at the one posted on 2023-03-22 which is the most recent?

> About the postrm, I'm not sure I understand what you mean. The diff
> you've posted is included in my patch. I have posted two patches, the
> older of the two being wrong/obsolete and I've removed it now. Did you
> look at the one posted on 2023-03-22 which is the most recent?

Apparently I did have the wrong one, sorry!

I'll re-review tomorrow after you've had a chance to remove the NEWS file.

Hello Adrien, or anyone else affected,

Accepted anacron into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/anacron/2.3-33ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I assume everything that was needed has been sponsored here. Removing ubuntu-sponsors.

Sorry, I hadn't yet done the verification step after proposed. Here is the log for a kinetic LXD which installs anacron and then updates it to the version in -proposed ((2.3-33ubuntu2) over (2.3-33ubuntu1)).

As can be seen, the service and timer are enabled both before and after the upgrade. We can also see that the work-around I added is triggered and points out that the service and timer have been disabled until the work-around kicks in.

======

RUN lxc launch ubuntu:kinetic anacron-sru
Creating anacron-sru
Starting anacron-sru
RUN@anacron-sru sh -c apt-get -qq update >/dev/null
RUN@anacron-sru apt-get -y -qq -o=Dpkg::Use-Pty=0 upgrade --autoremove
(Reading database ... 34189 files and directories currently installed.)
Removing libfreetype6:amd64 (2.12.1+dfsg-3ubuntu0.1) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
RUN@anacron-sru apt-get -y install anacron
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  anacron
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 25.9 kB of archives.
After this operation, 100 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu kinetic/main amd64 anacron amd64 2.3-33ubuntu1 [25.9 kB]
Fetched 25.9 kB in 0s (274 kB/s)
Selecting previously unselected package anacron.
(Reading database ... 34183 files and directories currently installed.)
Preparing to unpack .../anacron_2.3-33ubuntu1_amd64.deb ...
Unpacking anacron (2.3-33ubuntu1) ...
Setting up anacron (2.3-33ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/anacron.service → /lib/systemd/system/anacron.service.
Created symlink /etc/systemd/system/timers.target.wants/anacron.timer → /lib/systemd/system/anacron.timer.
Processing triggers for man-db (2.10.2-2) ...
Scanning processes...

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
RUN@anacron-sru sh -c echo deb http://fr.archive.ubuntu.com/ubuntu kinetic-proposed main restricted >> /etc/apt/sources.list
RUN@anacron-sru sh -c apt-get -qq update >/dev/null
RUN@anacron-sru apt-get install anacron
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  default-mta | mail-transport-agent
The following packages will be upgraded:
  anacron
1 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Need to get 26.7 kB of archives.
After this operation, 4096 B of additional disk space will be used.
Get:1 http://fr.archive.ubuntu.com/ubuntu kinetic-proposed/main amd64 anacron amd64 2.3-33ubuntu2 [26.7 kB]
Fetched 26.7 kB in 0s (292 kB/s)
(Reading database ... 34202 files and directories currently installed.)
Preparing to unpack .../anacron_2.3-33ubuntu2_amd64.deb ...
Warning: anacron.service will be disabled during the upgrade from anacron 2.3-33ubuntu1; we'll do our best to re-enable it (currently points to `/lib/systemd/system/anacron.service')
Warning: anacron.timer...