package squid 2.7.STABLE7-1ubuntu12.1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1

Hello Greg (and duplicate reporters), thanks for taking the time to file this bug report!

So unfortunately, this is likely the result of the squid process having disappeared because of bug #573853 , which ironically was fixed in the package that you are all trying to install.

This is a temporary problem, and the way to alleviate it is to kill squid off manually (it may still be running despite upstart having lost track of it), and then start it manually.

These three commands may print errors (and may leave your system in a bad state, so please be cautious), but should lead to the upgrade finishing:

sudo stop squid
sudo killall squid
sudo start squid
sudo dpkg --configure -a

Unfortunately, given the nature of the bug fixed there's no safe way that I have come up with to plan for this possible failure and get around it in the postinst. Given that, I believe this bug has to be closed as Won't Fix.

If someone comes up with a better way to handle the upgrade when the squid job is in an unknown state, then please re-open it.

Ok well after saying that, and talking a bit more with Chuck Short, there is a safe way to handle it, so I'm re-opening it as Triaged and assigning to myself.

The branch lp:~clint-fewbar/ubuntu/lucid/fix-upgrades-after-reload fixes the problem by checking to see if the problem the recent update is trying to correct has actually happened, and if so, doing a stop/start on the service using invoke-rc.d Between the stop/start, it also kills the errant pid (but only if invoke-rc.d worked, so policy is still respected).

I test this on a lucid vm in the following scenarios:

* bare bones no updates with just 'apt-get install squid' version 2.7.STABLE7-1ubuntu12
 - When preinst is run, the upstart pid and the pidfile agree, and so it exits doing nothing.

* same, but also running 'sudo reload squid', verifying that squid is not tracked by upstart anymore with 'status squid' showing stop/waiting, but 'ps auxw | grep squid' showing a running squid.
 - the job is stopped, the old squid is killed, squid is started again

* start squid, reload, kill -9, leaving stale pid file.
 - Code stops because /proc/`cat pidfile` does not exist.

* modify /etc/init/squid.conf to not use expect fork, and pass -N (basically manually putting in the new job file). This simulates the chance that a user may have applied the fix manually.
 - Code stops because the upstart pid matches the pidfile.

* stopped service before hand
 - pidfile is gone, code is not run

* removed upstart file
 - code checks for that, does not run.

Since creating the merge proposal I also tried testing it with a /usr/sbin/policy-rc.d that exitted with 102, the appropriate way to disable a service's stop/starting, and the code was skipped. When the policy-rc.d returned 0, the code worked as intended (invoke-rc.d's behavior was basically unchanged).

As I finish the SRU report, I realize that we actually need to address this in natty too, and may need to drop the version comparison to make it less complex and more likely to fix the problem given scenarios I haven't thought of.

Keeping in queue until bug 573853 is verified and the previous squid in -proposed goes to -updates.

This bug was fixed in the package squid - 2.7.STABLE9-2.1ubuntu3

---------------
squid (2.7.STABLE9-2.1ubuntu3) natty; urgency=low

  * Detect and deal with the situation where upstart has lost track of
    squid. (LP: #717397)
 -- Clint Byrum <email address hidden> Thu, 24 Feb 2011 10:54:40 -0500

Accepted squid into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

It doesn't seem that this package handles the state where squid is in the stop/waiting state prior to upgrade. The previous restart style stanzas are encapsulated in checking current status if statements.

Setting up squid (2.7.STABLE7-1ubuntu12.2) ...
restart: Job failed to restart
dpkg: error processing squid (--configure):
 subprocess installed post-installation script returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up squid (2.7.STABLE7-1ubuntu12.2) ...

Aptitude did successfully recover from this situation, currently unsure how apt-get would. Although this has an edge case, it's a better situation than leaving bug 725635 out in the wild.

Well, that's because the postinst isn't using the init script policy layer.

if status squid | grep "start/running" > /dev/null; then
        restart squid
fi

if [ "$FIXLINES" = "false" ]; then
  echo "squid.conf contains 2.2.5 syntax - cache_dir directive - not starting "
  echo "Run 'dpkg-reconfigure -plow squid' to fix this automatically,"
  echo "or fix the 'cache_dir'-entry in your squid.conf manually."
  echo "See documentation in /usr/share/doc/squid for nearer instructions."
else
  if status squid | grep "start/running" > /dev/null; then
        restart squid
  fi
fi

Why isn't this "invoke-rc.d squid restart" like it should be? Also, the fixlines check is pointless given that it's preceded by a squid restart command. This all seems to have been a wrong fix for bug #552360 and/or bug #509307.

This ought to be fixed up in SRU, probably at the same time.

On Sun, 2011-02-27 at 20:47 +0000, Steve Langasek wrote:
> Well, that's because the postinst isn't using the init script policy
> layer.
>
> if status squid | grep "start/running" > /dev/null; then
> restart squid
> fi
>
> if [ "$FIXLINES" = "false" ]; then
> echo "squid.conf contains 2.2.5 syntax - cache_dir directive - not starting "
> echo "Run 'dpkg-reconfigure -plow squid' to fix this automatically,"
> echo "or fix the 'cache_dir'-entry in your squid.conf manually."
> echo "See documentation in /usr/share/doc/squid for nearer instructions."
> else
> if status squid | grep "start/running" > /dev/null; then
> restart squid
> fi
> fi
>
> Why isn't this "invoke-rc.d squid restart" like it should be? Also, the
> fixlines check is pointless given that it's preceded by a squid restart
> command. This all seems to have been a wrong fix for bug #552360 and/or
> bug #509307.

This is particularly troubling because the 'restart' command does not
re-load the upstart job file, while invoke-rc.d's restart method, IIRC,
runs stop/start for this very reason.

That said, currently if the process is stop/waiting, its left in that
very state on upgrade. So while this is a bug, and should be fixed on
the next upload, I don't believe it is the cause of the issue.

The things that can cause a "failed to restart" will also cause a
'failed to stop' or 'failed to start'. Namely:

pre-start exits non zero
post-stop exits non zero
main process cannot be executed

There are a bunch of things in the pre-start in particular that might
fail. In this case, re-running the preinst would result in the postinst
succeeding, but squid not running.

I do think this is related, but not the same, so I've opened bug #726348
to address that issue separately.

This bug was fixed in the package squid - 2.7.STABLE7-1ubuntu12.2

---------------
squid (2.7.STABLE7-1ubuntu12.2) lucid-proposed; urgency=low

  * Detect and deal with the situation where upstart has lost track of
    squid. (LP: #717397)
 -- Clint Byrum <email address hidden> Mon, 14 Feb 2011 14:39:47 -0800

fwts (0.22.06) natty; urgency=low

  * acpi: cstates: fix progress calculation, tidy up code.

 -- Colin King <email address hidden> Fri, 4 Mar 2011 10:37:00 +0000

fwts (0.22.05) natty; urgency=low

  * Fix cstates test (LP: #728685)
  * acpi: acpidump: use more sensible heading, fix SCI text
  * hpet: Fail test and skip sanity check when base address not found
  * bios: ebda_region: close fd on error

 -- Colin King <email address hidden> Thu, 3 Mar 2011 20:42:02 +0000

fwts (0.22.04) natty; urgency=low

  * acpi: acpidump: use more sensible heading, fix SCI text
  * hpet: Fail test and skip sanity check when base address not
  * bios: ebda_region: close fd on error.

 -- Colin King <email address hidden> Thu, 3 Mar 2011 09:37:39 +0000

Oops, wrong bug :/

This fix appears to have broken the upgrade process.

On two systems, it ended up failing to finish the upgrade. After killing the upgrade processes I was unable to stop the still running squid using 'service squid stop' or 'stop squid' or '/etc/init.d/squid stop' or anything else.

After that I couldn't do anything with squid as it was in a 'stop/killed' (with a PID that doesn't exist). See bug: https://bugs.launchpad.net/upstart/+bug/406397

Can I suggest an upgrade to this crazy new init system I've heard of called 'System V'. It sounds badass. And reliable/predictable.

Also, any hints as to starting the daemon without rebooting would be appreciated.

Oh. This was on 10.04 (LTS).