systemd unable to detect running apache if invoked via "apache2ctl graceful"

Output showing how to reproduce the issue:

root@server:/usr/local/sbin# apache2ctl stop
root@server:/usr/local/sbin# apache2ctl graceful
httpd not running, trying to start
root@server:/usr/local/sbin# systemctl status apache2.service
? apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           +-apache2-systemd.conf
   Active: inactive (dead) since Mon 2019-06-10 01:04:13 MDT; 17s ago
  Process: 27370 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 11462 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
 Main PID: 1849 (code=exited, status=0/SUCCESS)

Jun 08 19:11:24 server.[redacted2].com apachectl[1408]: AH00112: Warning: DocumentRoot [/srv/apache2/www/[redacted0]
Jun 08 19:11:24 server.[redacted2].com apachectl[1408]: AH00112: Warning: DocumentRoot [/srv/apache2/www/[redacted0]
Jun 08 19:11:24 server.[redacted2].com apachectl[1408]: AH00112: Warning: DocumentRoot [/srv/apache2/www/[redacted1].c
Jun 08 19:11:24 server.[redacted2].com apachectl[1408]: AH00112: Warning: DocumentRoot [/srv/apache2/www/[redacted1].c
Jun 08 19:11:24 server.[redacted2].com apachectl[1408]: AH00112: Warning: DocumentRoot [/srv/apache2/www/[redacted1]
Jun 08 19:11:24 server.[redacted2].com apachectl[1408]: AH00112: Warning: DocumentRoot [/srv/apache2/www/[redacted1]
Jun 08 19:11:24 server.[redacted2].com systemd[1]: Started The Apache HTTP Server.
Jun 09 06:25:04 server.[redacted2].com systemd[1]: Reloading The Apache HTTP Server.
Jun 09 06:25:04 server.[redacted2].com systemd[1]: Reloaded The Apache HTTP Server.
Jun 10 01:04:13 server.[redacted2].com apachectl[27370]: httpd (no pid file) not running

Output after customizing the apache2ctl script and restarting it:

root@server:/usr/local/sbin# apache2ctl_custom graceful
Invoking 'systemctl start apache2'.
Use 'systemctl status apache2' for more info.
root@server:/usr/local/sbin# systemctl status apache2.service
? apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           +-apache2-systemd.conf
   Active: active (running) since Mon 2019-06-10 01:04:50 MDT; 4s ago
  Process: 27370 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 11462 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
  Process: 27432 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 27444 (apache2)
    Tasks: 6 (limit: 2318)
   CGroup: /system.slice/apache2.service
           +-27444 /usr/sbin/apache2 -k start
           +-27448 /usr/sbin/apache2 -k start
           +-27449 /usr/sbin/apache2 -k start
           +-27451 /usr/sbin/apache2 -k start
           +-27454 /usr/sbin/apache2 -k start
           +-27455 /usr/sbin/apache2 -k start

Jun 10 01:04:50 server.[redacted2].com systemd[1]: Starting The Apache HTTP Server...
Jun 10 01:04:50 server.[redacted2].com systemd[1]: Started The Apache HTTP Server.

(if someone could redact the domain names and file paths from the above comment that would be much appreciated)

The attachment "diff of /usr/sbin/apache2ctl after correcting issue" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Thanks for the detailed report. It looks like it's the same issue as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=927302, so I'm linking them both together.

Acknowledging that the bug still exists and is in our queue, but not at high priority.

UPDATE: I just had a server experience downtime due to this bug. An update of the apache2 package was automatically triggered on a client's 16.04 LTS server on Aug 31 via the unattended-upgrades cron script, and apparently before this update, the last time the Apache service had been restarted was done via "apachectl graceful", without an appropriately modified /usr/sbin/apachectl as described in my initial report, so the usual unattended-upgrades stop-and-start which follows an Apache update failed to perform as expected, leaving the service in the stopped state.

Therefore, it appears there is now a definite case where a user invoking "apachectl graceful" leads to downtime. To mitigate this, you can either disable/uninstall unattended-upgrades, or patch the /usr/sbin/apachectl file as described in this bug report.

I have deleted the old patch file and replaced it with an actual correct one based on the apache2ctl file on my client's server after the latest apache2 package update, now that I know how diff patch files are generated

Bumping importance on this bug, since it is causing downtime on production systems. It is already on our server-next queue and known in debian BTS.

@faisal, please don't change status on bugs without an explanation.

Hi RedScourge,

the attached patch is a bit dirty, probably because it has been generated by diffing apache2ctl from different package versions, and I think it has a bug: you call

  systemctl start "$APACHE_SYSTEMD_SERVICE"

in the 'restart|graceful)' case stanza, while we want 'systemctl reload' there. I'd also move the need_systemd detection logic out of the "case" to avoid duplication. For the rest I think your proposed solution is essentially correct, thanks for providing it.

Hello RedScourge, or anyone else affected,

Accepted apache2 into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.46-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello RedScourge, or anyone else affected,

Accepted apache2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.41-4ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello RedScourge, or anyone else affected,

Accepted apache2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello RedScourge, or anyone else affected,

Accepted apache2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.18-2ubuntu3.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Cool, glad to hear that an issue I uncovered has been fixed...a little
confused as to why an official fix took 18 months to deploy though,
considering it took me about 18 minutes.

On Fri, Nov 27, 2020 at 8:42 AM Timo Aaltonen <email address hidden>
wrote:

> Hello RedScourge, or anyone else affected,
>
> Accepted apache2 into focal-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/apache2/2.4.41-4ubuntu3.2 in a few
> hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
> to enable and use -proposed. Your feedback will aid us getting this
> update out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, what testing has been
> performed on the package and change the tag from verification-needed-
> focal to verification-done-focal. If it does not fix the bug for you,
> please add a comment stating that, and change the tag to verification-
> failed-focal. In either case, without details of your testing we will
> not be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s)
> fixed by this package have been verified and the package has been in
> -proposed for a minimum of 7 days.
>
> ** Tags added: verification-needed-bionic
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1832182
>
> Title:
> systemd unable to detect running apache if invoked via "apache2ctl
> graceful"
>
> Status in apache2 package in Ubuntu:
> Fix Committed
> Status in apache2 source package in Xenial:
> Fix Committed
> Status in apache2 source package in Bionic:
> Fix Committed
> Status in apache2 source package in Focal:
> Fix Committed
> Status in apache2 source package in Groovy:
> Fix Committed
> Status in apache2 source package in Hirsute:
> Fix Committed
> Status in apache2 package in Debian:
> New
>
> Bug description:
> [Impact]
>
> Unattended upgrade will attempt to gracefully reload the Apache2 httpd
> service, but it reloads apache directly, not via systemd, so systemctl
> will mis-report the true status. This can cause unexpected and
> unnecessary server outages when e.g. upgrading from one LTS to
> another.
>
>
> [Test Case]
>
> root@server:/usr/local/sbin# apache2ctl stop
> root@server:/usr/local/sbin# apache2ctl graceful
> httpd not running, trying to start
> root@server:/usr/local/sbin# systemctl status apache2.service
> ? apache2.service - The Apache HTTP Server
> Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor
> preset: enabled)
> Drop-In: /lib/systemd/system/apache2.service.d
> +-apache2-systemd.conf
> Active: inactive (dead) since Mon 2019-06-10 01:04:13 MDT; 17s ago
> Process: 27370 ExecStop=/usr/sbin/apachectl s...

All autopkgtests for the newly accepted apache2 (2.4.18-2ubuntu3.18) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

passenger/5.0.27-2 (amd64, armhf, s390x, arm64, ppc64el, i386)
mod-wsgi/4.3.0-1.1ubuntu1 (amd64, armhf, s390x, arm64, ppc64el, i386)
apache2/2.4.18-2ubuntu3.18 (amd64, armhf, s390x, arm64, ppc64el, i386)
horizon/2:9.1.2-0ubuntu5 (amd64, armhf, s390x, arm64, ppc64el, i386)
libapache2-mod-python/3.3.1-11ubuntu2 (amd64, armhf, s390x, arm64, ppc64el, i386)
subversion/1.9.3-2ubuntu1.3 (amd64, armhf, s390x, arm64, ppc64el, i386)
php7.0/7.0.33-0ubuntu0.16.04.16 (amd64, armhf, s390x, arm64, ppc64el, i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted apache2 (2.4.29-1ubuntu4.15) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntuone-dev-tools/13.10-0ubuntu6 (amd64)
pinto/0.97+dfsg-4ubuntu1 (arm64)
cacti/1.1.38+ds1-1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted apache2 (2.4.46-1ubuntu1.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

nova/unknown (armhf)
lemonldap-ng/2.0.8+ds1-2 (arm64)
gvfs/1.46.1-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted apache2 (2.4.41-4ubuntu3.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

plinth/20.3 (s390x)
munin/2.0.56-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

apache2 2.4.18-2ubuntu3.18 fails to start on my ppc machine (4.4.0-197-powerpc-smp #229-Ubuntu):

Nov 28 14:42:31 macmini systemd[1]: Starting LSB: Apache2 web server...
Nov 28 14:42:31 macmini apache2[1284]: * Starting Apache httpd web server apache2
Nov 28 14:43:03 macmini apache2[1284]: Invoking 'systemctl start '.
Nov 28 14:43:03 macmini apache2[1284]: Use 'systemctl status ' for more info.
Nov 28 14:43:03 macmini apache2[1284]: Failed to mangle name: Invalid argument
Nov 28 14:43:03 macmini apache2[1284]: Failed to expand names: Invalid argument
Nov 28 14:43:03 macmini apache2[1284]: Action 'start' failed.
Nov 28 14:43:03 macmini apache2[1284]: The Apache error log may have more information.
Nov 28 14:43:03 macmini apache2[1284]: *
Nov 28 14:43:03 macmini apache2[1711]: * Stopping Apache httpd web server apache2
Nov 28 14:43:03 macmini apache2[1711]: *
Nov 28 14:43:03 macmini systemd[1]: Started LSB: Apache2 web server.

Downgrading to 2.4.18-2ubuntu3.17 works.

My ppc machine is running Xenial. I realize that ppc isn't supported, and Xenial is on its last legs, so if ppc is the only platform where this SRU fails then I can figure out a way to live with it. But I thought I'd report it anyway in case it's not only failing on ppc.

These are the binary packages from the apache2 source package I have installed:

apache2
apache2-bin
apache2-data
apache2-utils

This bug was fixed in the package apache2 - 2.4.46-1ubuntu2

---------------
apache2 (2.4.46-1ubuntu2) hirsute; urgency=medium

  * d/apache2ctl: Also use systemd for graceful if it is in use.
    (LP: #1832182)
    - This extends an earlier fix for the start command to behave
      similarly for restart / graceful. Fixes service failures on
      unattended upgrade.

 -- Bryce Harrington <email address hidden> Mon, 05 Oct 2020 16:06:32 -0700

Richard Hansen (rhansen)

I have same problem. Did you solve this?

systemctl status apache2
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: inactive (dead) since Вск 2020-11-29 17:43:15 EET; 18min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13871 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
  Process: 13852 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)

Ноя 29 17:43:15 ns2 apache2[13852]: Invoking 'systemctl start '.
Ноя 29 17:43:15 ns2 apache2[13852]: Use 'systemctl status ' for more info.
Ноя 29 17:43:15 ns2 apache2[13852]: Failed to mangle name: Invalid argument
Ноя 29 17:43:15 ns2 apache2[13852]: Failed to expand names: Invalid argument
Ноя 29 17:43:15 ns2 apache2[13852]: Action 'start' failed.
Ноя 29 17:43:15 ns2 apache2[13852]: The Apache error log may have more information.
Ноя 29 17:43:15 ns2 apache2[13852]: *
Ноя 29 17:43:15 ns2 apache2[13871]: * Stopping Apache httpd web server apache2
Ноя 29 17:43:15 ns2 apache2[13871]: *
Ноя 29 17:43:15 ns2 systemd[1]: Started LSB: Apache2 web server.

> I have same problem.

What architecture? (you can run "uname -m")
What version of Ubuntu? (you can run "lsb_release -a")
What version of apache2?

> Did you solve this?

Yes, by downgrading to 2.4.18-2ubuntu3.17.

do you have a how to for downgrading to 2.4.18-2ubuntu3.17?

I had the same problem.

# uname -m
x86_64

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial

# dpkg -l \*apache\* | grep ^i
ii apache2 2.4.18-2ubuntu3.18 amd64 Apache HTTP Server
ii apache2-bin 2.4.18-2ubuntu3.18 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.18-2ubuntu3.18 all Apache HTTP Server (common files)
ii apache2-dev 2.4.18-2ubuntu3.18 amd64 Apache HTTP Server (development headers)
ii apache2-utils 2.4.18-2ubuntu3.18 amd64 Apache HTTP Server (utility programs for web servers)

only solution for me was to downgrade back to 2.4.18-2ubuntu3.17 too

> do you have a how to for downgrading to 2.4.18-2ubuntu3.17?

# wget https://<your-ubuntu-mirror>/ubuntu/pool/main/a/apache2/apache2-bin_2.4.18-2ubuntu3.17_amd64.deb
# wget https://<your-ubuntu-mirror>/ubuntu/pool/main/a/apache2/apache2-data_2.4.18-2ubuntu3.17_all.deb
# wget https://<your-ubuntu-mirror>/ubuntu/pool/main/a/apache2/apache2-dev_2.4.18-2ubuntu3.17_amd64.deb
# wget https://<your-ubuntu-mirror>/ubuntu/pool/main/a/apache2/apache2-utils_2.4.18-2ubuntu3.17_amd64.deb
# wget https://<your-ubuntu-mirror>/ubuntu/pool/main/a/apache2/apache2_2.4.18-2ubuntu3.17_amd64.deb

or those packages what you have installed and after downloading:

# dpkg -i apache2*.deb

I got the same problem, apache2 doesn't start after the update to 2.4.18-2ubuntu3.18. I got the same output as in Comment 25. I noticed the problem in the browser when I couldn't get to my locally served web pages.

$ uname -m
x86_64

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial

I also had to downgrade to get apache2 running again, using:

aptitude install apache2-bin=2.4.18-2ubuntu3.17 apache2-data=2.4.18-2ubuntu3.17 apache2=2.4.18-2ubuntu3.17

Thanks for testing and reporting the regression for xenial. I'll investigate.

[Note that the update is currently only available if you have the xenial-proposed pocket enabled in your server. We really do appreciate people testing and reporting problems against -proposed, but definitely note that if this impacted a production server you may want to consider enabling only xenial-updates and not xenial-proposed, to help avoid surprises like this.]

And yes, the command Valentin posted in comment #30 is the correct workaround:

sudo apt-get install apache2-bin=2.4.18-2ubuntu3.17 apache2-data=2.4.18-2ubuntu3.17 apache2=2.4.18-2ubuntu3.17

It seems the problem is an undefined variable APACHE_SYSTEMD_SERVICE. I think that's supposed to be defined in /etc/init.d/apache2.

@Nolan ah you're probably correct.

Meanwhile, I've prepared a PPA package that simply reverts the change. If someone can please install and test that this removes the problem, I will go ahead and upload it to back out the problematic change so it doesn't hit other folks.

The PPA with the revert is here: https://launchpad.net/~bryce/+archive/ubuntu/apache2-sru-lp1832182

All autopkgtests for the newly accepted apache2 (2.4.18-2ubuntu3.18) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

libapache2-mod-python/3.3.1-11ubuntu2 (amd64, armhf, arm64, i386, ppc64el, s390x)
php7.0/7.0.33-0ubuntu0.16.04.16 (amd64, armhf, arm64, i386, ppc64el, s390x)
mod-wsgi/4.3.0-1.1ubuntu1 (amd64, armhf, arm64, i386, ppc64el, s390x)
subversion/1.9.3-2ubuntu1.3 (amd64, armhf, arm64, i386, ppc64el, s390x)
horizon/2:9.1.2-0ubuntu5 (amd64, armhf, arm64, i386, ppc64el, s390x)
apache2/2.4.18-2ubuntu3.18 (amd64, armhf, arm64, i386, ppc64el, s390x)
passenger/5.0.27-2 (amd64, armhf, arm64, i386, ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted apache2 (2.4.18-2ubuntu3.18) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

mod-wsgi/4.3.0-1.1ubuntu1 (arm64, s390x, amd64, i386, armhf, ppc64el)
php7.0/7.0.33-0ubuntu0.16.04.16 (arm64, s390x, amd64, i386, armhf, ppc64el)
passenger/5.0.27-2 (arm64, s390x, amd64, i386, armhf, ppc64el)
apache2/2.4.18-2ubuntu3.18 (arm64, s390x, amd64, i386, armhf, ppc64el)
subversion/1.9.3-2ubuntu1.3 (arm64, s390x, amd64, i386, armhf, ppc64el)
horizon/2:9.1.2-0ubuntu5 (arm64, s390x, amd64, i386, armhf, ppc64el)
libapache2-mod-python/3.3.1-11ubuntu2 (arm64, s390x, amd64, i386, armhf, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

2.4.18-2ubuntu3.18 has been removed from xenial-proposed.

Xenial has transitioned to ESM, so it is unlikely that this bug will be fixed. Hence, I am setting it as WONTFIX.

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release

It seems
https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.15
https://launchpad.net/ubuntu/+source/apache2/2.4.41-4ubuntu3.2

Were only in proposed and subsequent uploads - held back by the discussion on regressions back then.
Later subsequent security uploads did NOT include those changes:
https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.16
https://launchpad.net/ubuntu/+source/apache2/2.4.41-4ubuntu3.3

Therefore I have to reset these back from fix-committed to triaged.
@Bryce do you intent to revisit this for a new chance?

The regression discussed on this bug report pertained only to the xenial aspect of the fix, which was caused by omission of defining APACHE_SYSTEMD_SERVICE in Xenial's apache2ctl, as pointed out in comment #33. We opted to abandon the xenial update for this since xenial was near to end of standard support.

The bionic and focal updates weren't affected by that because their apache2ctl already had APACHE_SYSTEMD_SERVICE defined. It looks like their transition was initially blocked due to autopkgtest failures in other packages, however it looks like those were all resolved by simple retriggers; they may have just been intermittent test framework issues rather than actual test failures. It looks like they may have been blocked at that point waiting on verification.

I'll re-upload the bionic and focal fixes rebased on top of the security fix.

Hello RedScourge, or anyone else affected,

Accepted apache2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.41-4ubuntu3.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello RedScourge, or anyone else affected,

Accepted apache2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted apache2 (2.4.29-1ubuntu4.19) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.36.1-0ubuntu1.3.3 (amd64)
cacti/1.1.38+ds1-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted apache2 (2.4.41-4ubuntu3.7) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

apache2/2.4.41-4ubuntu3.7 (armhf)
libsoup2.4/2.70.0-1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

This worked beautifully, thanks Brian! Here's the verification logs for bionic and focal:

=> bionic
$ lxc launch images:ubuntu/18.04 ab
$ lxc shell ab
root@ab:~# apt install --no-install-recommends -V -y apache2
...
root@ab:~# systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
        /run/systemd/system/apache2.service.d
           └─zzz-lxc-service.conf
   Active: active (running) since Thu 2021-09-30 14:33:02 UTC; 15s ago
 Main PID: 1656 (apache2)
    Tasks: 54 (limit: 38371)
   CGroup: /system.slice/apache2.service
           ├─1656 /usr/sbin/apache2 -k start
           ├─1657 /usr/sbin/apache2 -k start
           └─1658 /usr/sbin/apache2 -k start

Sep 30 14:33:02 ab systemd[1]: Starting The Apache HTTP Server...
Sep 30 14:33:02 ab apachectl[1645]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Sep 30 14:33:02 ab systemd[1]: Started The Apache HTTP Server.
Sep 30 14:33:02 ab systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Sep 30 14:33:02 ab systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Sep 30 14:33:03 ab systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
Sep 30 14:33:03 ab systemd[1]: apache2.service: Failed to reset devices.list: Operation not permitted
root@ab:~# dpkg -l| grep apache2
ii apache2 2.4.29-1ubuntu4.18 amd64 Apache HTTP Server
ii apache2-bin 2.4.29-1ubuntu4.18 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.29-1ubuntu4.18 all Apache HTTP Server (common files)
ii apache2-utils 2.4.29-1ubuntu4.18 amd64 Apache HTTP Server (utility programs for web servers)

root@ab:~# apache2ctl stop
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
root@ab:~# apache2ctl graceful
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
httpd not running, trying to start
root@ab:~# systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
        /run/systemd/system/apache2.service.d
           └─zzz-lxc-service.conf
   Active: inactive (dead) since Thu 2021-09-30 14:33:51 UTC; 6s ago
  Process: 2565 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
 Main PID: 1656 (code=exited, status=0/SUCCESS)

Sep 30 14:33:02 ab systemd[1]: Starting The Apache HTTP Server...
Sep 30 14:33:02 ab apachectl[1645]: AH00558: apa...

Thanks Simon!

As previously, there have been some flaky autopkgtest failures. I have been retriggering them and they mostly started passing after one or two retries. There is one persistent failure that occurs in libsoup2.4's basic brotli test, however looking through libsoup2.4's autopkgtest history for focal on amd64, this same test case has randomly passed and failed off and on, and so AFICT is unrelated to this SRU's change.

I've filed LP: #1945713 about this problem. I guess I'll keep retriggering until it passes, unless someone else has an idea?

This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.19

---------------
apache2 (2.4.29-1ubuntu4.19) bionic; urgency=medium

  * d/apache2ctl: Also use systemd for graceful if it is in use.
    (LP: #1832182)
    - This extends an earlier fix for the start command to behave
      similarly for restart / graceful. Fixes service failures on
      unattended upgrade.

 -- Bryce Harrington <email address hidden> Tue, 28 Sep 2021 22:27:27 +0000

The verification of the Stable Release Update for apache2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package apache2 - 2.4.41-4ubuntu3.7

---------------
apache2 (2.4.41-4ubuntu3.7) focal; urgency=medium

  * d/apache2ctl: Also use systemd for graceful if it is in use.
    (LP: #1832182)
    - This extends an earlier fix for the start command to behave
      similarly for restart / graceful. Fixes service failures on
      unattended upgrade.

 -- Bryce Harrington <email address hidden> Tue, 28 Sep 2021 22:28:10 +0000

What? Why did you effectively remove both `apachectl graceful` and `systemctl reload apache2` and not leave any alternative?

We're using this pretty much every day to avoid downtime on our mod_security web filter, since it takes a while to do a full restart of Apache with 600+ vhosts:

vegar@webwash02:~$ sudo apache2ctl -S | grep -c namevhost
667

vegar@webwash02:/etc/apache2/sites-enabled$ sudo apachectl -t
Syntax OK
vegar@webwash02:/etc/apache2/sites-enabled$ sudo apachectl graceful
Invoking 'systemctl restart apache2'.
Use 'systemctl status apache2' for more info.

I definitely did not expect a change like this to just be implemented in an LTS version with no notice in security-announce.

So every single time we add a new vhost to our web filter, we'll have to do something like this instead of just running `sudo apachectl graceful`:

vegar@webwash02:~$ sudo apachectl -t
Syntax OK
vegar@webwash02:~$ systemctl status apache2 | grep "Main PID"
   Main PID: 97192 (apache2)
vegar@webwash02:~$ sudo kill -HUP 97192

This definitely makes things easier, thanks.

Thanks for raising this feedback Vergar, I'm glad you have a workaround identified, but we'll examine this further.

I am in the same boat as Vegar Nilsen. Not being able to do a graceful apache restart is a bummer.

Not an expert in systemd affairs, but if systemd is unable to track PIDs of services it started, doesn't that make it a bug in systemd (or the apache unit file)?

apache2 is run in cgroup /system.slice/apache2.service, and it should not be able to escape from it in any way. So how comes that systemd cannot find/manage these processes after a graceful restart?

> Not being able to do a graceful apache restart is a bummer.

That's putting it mildly, and it seems totally unnecessary because systemd can reload apache just fine, see below.

First a note on the effect of restarting Apache:

Everytime Apache restarts we kick out up to 100s of active client connections.

On We also have tens of JK workers on some servers, connected to tens of Tomcat instances. This slows down the restart considerably.

Secondly to the "unnecessary" part:

SystemD is not unable to reload Apache!

A simple
  systemctl reload apache2
does exactly what
  apache2ctl graceful
used to do for decades, and it is reflected in Apache's error.log accordingly:

[Wed Oct 13 13:23:23.248679 2021] [mpm_event:notice] [pid 20482:tid 140060367948864] AH00493: SIGUSR1 received. Doing graceful restart
Wed Oct 13 13:23:23.295354 2021] [mpm_event:notice] [pid 20482:tid 140060367948864] AH00489: Apache/2.4.41 (Ubuntu) mod_jk/1.2.46 OpenSSL/1.1.1f configured -- resuming normal operations

I also have verified that the PID of Apache's main process stays the same and only it's child processes are new.

All done on Ubuntu 20.04 which currently would do an apache restart for "graceful" by mistake.

So PLEASE, if you insinst on rerouting apache2ctl graceful to systemd make it reload and not restart Apache.

Thanks, regards, Christoph Lechleitner

Hello Christoph,

`systemctl reload apache2` is indeed the same of `apache2ctl graceful`:

ExecReload=/usr/sbin/apachectl graceful

(apachectl is a symlink to apache2ctl). This said, I see the issue you are reporting: there is effectively no easy way to do a graceful reload of apache2 now. I think one straightforward way to fix this consists in handling the "graceful" and "restart" commands in separate "case" stanzas in apache2ctl.

However, at least for the devel release, I think the best way forward would be to expand the service file to cover what apache2ctl does, and make apache2ctl a simple compatibility wrapper around systemctl.

I partially take it back: there's an easy way to do a graceful reload and that's by `systemctl reload apache2` (as Christoph mentioned, but I misinterpreted the package version he was referring to). I can confirm the error.log shows:

[pid 3014:tid 139989240114240] AH00493: SIGUSR1 received. Doing graceful restart

What doesn't work is `apache2ctl graceful`, which unconditionally calls `systemctl restart`. Excerpt from `apache2ctl`:

restart|graceful)
    if $HTTPD ${APACHE_ARGUMENTS} -t 2> /dev/null ; then
        if need_systemd; then
            # If running on systemd we should not directly restart httpd since
            # systemd would be confused about httpd's status.
            # (See LP: #1832182)
            echo "Invoking 'systemctl restart ${APACHE_SYSTEMD_SERVICE}'."
            echo "Use 'systemctl status ${APACHE_SYSTEMD_SERVICE}' for more info."
            systemctl restart "${APACHE_SYSTEMD_SERVICE}"

So we have a regression when `apache2ctl graceful` is called directly, which I think it's still a common thing, even if systemctl is to be preferred.

First, sorry if I'm in the slightly wrong issue here. The most recent comment before mine fits our situation perfectly and wasn't old, so I think it's an acceptable place for this discussion.

> So we have a regression when `apache2ctl graceful` is called directly, which I think it's still a common thing

I agreed.

'common thing' is more than true: apachectl resp. apache2ctl were here before init systems started to support 'reload', and propably before Debian's 'service' wrapper around several supported init systems.

Hence
1. long-term admins are almost hard-wired to call apache2ctl directly
2. many many scripts do so
3. many many how-to guides, readmes, ..., do so too

IMO the 'restart' and 'graceful' branch in apache2ctl should be split up.

But I haven't investiagtes why graceful -> restart might make sense in the first place. At all.

Sounds like the maintainers have Mozilla'd everything up (not fix an important bug for years then finally go with a potentially frustrating solution) for some of you here. You may want to use the Apache2 PPA repo of the Debian/Ubuntu PHP maintainer Ondřej Surý instead of using this hot mess, like I have for the past few years, since I find that I need it for PHP anyway. You still may have to alter your init script as I did in my patch above. The PPA homepage and usage instructions can be found at: https://launchpad.net/~ondrej/+archive/ubuntu/apache2/

another hack around this mess:

  apache2ctl -k graceful

(note the "-k")

this leads to the script not matching on "graceful", going to the "*" case instead and passing the arguments to /usr/sbin/apache2, bypassing any systemctl invocation.

Actually this is what I usually type, but not all of our scripts/crontabs/fellow admins use it.
So my mailbox is still flooded with these messages:

  Invoking 'systemctl restart apache2'.
  Use 'systemctl status apache2' for more info.

A revert of this update for focal has been accepted to ubuntu-proposed today. This should get released to focal-updates shortly.

This bug was fixed in the package apache2 - 2.4.41-4ubuntu3.9

---------------
apache2 (2.4.41-4ubuntu3.9) focal-security; urgency=medium

  * SECURITY UPDATE: DoS or SSRF via forward proxy
    - debian/patches/CVE-2021-44224-1.patch: enforce that fully qualified
      uri-paths not to be forward-proxied have an http(s) scheme, and that
      the ones to be forward proxied have a hostname in
      include/http_protocol.h, modules/http/http_request.c,
      modules/http2/h2_request.c, modules/proxy/mod_proxy.c,
      modules/proxy/proxy_util.c, server/protocol.c.
    - debian/patches/CVE-2021-44224-2.patch: don't prevent forwarding URIs
      w/ no hostname in modules/proxy/mod_proxy.c,
      modules/proxy/proxy_util.c.
    - CVE-2021-44224
  * SECURITY UPDATE: overflow in mod_lua multipart parser
    - debian/patches/CVE-2021-44790.patch: improve error handling in
      modules/lua/lua_request.c.
    - CVE-2021-44790

 -- Marc Deslauriers <email address hidden> Wed, 05 Jan 2022 09:49:56 -0500

This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.21

---------------
apache2 (2.4.29-1ubuntu4.21) bionic-security; urgency=medium

  * SECURITY UPDATE: DoS or SSRF via forward proxy
    - debian/patches/CVE-2021-44224-1.patch: enforce that fully qualified
      uri-paths not to be forward-proxied have an http(s) scheme, and that
      the ones to be forward proxied have a hostname in
      include/http_protocol.h, modules/http/http_request.c,
      modules/http2/h2_request.c, modules/proxy/mod_proxy.c,
      modules/proxy/proxy_util.c, server/protocol.c.
    - debian/patches/CVE-2021-44224-2.patch: don't prevent forwarding URIs
      w/ no hostname in modules/proxy/mod_proxy.c,
      modules/proxy/proxy_util.c.
    - CVE-2021-44224
  * SECURITY UPDATE: overflow in mod_lua multipart parser
    - debian/patches/CVE-2021-44790.patch: improve error handling in
      modules/lua/lua_request.c.
    - CVE-2021-44790

 -- Marc Deslauriers <email address hidden> Wed, 05 Jan 2022 09:50:41 -0500

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.