IncludeOptional fails when a directory does not exist

Bug #1766186 reported by Dmitrii Shcherbakov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apache2 (Ubuntu)
Fix Released
Low
Andreas Hasenack
Xenial
Fix Released
Low
Andreas Hasenack
Artful
Fix Released
Low
Andreas Hasenack
Bionic
Fix Released
Low
Andreas Hasenack

Bug Description

[Impact]

An instruction in apache config such as the following will make apache2 fail to start if the target path does not exist:

IncludeOptional /etc/apache2/mellon/sp-location*.conf

https://paste.ubuntu.com/p/f2Tb9rgrBB/ (full file: /etc/apache2/sites-enabled/openstack_https_frontend.conf)

Apr 23 06:58:31 maas-xenial1 apache2[28269]: * The apache2 configtest failed.
Apr 23 06:58:31 maas-xenial1 apache2[28269]: Output of config test was:
Apr 23 06:58:31 maas-xenial1 apache2[28269]: apache2: Syntax error on line 219 of /etc/apache2/apache2.conf: Syntax error on line 17 of /etc/apache2/sites-enabled/openstack_https_frontend.conf: Could not o
Apr 23 06:58:31 maas-xenial1 apache2[28269]: Action 'configtest' failed.
Apr 23 06:58:31 maas-xenial1 apache2[28269]: The Apache error log may have more information.
Apr 23 06:58:31 maas-xenial1 systemd[1]: apache2.service: Control process exited, code=exited status=1
Apr 23 06:58:31 maas-xenial1 systemd[1]: Failed to start LSB: Apache2 web server.

A fix is available upstream:

https://bz.apache.org/bugzilla/show_bug.cgi?id=57585
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878920

https://github.com/apache/httpd/commit/a17ce7dd5e6277867ca48659f70c4bb8a11add56#diff-18a4bdb1520687ad43a0b4dd2b51d957R1954

[Test Case]

 * add IncludeOptional /etc/apache2/mellon/sp-location*.conf to an apache config file and try to restart the apache2 service

[Regression Potential]

 * minimal, the patch eases up requirements without rewriting or removing the existing code;
 * the patch author says that it's been used for ~ 3 years in clearlinux https://bz.apache.org/bugzilla/show_bug.cgi?id=57585#c4
 * applied in debian https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878920#22

Related branches

description: updated
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Tried 2.4.33 from Debian sid with the fix included and a dummy path ("/etc/apache2/whatever" dir does not exist):

IncludeOptional /etc/apache2/whatever/*

stat /etc/apache2/whatever
stat: cannot stat '/etc/apache2/whatever': No such file or directory

apt policy apache2
apache2:
  Installed: 2.4.33-2
  Candidate: 2.4.33-2
  Version table:
 *** 2.4.33-2 500
        500 http://ftp.de.debian.org/debian sid/main amd64 Packages
        100 /var/lib/dpkg/status

It works properly as opposed to the version without the fix which fails with:

Apr 25 22:19:27 maas-xenial1 apache2[18520]: apache2: Syntax error on line 219 of /etc/apache2/apache2.conf: Syntax error on line 16 of /etc/apache2/sites-enabled/openstack_https_frontend.conf: Could not o

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

At the same time including a wildcard at the end of the non-existent directory name with 2.4.18-2ubuntu3.8 makes apache2 happy:

IncludeOptional /etc/apache2/whatever*/*

apt policy apache2
apache2:
  Installed: 2.4.18-2ubuntu3.8
  Candidate: 2.4.18-2ubuntu3.8

Changed in apache2 (Ubuntu):
assignee: nobody → Andreas Hasenack (ahasenack)
status: New → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

PPA with test packages: ppa:ahasenack/apache-includeoptional-1766186 (https://launchpad.net/~ahasenack/+archive/ubuntu/apache-includeoptional-1766186/)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Packages uploaded to proposed unapproved.

Changed in apache2 (Ubuntu):
status: In Progress → Fix Released
Changed in apache2 (Ubuntu Xenial):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in apache2 (Ubuntu Artful):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in apache2 (Ubuntu Bionic):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in apache2 (Ubuntu Xenial):
status: New → In Progress
Changed in apache2 (Ubuntu Artful):
status: New → In Progress
Changed in apache2 (Ubuntu Bionic):
status: New → In Progress
Changed in apache2 (Ubuntu Xenial):
importance: Undecided → Low
Changed in apache2 (Ubuntu Artful):
importance: Undecided → Low
Changed in apache2 (Ubuntu Bionic):
importance: Undecided → Low
Changed in apache2 (Ubuntu):
importance: Undecided → Low
Changed in apache2 (Ubuntu Xenial):
milestone: none → ubuntu-16.04.5
Revision history for this message
Robie Basak (racb) wrote :

I suggest that the negative case also be checked during SRU verification: please also verify that if a file is included by IncludeOptional with this syntax that it is actually being included rather than being silently ignored.

Changed in apache2 (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Dmitrii, or anyone else affected,

Accepted apache2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.29-1ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apache2 (Ubuntu Artful):
status: In Progress → Fix Committed
tags: added: verification-needed-artful
Revision history for this message
Robie Basak (racb) wrote :

Hello Dmitrii, or anyone else affected,

Accepted apache2 into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.27-2ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apache2 (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Robie Basak (racb) wrote :

Hello Dmitrii, or anyone else affected,

Accepted apache2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.18-2ubuntu3.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Checked with xenial-proposed:

root@maas-vhost2:/etc/apache2# apt policy apache2
apache2:
  Installed: 2.4.18-2ubuntu3.9
  Candidate: 2.4.18-2ubuntu3.9
  Version table:
 *** 2.4.18-2ubuntu3.9 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.18-2ubuntu3.8 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     2.4.18-2ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

grep IncludeOptional sites-enabled/openstack_https_frontend.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf

stat mellon/sp-location.keystone-saml-mellon.conf
  File: 'mellon/sp-location.keystone-saml-mellon.conf'
# ...

systemctl restart apache2 && sleep 10 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:17:04 UTC; 10s ago

mv mellon/sp-location.keystone-saml-mellon.conf /tmp/

systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:19:02 UTC; 5s ago

mv /tmp/sp-location.keystone-saml-mellon.conf mellon/

echo deadbeef >> mellon/sp-location.keystone-saml-mellon.conf

# looks like it actually gets imported which is what we are after
systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

artful-proposed:

apt policy apache2
apache2:
  Installed: 2.4.27-2ubuntu4.2
  Candidate: 2.4.27-2ubuntu4.2
  Version table:
 *** 2.4.27-2ubuntu4.2 500
        500 http://archive.ubuntu.com/ubuntu artful-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

grep IncludeOptional sites-enabled/openstack_https_frontend.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf

stat mellon/sp-location.keystone-saml-mellon.conf
  File: 'mellon/sp-location.keystone-saml-mellon.conf'
# ...

systemctl restart apache2 && sleep 10 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:26:15 UTC; 10s ago

mv mellon/sp-location.keystone-saml-mellon.conf /tmp/

systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:27:59 UTC; 5s ago

systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

artful-proposed (added missing statements):

apt policy apache2
apache2:
  Installed: 2.4.27-2ubuntu4.2
  Candidate: 2.4.27-2ubuntu4.2
  Version table:
 *** 2.4.27-2ubuntu4.2 500
        500 http://archive.ubuntu.com/ubuntu artful-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

grep IncludeOptional sites-enabled/openstack_https_frontend.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf

stat mellon/sp-location.keystone-saml-mellon.conf
  File: 'mellon/sp-location.keystone-saml-mellon.conf'
# ...

systemctl restart apache2 && sleep 10 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:26:15 UTC; 10s ago

mv mellon/sp-location.keystone-saml-mellon.conf /tmp/

systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:27:59 UTC; 5s ago

mv /tmp/sp-location.keystone-saml-mellon.conf mellon/

echo deadbeef >> mellon/sp-location.keystone-saml-mellon.conf

systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

bionic-proposed:

apt policy apache2
apache2:
  Installed: 2.4.29-1ubuntu4.2
  Candidate: 2.4.29-1ubuntu4.2
  Version table:
 *** 2.4.29-1ubuntu4.2 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

grep IncludeOptional sites-enabled/openstack_https_frontend.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf
    IncludeOptional /etc/apache2/mellon/sp-location*.conf

stat mellon/sp-location.keystone-saml-mellon.conf
  File: 'mellon/sp-location.keystone-saml-mellon.conf'

systemctl restart apache2 && sleep 10 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:32:18 UTC; 10s ago

mv mellon/sp-location.keystone-saml-mellon.conf /tmp/

systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
   Active: active (running) since Fri 2018-06-22 22:32:51 UTC; 5s ago

mv /tmp/sp-location.keystone-saml-mellon.conf mellon/

echo deadbeef >> mellon/sp-location.keystone-saml-mellon.conf

systemctl restart apache2 && sleep 5 && systemctl status apache2 | grep Active
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

tags: added: verification-done verification-done-artful verification-done-bionic verification-done-xenial
removed: verification-needed verification-needed-artful verification-needed-bionic verification-needed-xenial
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'm investigating the DEP8 errors.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.2

---------------
apache2 (2.4.29-1ubuntu4.2) bionic; urgency=medium

  * debian/patches/includeoptional-ignore-non-existent.patch: silently
    ignore a not existent file path with IncludeOptional . Closes LP:
    #1766186.

 -- Andreas Hasenack <email address hidden> Thu, 07 Jun 2018 18:10:10 -0300

Changed in apache2 (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for apache2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Artful:
- the cacti regressions are due to a mariadb regression: https://bugs.launchpad.net/ubuntu/+source/mariadb-10.1/+bug/1757107
After being installed, mariadb isn't running, and thus the cacti postinst can't configure a database and cacti fails to work.
- resource-agents on s390x: will investigate next

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Artful:
- the resource-agents failures (not just in s390) were fixed in https://bugs.launchpad.net/ubuntu/+source/resource-agents/+bug/1734148 for bionic. Since Artful is EOL soon, probably not worth an SRU.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

TL;DR artful:
- cacti regressions are due to mariadb-server regression bug #1757107. After being installed, mariadb isn't running. I applied this crude workaround (https://pastebin.ubuntu.com/p/TwJBtJq96J/) and ran the tests locally, and they passed: http://people.ubuntu.com/~ahasenack/artful-dep8-cacti-workaround/
Note that the workaround doesn't work if what you want to test is a new upload of cacti itself, since I purge and reinstall it. But this SRU is about apache, so it should be enough.
- resource-agents failures were fixed for bionic in bug #1734148 and are probably not worth an SRU since artful is EOL in a few days.

There are no other artful failures.

Therefore, I ask to let this SRU continue for Artful.

I'll now investigate the xenial DEP8 failures.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Xenial:

a) libapache2-mod-perl2
two test errors: t/apache/read.t and t/filter/in_bbs_inject_header.t. Both are fixed in 2.0.10-2 in debian:
libapache2-mod-perl2 (2.0.10-2) unstable; urgency=medium

  * Patch the test suite for Apache 2.4.24 compatibility.
    Thanks to Stefan Fritsch. (Closes: #849082)

 -- Niko Tyni <email address hidden> Sun, 25 Dec 2016 11:51:10 +0200

This is the diff:
https://git.launchpad.net/ubuntu/+source/libapache2-mod-perl2/commit/?h=debian/buster&id=1a0e6c046960f816f0089d8d164f7fa9b077719f

I filed a bug and attached a branch to it: https://bugs.launchpad.net/ubuntu/+source/libapache2-mod-perl2/+bug/1779400

Here is the MP: https://code.launchpad.net/~ahasenack/ubuntu/+source/libapache2-mod-perl2/+git/libapache2-mod-perl2/+merge/348780

PPA with test packages: https://launchpad.net/~ahasenack/+archive/ubuntu/libapache2-mod-perl2-dep8/+packages

DEP8 run with the PPA packages: http://people.ubuntu.com/~ahasenack/deb8-modperl2-xenial-ppa/

b) gvfs
This is an "old friend". The gvfs tests are very bad, just see the history in http://autopkgtest.ubuntu.com/packages/g/gvfs/xenial/s390x. I'll give it a few retries.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Xenial tldr:

a) libapache2-mod-perl2 failing across the board. Addressed in bug #1779400 (SRU), which is in review. That bug also has a sample dep8 run with the fixed packages, showing they pass with this apache from proposed.

b) gvfs
flaky tests, with a big tendency to fail, finally passed.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apache2 - 2.4.18-2ubuntu3.9

---------------
apache2 (2.4.18-2ubuntu3.9) xenial; urgency=medium

  * debian/patches/includeoptional-ignore-non-existent.patch: silently
    ignore a not existent file path with IncludeOptional . Closes LP:
    #1766186.

 -- Andreas Hasenack <email address hidden> Thu, 07 Jun 2018 16:43:03 -0300

Changed in apache2 (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apache2 - 2.4.27-2ubuntu4.2

---------------
apache2 (2.4.27-2ubuntu4.2) artful; urgency=medium

  * debian/patches/includeoptional-ignore-non-existent.patch: silently
    ignore a not existent file path with IncludeOptional . Closes LP:
    #1766186.

 -- Andreas Hasenack <email address hidden> Thu, 07 Jun 2018 17:53:23 -0300

Changed in apache2 (Ubuntu Artful):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.