Apache2 Certificate Chain Verification within Proxy not Working after dist-upgrade to focal
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Apache2 Web Server |
Fix Released
|
Medium
|
|||
apache2 (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
Unassigned |
Bug Description
[Impact]
* Due to https:/
Online Certificate Status Protocol (OCSP) fails in proxy mode.
* The fix is simple (the wrong context was checked) and is upstream for
a while without further changes.
* Backporting that fix [1] resolves the use case
[1]: https:/
[Test Plan]
* Autopkgtest plus the steps that were outlined in comment 8 & 9.
* [racb] Also see the request for further testing in comment 14.
[Where problems could occur]
* Apache does many things, but the change "only" affects the ssl
engine. Therefore unexpected problems would be around any sort
of ssl activity.
But the way the change works is actually ont he SSLVerify path,
so it comes down to "making ssl connections" not e.g. later SSL
transmission behavior or throughtput.
[Other Info]
* If we manage to get a certbot system up on canonistack (as I did in
the past) to hit this issue we will use that testbed instead of the
local tests.
----
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
After dist-upgrade bionic -> focal and Apache Update
from: 2.4.29-1ubuntu4.14
to: 2.4.41-4ubuntu3.1
Overall I found a hint in
https:/
[...]
*) mod_ssl: OCSP does not apply to proxy mode. PR 63679.
[Lubos Uhliarik <luhliari redhat.com>, Yann Ylavic]
[...]
https:/
Backported to 2.4.x (r1872226), will be in the next release.
-> This is part of 2.4.42 <-
and a overall Question is can you please also backport that Version from
ssl_engine_kernel.c in your 2.4.41-4ubuntu3.1 Apache?
My Further on investigation. I Create a new VM with 20.04 an compile Apache
:~$ apt-get source apache2
The Only thing i do is to replace
:~$ apache2-
with the downloaded Version from upstream Apache
The *.deb Packages i Saved away.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Reproduce the Error
Create a New VM with 20.04
:~# apt-get install apache2
:~# mkdir /etc/apache2/ssl
:~# vim /etc/apache2/
in letsencryt.crt has only the intermediate ans rootCA from letsencryt
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
:~# vim /etc/apache2/
<VirtualHost 127.0.0.1:80>
ServerAdmin <email address hidden>
ServerName localhost
ProxyPreser
ProxyRequests Off
SSLProxyEngine On
SSLProxyVerify require
SSLProxyChe
SSLProxyChe
SSLProxyVer
SSLProxyCAC
SSLProxyCip
SSLProxyPro
ProxyPass / https:/
LogLevel debug
CustomLog ${APACHE_
</VirtualHost>
:~# vim /etc/apache2/
LogLevel debug
:~# a2enmod proxy_http ssl
:~# systemctl restart apache2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
I Create a local Firewall for better overview Block outgoing Traffic
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The Proxy crashed because -> connecting to OCSP responder. With the Apache
Version within bionic this does not happend. There is no connection to the
OCSP responder.
:~# curl http://
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server<
<hr>
<address>
</body></html>
:~# tail -f /var/log/
[Tue Jun 01 14:04:11.286448 2021] [authz_core:debug] [pid 6009:tid 140286852331264] mod_authz_
[Tue Jun 01 14:04:11.286530 2021] [proxy:debug] [pid 6009:tid 140286852331264] mod_proxy.c(1253): [client 127.0.0.1:47958] AH01143: Running scheme https handler (attempt 0)
[Tue Jun 01 14:04:11.286549 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org)
[Tue Jun 01 14:04:11.286588 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2379): [client 127.0.0.1:47958] AH00944: connecting https:/
[Tue Jun 01 14:04:11.288378 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2588): [client 127.0.0.1:47958] AH00947: connected / to localhorst.org:443
[Tue Jun 01 14:04:11.318587 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:04:11.318697 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:04:11.318726 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80)
[Tue Jun 01 14:04:11.368501 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_
[Tue Jun 01 14:04:11.369207 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_
[Tue Jun 01 14:04:11.369934 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_
[Tue Jun 01 14:04:11.370521 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_engine_
[Tue Jun 01 14:04:11.517640 2021] [ssl:debug] [pid 6009:tid 140286852331264] ssl_util_
[Tue Jun 01 14:04:11.521410 2021] [ssl:error] [pid 6009:tid 140286852331264] (101)Network is unreachable: [remote 94.130.99.225:443] AH01974: could not connect to OCSP responder 'r3.o.lencr.org'
[Tue Jun 01 14:04:11.521875 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834
[Tue Jun 01 14:04:11.529291 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed
[Tue Jun 01 14:04:11.529591 2021] [ssl:info] [pid 6009:tid 140286852331264] SSL Library Error: error:1416F086:SSL routines:
[Tue Jun 01 14:04:11.529708 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80)
[Tue Jun 01 14:04:11.529999 2021] [ssl:info] [pid 6009:tid 140286852331264] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502
[Tue Jun 01 14:04:11.530169 2021] [proxy:error] [pid 6009:tid 140286852331264] (20014)Internal error (specific information not available): [client 127.0.0.1:47958] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:04:11.530288 2021] [proxy:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH00898: Error during SSL Handshake with remote server returned by /
[Tue Jun 01 14:04:11.530379 2021] [proxy_http:error] [pid 6009:tid 140286852331264] [client 127.0.0.1:47958] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 ()
[Tue Jun 01 14:04:11.530482 2021] [proxy:debug] [pid 6009:tid 140286852331264] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org)
:~# tail -f /var/log/
Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.160 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=59096 DF PROTO=TCP SPT=52194 DPT=80 SEQ=2173056195 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0
Jun 1 14:04:12 devubu2004 fw-net REJECT IN= OUT=enp0s3 MAC= SRC=10.0.2.15 DST=95.101.91.146 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=32240 DF PROTO=TCP SPT=40016 DPT=80 SEQ=508673920 ACK=0 WINDOW=64240 SYN URGP=0 UID=33 GID=33 MARK=0
:~$ host r3.o.lencr.org
r3.o.lencr.org is an alias for o.lencr.
o.lencr.
a1887.dscq.
a1887.dscq.
a1887.dscq.
a1887.dscq.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Try out open the local Firewall
:~# vim /etc/shorewall/
[...]
ACCEPT $FW net:95.101.91.160 tcp http
ACCEPT $FW net:95.101.91.146 tcp http
:~# systemctl reload shorewall
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Does not help crashed with the Following Error
:~$ curl http://
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
The proxy server could not handle the request<p>Reason: <strong>Error during SSL Handshake with remote server<
<hr>
<address>
</body></html>
:~# tail -f /var/log/
[Tue Jun 01 14:08:02.137740 2021] [authz_core:debug] [pid 6009:tid 140286835545856] mod_authz_
[Tue Jun 01 14:08:02.137793 2021] [proxy:debug] [pid 6009:tid 140286835545856] mod_proxy.c(1253): [client 127.0.0.1:47974] AH01143: Running scheme https handler (attempt 0)
[Tue Jun 01 14:08:02.137803 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org)
[Tue Jun 01 14:08:02.137810 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2379): [client 127.0.0.1:47974] AH00944: connecting https:/
[Tue Jun 01 14:08:02.137817 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2588): [client 127.0.0.1:47974] AH00947: connected / to localhorst.org:443
[Tue Jun 01 14:08:02.167485 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:08:02.168160 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:08:02.168655 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80)
[Tue Jun 01 14:08:02.216198 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_
[Tue Jun 01 14:08:02.217565 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_
[Tue Jun 01 14:08:02.218976 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_
[Tue Jun 01 14:08:02.219265 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_engine_
[Tue Jun 01 14:08:02.358471 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.386985 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.579215 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581036 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581749 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581822 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581843 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581859 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581875 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581891 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581906 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.581922 2021] [ssl:debug] [pid 6009:tid 140286835545856] ssl_util_
[Tue Jun 01 14:08:02.583980 2021] [ssl:error] [pid 6009:tid 140286835545856] AH01924: Bad OCSP responder answer (bad nonce)
[Tue Jun 01 14:08:02.585222 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02276: Certificate Verification: Error (50): application verification failure [subject: CN=localhorst.org / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 04235D2681C6834
[Tue Jun 01 14:08:02.586201 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH02003: SSL Proxy connect failed
[Tue Jun 01 14:08:02.587160 2021] [ssl:info] [pid 6009:tid 140286835545856] SSL Library Error: error:1416F086:SSL routines:
[Tue Jun 01 14:08:02.587226 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01998: Connection closed to child 0 with abortive shutdown (server localhost:80)
[Tue Jun 01 14:08:02.587272 2021] [ssl:info] [pid 6009:tid 140286835545856] [remote 94.130.99.225:443] AH01997: SSL handshake failed: sending 502
[Tue Jun 01 14:08:02.587354 2021] [proxy:error] [pid 6009:tid 140286835545856] (20014)Internal error (specific information not available): [client 127.0.0.1:47974] AH01084: pass request body failed to 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:08:02.587391 2021] [proxy:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH00898: Error during SSL Handshake with remote server returned by /
[Tue Jun 01 14:08:02.587407 2021] [proxy_http:error] [pid 6009:tid 140286835545856] [client 127.0.0.1:47974] AH01097: pass request body failed to 94.130.99.225:443 (localhorst.org) from 127.0.0.1 ()
[Tue Jun 01 14:08:02.587424 2021] [proxy:debug] [pid 6009:tid 140286835545856] proxy_util.c(2340): AH00943: HTTPS: has released connection for (localhorst.org)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Close the Firewall Again
:~# vim /etc/shorewall/
[...]
#ACCEPT $FW net:95.101.91.160 tcp http
#ACCEPT $FW net:95.101.91.146 tcp http
:~# systemctl reload shorewall
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Installed the self compiled apache Version withe the Pateched ssl_engine_kernel.c
Version
:~# cd /home/vagrant/deb/
:~# dpkg -i apache2_
:~# systemctl stop apache2
:~# systemctl start apache2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Apache Proxy is working again as expected
:~# curl http://
-> webite is comming
:~# tail -f /var/log/
[Tue Jun 01 14:11:47.953485 2021] [authz_core:debug] [pid 7437:tid 140452002883328] mod_authz_
[Tue Jun 01 14:11:47.953554 2021] [proxy:debug] [pid 7437:tid 140452002883328] mod_proxy.c(1253): [client 127.0.0.1:47980] AH01143: Running scheme https handler (attempt 0)
[Tue Jun 01 14:11:47.953570 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2325): AH00942: HTTPS: has acquired connection for (localhorst.org)
[Tue Jun 01 14:11:47.953576 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2379): [client 127.0.0.1:47980] AH00944: connecting https:/
[Tue Jun 01 14:11:47.955415 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2588): [client 127.0.0.1:47980] AH00947: connected / to localhorst.org:443
[Tue Jun 01 14:11:47.985343 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3054): AH02824: HTTPS: connection established with 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:11:47.985479 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(3240): AH00962: HTTPS: connection complete to 94.130.99.225:443 (localhorst.org)
[Tue Jun 01 14:11:47.985505 2021] [ssl:info] [pid 7437:tid 140452002883328] [remote 94.130.99.225:443] AH01964: Connection to child 0 established (server localhost:80)
[Tue Jun 01 14:11:48.034945 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_
[Tue Jun 01 14:11:48.035920 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_
[Tue Jun 01 14:11:48.036745 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_
[Tue Jun 01 14:11:48.067180 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_engine_
[Tue Jun 01 14:11:48.068469 2021] [ssl:debug] [pid 7437:tid 140452002883328] ssl_util_
[Tue Jun 01 14:11:48.227809 2021] [proxy:debug] [pid 7437:tid 140452002883328] proxy_util.c(2340): AH00943: https: has released connection for (localhorst.org)
Regards Horst
Related branches
- Utkarsh Gupta (community): Approve
- Canonical Server packageset reviewers: Pending requested
- Canonical Server: Pending requested
-
Diff: 62 lines (+40/-0)3 files modifieddebian/changelog (+7/-0)
debian/patches/lp-1930430-Backport-r1865740.patch (+32/-0)
debian/patches/series (+1/-0)
Changed in apache2 (Ubuntu Focal): | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in apache2 (Ubuntu): | |
status: | Triaged → Fix Released |
Changed in apache2: | |
importance: | Unknown → Medium |
status: | Unknown → Fix Released |
tags: |
added: verification-done removed: verification-needed |
Created attachment 36728
Patch fixing the bug
Hi all,
in the commit r1826995 a following change has been made to ssl_callback_ SSLVerify function in ssl_engine_ kernel. c:
- if (ok && sc->server- >ocsp_enabled == TRUE) { >ocsp_mask & SSL_OCSPCHECK_ CHAIN) || >ocsp_mask & SSL_OCSPCHECK_ LEAF))) ) {
+ if (ok && ((sc->server-
+ (errdepth == 0 && (sc->server-
Instead of using sc->server, mctx should be used. It causes now weird behavior, since ocsp_mask is by default set to UNSET (which is -1, translated to signed int...). When proxy is set set on the same server, if-condition above will be true.
I'm proposing this change:
- if (ok && sc->server- >ocsp_enabled) { CHAIN) || LEAF))) ) {
+ if (ok && ((mctx->ocsp_mask & SSL_OCSPCHECK_
+ (errdepth == 0 && (mctx->ocsp_mask & SSL_OCSPCHECK_
It was working before, because ocsp_enabled was by default set to FALSE. ocsp_mask is UNSET by default now and is set either to proxy or server structure in sc. If sc with is_proxy is passed here, it will result in bug.
Attaching patch. Please merge it to 2.4.x if possible.