Apache2 Balancer Manager mod_proxy_balancer not working after Update

Thanks for your bug report. The "ignoring params in balancer-manager cross-site access" error message has been introduced as part of the patchset fixing CVE-2019-10092, see [1], so this definitely looks like a regression.

[1] https://git.launchpad.net/ubuntu/+source/apache2/tree/debian/patches/CVE-2019-10092-3.patch?id=e7a4a4340e4c6bae39d8f974aab81fdc05518e62

I subscribed and pinged ubuntu-security on this one, let's see if they chime in and what their opinion is.

i found https://bz.apache.org/bugzilla/show_bug.cgi?id=63688 and this sounds like of a similar problem and i can reporduce that within debian 10 which i described there.

https://bz.apache.org/bugzilla/show_bug.cgi?id=63688#c5

and there is a Patch available

with that patch from here

https://bz.apache.org/bugzilla/show_bug.cgi?id=63688#c3

and with the ubuntu 18.04 apache2 sources

:~$ apt-get source apache2

:~$ find . -name mod_proxy_balancer.c
./apache2-2.4.29/.pc/balance-member-long-hostname-part2.patch/modules/proxy/mod_proxy_balancer.c
./apache2-2.4.29/.pc/CVE-2019-10092-3.patch/modules/proxy/mod_proxy_balancer.c
./apache2-2.4.29/modules/proxy/mod_proxy_balancer.c

i copy and patched only that single file

:~$ cp ./apache2-2.4.29/modules/proxy/mod_proxy_balancer.c ~/

:~$ patch mod_proxy_balancer.c mod_proxy_balancer_patch.c
patching file mod_proxy_balancer.c
Hunk #1 succeeded at 1078 (offset -107 lines).

compile it

:~# apxs2 -c -i mod_proxy_balancer.c

i got also no more error log entries in my sandbox env from above. i copied that compiled binary away and i try early next week with that one if the initial problem behinde the proxy is also solved.

Thanks for linking the upstream bug and your experiments Horst!

In the bug there it was mentioned that this would not be related to the CVE fix CVE-2019-10092.
But it made me think as Horst clearly found it to be related to that update.

I did some of the same checks Horst did (in which patch is the balancer touched).
There are three patches in the package referenced for this CVE:
- debian/patches/CVE-2019-10092-1.patch: based on [1] which matches the upstream referred [2]
- debian/patches/CVE-2019-10092-2.patch: based on [3] which might be some related cleanup and no
  big changes (but not part of the upstream CVE change)
- debian/patches/CVE-2019-10092-3.patch: based on [4]
  This last one is what brings changes to proxy/mod_proxy_balancer.c
  It is not directly tied to CVE-2019-10092 but seems to be picked up in that context.

That at least somewhat explains upstreams confusion on "referenced change to mod_proxy/mod_proxy_balancer has NOTHING to do with CVE-2019-10092". I agree that this was an extra change unrelated to that.

And if I got Horst right in the former comment he confirmed that if he drops that change it seems to work again.

But it seems (other than the mis-tag to CVE-2019-10092) this hardening to XSRF was an intended change by upstream [5].
I wasn't able to follow all comments of the upstream bug, they mentioned lynx might be incompatible to it- but does that apply to some proxies as well then?
In that case this might be a hard call on security-SRUing this into Bionic and breaking things. But while this is a no-go for normal SRUs security sometimes required changes like that.

@sbeattie - could you outline what was going on in the CVE discussions when this XSRF protection was added. And if you have any known discussions on adding XSRF protection that includes balancing those proxies/browsers.

[1]: https://svn.apache.org/viewvc?view=revision&revision=1864207
[2]: https://svn.apache.org/viewvc?view=revision&revision=1864191
[3]: https://svn.apache.org/viewvc?view=revision&revision=1864702
[4]: https://svn.apache.org/viewvc?view=revision&revision=1864787
[5]: https://bz.apache.org/bugzilla/show_bug.cgi?id=63688#c7

@Horst
I have put a preliminary build of the packaged Apache to the PPA [1] with the fix that was suggested on the upstream bug [2]. Could you give that one a try?

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1842701-mod-proxy-xsrf
[2]: https://bz.apache.org/bugzilla/show_bug.cgi?id=63688#c3

Sorry for the problems that people are experiencing.

Christian, the Ubuntu Security Team will sometimes incorporate a hardening measure like the extra XSRF that upstream included in the 2.4.41 release, if it appears to address similar issues as the original vulnerability. Looking at the history of modules/proxy/ in the 2.4.x branch made it look like they were mildly related. Unfortunately, upstream did not make explicitly clear in the 2.4.x branch which commits specifically addressed each vulnerability (and in fact, upstream managed to silently break an embargo with their fix for CVE-2019-9517).

The debian/patches/CVE-2019-10092-2.patch is a fixup to the first patch, because in the first patch, a couple of log numbers were missed in the emitted error messages.

The issues with the https://svn.apache.org/viewvc?view=revision&revision=1864787 (so misnamed as CVE-2019-10092-3.patch) should affect xenial and disco as well, not just bionic, since it was backported to those releases as well.

I've made available pacakges which incorporate both patches mentioned in the upstream bug report (the one for the strcasecmp change and the change in the referrer test) in the ppa https://launchpad.net/~sbeattie/+archive/ubuntu/lp1842701/ for testing. Please let me now if these address the issues that people are seeing.

Thanks, and again, my apologies.

Thanks for the explanations Steve.
I almost assumed something like this (adding related hardening) and this should not have been any blaming. I was just dissecting the case one step at a time.

Thanks for doing the next step already with the builds for all affected releases.

In that case I can stop the coding myself but I want to continue to help. And that might be with some testing instead.
I'll try if I can set up and repro the issues that were reported ...

First of all, thanks to the great steps by Horst I was able to reproduce this on X/B/D releases.
like:
[Tue Sep 10 06:39:37.715128 2019] [proxy_balancer:error] [pid 3314:tid 140601611724544] [client 127.0.0.1:50998] AH10187: ignoring params in balancer-manager cross-site accessWith

With that set up I upgraded all those to the PPA [1] and retried the access.
It works without the AH10187 error now.
I also found no other new issue in the logs triggered by the update - at least not for this setup.

In an SRU sense I'd now call it verified.
I hope that helps to get this processed further, since it needs pushing to the -security pocket I can't help much further.

[1]: https://launchpad.net/~sbeattie/+archive/ubuntu/lp1842701

sorry i can't use your PPAs in the production. for a quick test i used my patched compiled module where only one line is changing from the patch i discribed above

:$ diff mod_proxy_balancer.c_org mod_proxy_balancer.c
1081c1081
< && (!ref || !safe_referer(r, ref))) {
---
> && (ref && !safe_referer(r, ref))) {

updated on of my production machine with the apache packages 2.4.29-1ubuntu4.10 and copy that module. lynx ist start working but the initial problem from outside over the bastian host proxy is not solved.

i will try to create a more better test env to use your ppas behinde a proxy but i'am sorry this needs a while.

thx for all your work, horst

unfortunately the ppa solve also not the behind a proxy problem.
usualy in my produktion in front (bastion/proxy host) is debian 9
so i test both with debian 9 and ubuntu 18.04 ppa at on the proxy
host.

i modified a littel the configuration to get closer for the
production env.

VM with LB Manager IP:192.168.56.211

i start again with the old apache version

:~# apt-get install libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap liblua5.2-0
:~# dpkg -i apache2_2.4.29-1ubuntu4.8_amd64.deb apache2-bin_2.4.29-1ubuntu4.8_amd64.deb apache2-data_2.4.29-1ubuntu4.8_all.deb apache2-utils_2.4.29-1ubuntu4.8_amd64.deb

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# vim /etc/apache2/sites-available/management.conf
<VirtualHost 192.168.56.211:81 127.0.0.1:81>
    Servername 127.0.0.1
    ServerAdmin root@localhost

    <Location /balancer-manager>
        SetHandler balancer-manager
        Require local
        #Require ip 192.168.56.0/24 127.0.0.1/24
        Require all granted
    </Location>

    <Location /test-web01/balancer-manager>
        SetHandler balancer-manager
        Require local
        #Require ip 192.168.56.0/24 127.0.0.1/24
        Require all granted
    </Location>

    LogLevel warn
    ErrorLog ${APACHE_LOG_DIR}/management_error.log
    CustomLog ${APACHE_LOG_DIR}/management_access.log combined

</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# vim /etc/apache2/sites-available/proxytest.conf
<Proxy "balancer://test">
        BalancerMember "http://192.168.168.130/test"
        BalancerMember "http://192.168.168.131/test" status=+H
        ProxySet lbmethod=bybusyness
</Proxy>

<VirtualHost 127.0.0.1:8100>
ServerAdmin root@localhost
ServerName testapp01
ServerAlias 127.0.0.1:8100

    ProxyPass "/test" "balancer://test"
    ProxyPassReverse "/test" "balancer://test"

    CustomLog ${APACHE_LOG_DIR}/test-access.log combined
    ErrorLog ${APACHE_LOG_DIR}/test-error.log

</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:~# a2enmod proxy_balancer proxy_http lbmethod_bybusyness lbmethod_byrequests
:~# a2ensite management proxytest

:~# vim /etc/apache2/ports.conf
[...]
Listen 81
Listen 8100

:~# systemctl restart apache2

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

check localhost LB Manager

:~# apt-get install lynx

:~# lynx 127.0.0.1:81/balancer-manager
:~# lynx 127.0.0.1:81/test-web01/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
-> worked as expectet
-> no log entries on the LB Manager VM

-------------------------------------------------------------------------

Bastion Host Proxy VM IP:192.168.56.230

:~# apt-get install apache2 lynx

check from proxy VM that LB Manager is working without a proxy config in
front of them.

:~# lynx 192.168.56.211:81/balancer-manager
:~# lynx 192.168.56.211:81/test-web01/balancer-manager

:~# tail -f /var/log/apache2/management_error.log
-> no log entries on the LB Manager VM

start proxy configuration

:~# vim /etc...

This bug was fixed in the package apache2 - 2.4.18-2ubuntu3.13

---------------
apache2 (2.4.18-2ubuntu3.13) xenial-security; urgency=medium

  * SECURITY REGRESSION: mod_proxy balancer XSS/CSRF hardening broke
    browsers which change case in headers and breaks balancers
    loading in some configurations (LP: #1842701)
    - drop d/p/CVE-2019-10092-3.patch

 -- Steve Beattie <email address hidden> Mon, 16 Sep 2019 06:13:53 -0700

This bug was fixed in the package apache2 - 2.4.29-1ubuntu4.11

---------------
apache2 (2.4.29-1ubuntu4.11) bionic-security; urgency=medium

  * SECURITY REGRESSION: mod_proxy balancer XSS/CSRF hardening broke
    browsers which change case in headers and breaks balancers
    loading in some configurations (LP: #1842701)
    - drop d/p/CVE-2019-10092-3.patch

 -- Steve Beattie <email address hidden> Mon, 16 Sep 2019 05:58:48 -0700

This bug was fixed in the package apache2 - 2.4.38-2ubuntu2.3

---------------
apache2 (2.4.38-2ubuntu2.3) disco-security; urgency=medium

  * SECURITY REGRESSION: mod_proxy balancer XSS/CSRF hardening broke
    browsers which change case in headers and breaks balancers
    loading in some configurations (LP: #1842701)
    - drop d/p/CVE-2019-10092-3.patch

 -- Steve Beattie <email address hidden> Mon, 16 Sep 2019 05:36:25 -0700

with the new packages my problem is solved.

on more question in the next Ubuntu release for example 20.04 with a newer apache version. it is possible that this kind of problem is comming back again? because the patches are in the newer version from apache.org.

thx again, regards horst

Hi Horst,
yes I checked and the issue is in Eoan 2.4.41 - I checked that already last week and let Steve now.

Steve wanted to track the upstream discussions on this as going forward we most likely want to follow upstreams guidance on this (e.g. want to have it broken for better security).

But thanks for the ping, we might want to mark the bug tasks accordingly to make this clear.

hi Christian,

thx for the info and please let me know if there is a posibility solution for the future releases.

I'll if I hear something, but I'll leave that task mostly to Steve who said that he wanted to keep an eye on it (for potentially backporting the hardening once we know how to handle the regression).

Looks like this is still open for Groovy, but will be resolved when we merge 2.4.42.

To close this out, fixed in Groovy
 apache2 | 2.4.46-1ubuntu1 | groovy | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390x

Likely related (or even duplicate): LP: #1939678.