removing a unit from an application which is in a reverse proxy relation does not remove unit from balance

What I observed:

An Application with 2 units related to apache2 with balancer relation. One of those units is removed. The balancer config is not updated.

$ cat /etc/apache2/conf-enabled/blues-browser.balancer.conf
<Proxy balancer://blues-browser>
BalancerMember http://10.142.0.6:6543 timeout=60
BalancerMember http://10.142.0.10:6543 timeout=60

    ProxySet lbmethod=byrequests
    RequestHeader set X-Balancer-Name "blues-browser"
</Proxy>

The 10.142.0.10 machine does not exist. It was removed, but the charm did not update this config.