Too many vhost in apache2 are produced when juju config keystone os-*-hostname is in used
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Keystone Charm |
Triaged
|
High
|
Unassigned |
Bug Description
[IMPACT]
* Horizon oepnstack-dashboard
(see attached: openstack_
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/
[TEST CASE]
Reproducer can be found on comment #2.
[ORIGINAL DESCRIPTION]
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https:/
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateC
SSLCertificateK
ProxyPass / http://
ProxyPassReverse / http://
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.
Traceback (most recent call last):
File "/usr/lib/
result = method(req, **params)
File "/usr/lib/
return f(self, request, *args, **kwargs)
File "/usr/lib/
CONF.signing.
File "/usr/lib/
signing_
File "/usr/lib/
raise subprocess.
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
summary: |
- charm-keystones seems to produce more than 3 vhost for each endpoints + charm-keystone seems to produce more than 3 vhost for each endpoints |
description: | updated |
description: | updated |
description: | updated |
tags: | added: seg sts |
Changed in charm-keystone: | |
status: | New → Triaged |
importance: | Undecided → High |
description: | updated |
I have found a reproducer
Deploy keystone in HA and with SSL enabled.
If there is no hostname set, everything will works as expected.
The bug starts once one have set os-*-hostname config in keystone as follow:
$ juju config keystone os-admin- hostname= keystone. admin.local hostname= keystone. internal. local hostname= keystone. public. local
$ juju config keystone os-internal-
$ juju config keystone os-public-
# cat /etc/apache2/ sites-enabled/ openstack_ http_frontend. conf admin.local !MD5:!aNULL: !eNULL: !EXP:!LOW: !MEDIUM ateFile /etc/apache2/ ssl/keystone/ cert_keystone. admin.pub ateChainFile /etc/apache2/ ssl/keystone/ cert_keystone. admin.pub ateKeyFile /etc/apache2/ ssl/keystone/ key_keystone. admin.pub localhost: 4980/ verse / http:// localhost: 4980/ veHost on mellon* /sp-location* .conf admin.local !MD5:!aNULL: !eNULL: !EXP:!LOW: !MEDIUM ateFile /etc/apache2/ ssl/keystone/ cert_keystone. admin.pub ateChainFile /etc/apache2/ ssl/keystone/ cert_keystone. admin.pub ateKeyFile /etc/apache2/ ssl/keystone/ key_keystone. admin.pub localhost: 35337/ verse / http:// localhost: 35337/ veHost on mellon* /sp-location* .conf !MD5:!aNULL: !eNULL: !EXP:!LOW: !MEDIUM ateFile /etc/apache2/ ssl/keystone/ cert_keystone. int.pub ateChainFile /etc/apache2/ ssl/keystone/ cert_keystone. int.pub ateKeyFile /etc/apache2/ ssl/keystone/ key_keystone. int.pub localhost: 4980/ verse / http:// localhost: 4980/ veHost on mellon* /sp-location* .conf !MD5:!aNULL: !eNULL: !EXP:!LOW: !MEDIUM ateFile /etc/apache2/ ssl/keystone/ cert_keystone. int.pub ateChainFile /etc/apache2/ ssl/keystone/ cert_keystone. int.pub ateKeyFile /etc/apache2/ ssl/keystone/ key_keystone. int.pub localhost: 35337/ verse / http:// localhost: 35337/
Listen 4990
Listen 35347
<VirtualHost 10.5.0.32:4990>
ServerName keystone.
SSLEngine on
SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite HIGH:!RC4:
SSLCertific
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertific
SSLCertific
ProxyPass / http://
ProxyPassRe
ProxyPreser
RequestHeader set X-Forwarded-Proto "https"
IncludeOptional /etc/apache2/
</VirtualHost>
<VirtualHost 10.5.0.32:35347>
ServerName keystone.
SSLEngine on
SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite HIGH:!RC4:
SSLCertific
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertific
SSLCertific
ProxyPass / http://
ProxyPassRe
ProxyPreser
RequestHeader set X-Forwarded-Proto "https"
IncludeOptional /etc/apache2/
</VirtualHost>
<VirtualHost 10.5.0.32:4990>
ServerName keystone.int.local
SSLEngine on
SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite HIGH:!RC4:
SSLCertific
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertific
SSLCertific
ProxyPass / http://
ProxyPassRe
ProxyPreser
RequestHeader set X-Forwarded-Proto "https"
IncludeOptional /etc/apache2/
</VirtualHost>
<VirtualHost 10.5.0.32:35347>
ServerName keystone.int.local
SSLEngine on
SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite HIGH:!RC4:
SSLCertific
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertific
SSLCertific
ProxyPass / http://
ProxyPassRe
ProxyPreser...