OpenStack Keystone Charm

Too many vhost in apache2 are produced when juju config keystone os-*-hostname is in used

Bug #1919148 reported by Eric Desrochers
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
Triaged
High
Unassigned

Bug Description

[IMPACT]

* Horizon oepnstack-dashboard
(see attached: openstack_dashboard_screenshot)
"Unable to establish connection to keystone endpoint"

* Users are getting intermittent authentication errors like:

Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))

[TEST CASE]
Reproducer can be found on comment #2.

[ORIGINAL DESCRIPTION]
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.

# charm-helpers src code:

{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on

# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off

SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}

Seems like there is a 'for loop' creating a vhost entry for each address found.

I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.

Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.

Leading to connection failure and OpenSSL errors as such:

# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3

"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.

See original description
Tags: seg sts
Eric Desrochers (slashd)
summary: - charm-keystones seems to produce more than 3 vhost for each endpoints
+ charm-keystone seems to produce more than 3 vhost for each endpoints
description: updated
description: updated
description: updated
tags: added: seg sts
Revision history for this message
Eric Desrochers (slashd) wrote :
Download full text (4.8 KiB)

I have found a reproducer

Deploy keystone in HA and with SSL enabled.

If there is no hostname set, everything will works as expected.
The bug starts once one have set os-*-hostname config in keystone as follow:

$ juju config keystone os-admin-hostname=keystone.admin.local
$ juju config keystone os-internal-hostname=keystone.internal.local
$ juju config keystone os-public-hostname=keystone.public.local

# cat /etc/apache2/sites-enabled/openstack_http_frontend.conf
Listen 4990
Listen 35347
<VirtualHost 10.5.0.32:4990>
    ServerName keystone.admin.local
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/keystone/cert_keystone.admin.pub
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/keystone/cert_keystone.admin.pub
    SSLCertificateKeyFile /etc/apache2/ssl/keystone/key_keystone.admin.pub
    ProxyPass / http://localhost:4980/
    ProxyPassReverse / http://localhost:4980/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
    IncludeOptional /etc/apache2/mellon*/sp-location*.conf
</VirtualHost>
<VirtualHost 10.5.0.32:35347>
    ServerName keystone.admin.local
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/keystone/cert_keystone.admin.pub
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/keystone/cert_keystone.admin.pub
    SSLCertificateKeyFile /etc/apache2/ssl/keystone/key_keystone.admin.pub
    ProxyPass / http://localhost:35337/
    ProxyPassReverse / http://localhost:35337/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
    IncludeOptional /etc/apache2/mellon*/sp-location*.conf
</VirtualHost>
<VirtualHost 10.5.0.32:4990>
    ServerName keystone.int.local
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/keystone/cert_keystone.int.pub
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/keystone/cert_keystone.int.pub
    SSLCertificateKeyFile /etc/apache2/ssl/keystone/key_keystone.int.pub
    ProxyPass / http://localhost:4980/
    ProxyPassReverse / http://localhost:4980/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
    IncludeOptional /etc/apache2/mellon*/sp-location*.conf
</VirtualHost>
<VirtualHost 10.5.0.32:35347>
    ServerName keystone.int.local
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/keystone/cert_keystone.int.pub
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/keystone/cert_keystone.int.pub
    SSLCertificateKeyFile /etc/apache2/ssl/keystone/key_keystone.int.pub
    ProxyPass / http://localhost:35337/
    ProxyPassReverse / http://localhost:35337/
    ProxyPreser...

Read more...

description: updated
Revision history for this message
Eric Desrochers (slashd) wrote :

Instrumenting a local charm revealed that after the sorted() and set(), get_network_addresses() returns:

2021-03-22 18:58:23 ERROR juju-log debug unsorted: [('10.5.0.32', 'keystone.int.local'), ('10.5.0.32', 'keystone.admin.local'), ('10.5.0.32', 'keystone.public.local')]
2021-03-22 18:58:23 ERROR juju-log debug sorted: [('10.5.0.32', 'keystone.admin.local'), ('10.5.0.32', 'keystone.int.local'), ('10.5.0.32', 'keystone.public.local')]

Then later for loop inside __call__() consume everything as is and create more vhosts than it should.

        addresses = self.get_network_addresses()
        for address, endpoint in addresses:
            for api_port in self.external_ports:
                ext_port = determine_apache_port(api_port,
                                                 singlenode_mode=True)
                int_port = determine_api_port(api_port, singlenode_mode=True)
                portmap = (address, endpoint, int(ext_port), int(int_port))
                ctxt['endpoints'].append(portmap)
                ctxt['ext_ports'].append(int(ext_port))

        ctxt['ext_ports'] = sorted(list(set(ctxt['ext_ports'])))
        return ctxt

So it is working fine if no hostname(s) are config set in keystone, but as soon as it is, the charm cannot handle it logically.

summary: - charm-keystone seems to produce more than 3 vhost for each endpoints
+ charm-keystone produced too many vhosts when config os-*-hostname is in
+ used
summary: - charm-keystone produced too many vhosts when config os-*-hostname is in
- used
+ Too many vhost in apache2 are produced when juju config keystone
+ os-*-hostname is in used
description: updated
Alex Kavanagh (ajkavanagh)
Changed in charm-keystone:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Eric Desrochers (slashd) wrote :

# Same instrumental local charm without os-*-hostname config set:

2021-03-23 13:30:15 ERROR juju-log debug unsorted: [('10.5.3.136', '10.5.100.2'), ('10.5.3.136', '10.5.100.2'), ('10.5.3.136', '10.5.100.2')]
2021-03-23 13:30:15 ERROR juju-log debug sorted: [('10.5.3.136', '10.5.100.2')]

10.5.100.2 => VIP
10.5.3.136 => keystone/0 IP

We see here that duplicates are removed in this scenario, as opposed to when os-*-hostname are set.

$ juju config keystone vip
10.5.100.2

Revision history for this message
Eric Desrochers (slashd) wrote :

It would be important to see if the broken logic is present in other charms as well.
Several other services uses Apache for their API ? IIRC cinder is one of them ?

I focused my time on keystone, but since the code is part of charmhelper, it may impact other charm that uses similar functionalities I would assume.

- Eric

Revision history for this message
Eric Desrochers (slashd) wrote :

openstack_dashboard_screenshot

description: updated
description: updated
description: updated
description: updated
Eric Desrochers (slashd)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.