| 2021-03-15 12:24:10 |
Eric Desrochers |
bug |
|
|
added bug |
| 2021-03-15 12:24:22 |
Eric Desrochers |
summary |
charm-keystones seems to produce more than 3 vhost for each endpoints |
charm-keystone seems to produce more than 3 vhost for each endpoints |
|
| 2021-03-15 12:24:54 |
Eric Desrochers |
description |
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each addresses found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts, which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
Seeems like I found a corner case bug working with an impacted user.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each addresses found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts, which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
|
| 2021-03-15 12:26:58 |
Eric Desrochers |
description |
Seeems like I found a corner case bug working with an impacted user.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each addresses found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts, which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
Seeems like I found a corner case bug working with an impacted user.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
|
| 2021-03-15 12:27:34 |
Eric Desrochers |
description |
Seeems like I found a corner case bug working with an impacted user.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
|
| 2021-03-15 12:27:48 |
Eric Desrochers |
tags |
|
seg sts |
|
| 2021-03-17 01:42:32 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
| 2021-03-22 18:52:06 |
Eric Desrochers |
description |
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
|
| 2021-03-22 19:58:28 |
Eric Desrochers |
summary |
charm-keystone seems to produce more than 3 vhost for each endpoints |
charm-keystone produced too many vhosts when config os-*-hostname is in used |
|
| 2021-03-22 19:58:58 |
Eric Desrochers |
summary |
charm-keystone produced too many vhosts when config os-*-hostname is in used |
Too many vhost in apache2 are produced when juju config keystone os-*-hostname is in used |
|
| 2021-03-22 20:00:04 |
Eric Desrochers |
description |
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should.
Question is how charm-helper or charm-keystone manage to create that many vhosts ?
When it should have only created 3 of them. |
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
|
| 2021-03-23 10:35:04 |
Alex Kavanagh |
charm-keystone: status |
New |
Triaged |
|
| 2021-03-23 10:35:13 |
Alex Kavanagh |
charm-keystone: importance |
Undecided |
High |
|
| 2021-03-24 15:07:40 |
Eric Desrochers |
description |
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
[IMPACT]
* Horizon dashboard:
"Unable to establish to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
|
| 2021-03-24 15:08:17 |
Eric Desrochers |
description |
[IMPACT]
* Horizon dashboard:
"Unable to establish to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
[IMPACT]
* Horizon dashboard:
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
|
| 2021-03-24 15:10:37 |
Eric Desrochers |
attachment added |
|
openstack_dashboard_screenshot https://bugs.launchpad.net/charm-keystone/+bug/1919148/+attachment/5480511/+files/horizon_screenshot.png |
|
| 2021-03-24 15:11:10 |
Eric Desrochers |
description |
[IMPACT]
* Horizon dashboard:
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
[IMPACT]
* Horizon oepnstack-dashboard (see attached: openstack_dashboard_screenshot)
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
|
| 2021-03-24 15:11:21 |
Eric Desrochers |
description |
[IMPACT]
* Horizon oepnstack-dashboard (see attached: openstack_dashboard_screenshot)
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
[IMPACT]
* Horizon oepnstack-dashboard
(see attached: openstack_dashboard_screenshot)
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
|
| 2021-03-24 15:13:09 |
Eric Desrochers |
description |
[IMPACT]
* Horizon oepnstack-dashboard
(see attached: openstack_dashboard_screenshot)
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[Original Description]
Reproducer can be found on comment #2.
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
[IMPACT]
* Horizon oepnstack-dashboard
(see attached: openstack_dashboard_screenshot)
"Unable to establish connection to keystone endpoint"
* Users are getting intermittent authentication errors like:
Unable to establish connection to https://<URL>:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
[TEST CASE]
Reproducer can be found on comment #2.
[ORIGINAL DESCRIPTION]
Seems like I found a corner case bug working with an impacted user.
I say corner case, because I am unable to reproduce this behaviour in lab.
# charm-helpers src code:
{% for address, endpoint, ext, int in endpoints -%}
<VirtualHost {{ address }}:{{ ext }}>
ServerName {{ endpoint }}
SSLEngine on
# This section is based on Mozilla's recommendation
# as the "intermediate" profile as of July 7th, 2020.
# https://wiki.mozilla.org/Security/Server_Side_TLS
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCertificateFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
SSLCertificateChainFile /etc/apache2/ssl/{{ namespace }}/cert_{{ endpoint }}
SSLCertificateKeyFile /etc/apache2/ssl/{{ namespace }}/key_{{ endpoint }}
ProxyPass / http://localhost:{{ int }}/
ProxyPassReverse / http://localhost:{{ int }}/
ProxyPreserveHost on
RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
{% endfor -%}
Seems like there is a 'for loop' creating a vhost entry for each address found.
I'm not sure yet from where the charm consumes the addresses, but in this specific case I worked on. The user have more than 3 vhost (1 for each: admin, public, internal), he has in fact 6 vhosts (2 vhosts per endpoints for each port 4990 & 35347), which cause redirection to fail and hit the wrong SSL certificate.
Only redirection to admin endpoints work 100% of the time. Most likely because the admin vhost are the first loaded in Apache2 (At the top of the config file). So public redirection will most likely fail all the time.
Leading to connection failure and OpenSSL errors as such:
# keystone-0's keystone logs
(keystone.common.wsgi): 2021-03-05 09:31:40,178 ERROR Command 'openssl' returned non-zero exit status 3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
result = method(req, **params)
File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 82, in inner
return f(self, request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 361, in revocation_list
CONF.signing.keyfile)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 336, in cms_sign_text
signing_key_file_name, message_digest=message_digest)
File "/usr/lib/python2.7/dist-packages/keystoneclient/common/cms.py", line 384, in cms_sign_data
raise subprocess.CalledProcessError(retcode, 'openssl')
CalledProcessError: Command 'openssl' returned non-zero exit status 3
"openstack endpoint list --service keystone" command report only 1 region with 3 endpoints, as it should. |
|
