Zaza vault initialization often fails with CERTIFICATE_VERIFY_FAILED

Bug #1911920 reported by Aurelien Lourot
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Keystone SAML Mellon Charm
In Progress
High
Aurelien Lourot

Bug Description

Seen several times on OSCI, so far always on focal-ussuri and never on bionic-queens:

https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/761564
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/761564/2/7686/index.html
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/761564/2/7724/index.html

https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/770056
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/770056/1/7721/index.html

Traceback (most recent call last):
  File "/tmp/tmp.Ipg3xRHGbV/func/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 278, in main
    test_directory=args.test_directory)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 213, in func_test_runner
    force=force, test_directory=test_directory)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 143, in run_env_deployment
    test_directory=test_directory)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
    run_configure_list(functions)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/vault/setup.py", line 181, in auto_initialize
    validate_ca(cacertificate, application=validation_application)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/vault/setup.py", line 238, in validate_ca
    requests.get('https://{}:{}'.format(ip, str(port)), verify=fp.name)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/tmp.Ipg3xRHGbV/func/lib/python3.5/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='172.17.103.230', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))

Revision history for this message
Aurelien Lourot (aurelien-lourot) wrote :

Actually this might have been fixed by recent changes in charmhelpers and charms.openstack. Re-triggering the linked reviews.

Changed in charm-keystone-saml-mellon:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Aurelien Lourot (aurelien-lourot)
Revision history for this message
Aurelien Lourot (aurelien-lourot) wrote :

We'll have to get this one landed so that the suspected fixes make their way into the vault charm: https://review.opendev.org/c/openstack/charm-vault/+/770438

Revision history for this message
Aurelien Lourot (aurelien-lourot) wrote :

For the record, here are the fixes I had in mind: https://bugs.launchpad.net/charm-placement/+bug/1893847

There the bug has been marked as Invalid for the vault charm, so I'll go ahead and re-trigger the keystone-saml-mellon reviews now.

Revision history for this message
Aurelien Lourot (aurelien-lourot) wrote :

Happening again:
https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/761564
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-keystone-saml-mellon/761564/3/7824/index.html

We're getting a CERTIFICATE_VERIFY_FAILED in zaza/openstack/charm_tests/vault/setup.py:validate_ca(cacertificate, application=validation_application) with validation_application being keystone.

It seems indeed like the latest cert-related fixes may haven't made their way into the latest keystone charm yet as this review hasn't landed yet:
https://review.opendev.org/c/openstack/charm-keystone/+/771022

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.