Unable to connect to object store with TLS enabled

Bug #1883842 reported by Frode Nordahl
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
simplestreams
In Progress
High
James Page

Bug Description

As run by the gss charm with snap support:

The Keystnoe bits appear to dtrt, but when also the Swift API is on a HTTPS URL things blow up.

Tested using cs:~openstack-charmers-next/ceph-radosgw with relation to vault as object store provider.

$ juju run-action --wait glance-simplestreams-sync/1 sync-images
unit-glance-simplestreams-sync-1:
  UnitId: glance-simplestreams-sync/1
  id: "22"
  results:
    Stderr: |
      Traceback (most recent call last):
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
          cnx.do_handshake()
        File "/snap/simplestreams/12/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
          self._raise_ssl_error(self._ssl, result)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
          _raise_current_error()
        File "/snap/simplestreams/12/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
          raise exception_type(errors)
      OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
          chunked=chunked,
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
          self._validate_conn(conn)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
          conn.connect()
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connection.py", line 360, in connect
          ssl_context=context,
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
          return context.wrap_socket(sock)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
          raise ssl.SSLError("bad handshake: %r" % e)
      ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
          timeout=timeout
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
          method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
        File "/snap/simplestreams/12/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
          raise MaxRetryError(_pool, url, error or ResponseError(cause))
      urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.16.122.72', port=443): Max retries exceeded with url: /swift/v1/simplestreams (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/snap/simplestreams/12/bin/sstream-mirror-glance", line 185, in <module>
          main()
        File "/snap/simplestreams/12/bin/sstream-mirror-glance", line 161, in main
          tstore = swift.SwiftObjectStore(args.output_swift, region=region)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/simplestreams/objectstores/swift.py", line 78, in __init__
          '.r:*,.rlistings'})
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 1836, in put_container
          query_string=query_string)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 1748, in _retry
          service_token=self.service_token, **kwargs)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 1081, in put_container
          conn.request(method, path, '', req_headers)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 469, in request
          files=files, **self.requests_args)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/swiftclient/client.py", line 452, in _request
          return self.request_session.request(*arg, **kwarg)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
          resp = self.send(prep, **send_kwargs)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
          r = adapter.send(request, **kwargs)
        File "/snap/simplestreams/12/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
          raise SSLError(e, request=request)
      requests.exceptions.SSLError: HTTPSConnectionPool(host='172.16.122.72', port=443): Max retries exceeded with url: /swift/v1/simplestreams (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
      /usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py:108: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
        confobj = yaml.load(f)
      /usr/lib/python3/dist-packages/keystoneauth1/adapter.py:235: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1.
        warnings.warn('Using keystoneclient sessions has been deprecated. '
  status: completed
  timing:
    completed: 2020-06-17 06:54:09 +0000 UTC
    enqueued: 2020-06-17 06:54:06 +0000 UTC
    started: 2020-06-17 06:54:06 +0000 UTC

Related branches

Revision history for this message
Paride Legovini (paride) wrote :

Hi Frode,

Just to be sure: this does not happen with the simplestreams .deb from the Ubuntu repos, but it does happen with the snap, everything else being the same. Correct?

James Page (james-page)
Changed in simplestreams:
status: New → Triaged
importance: Undecided → High
assignee: nobody → James Page (james-page)
status: Triaged → In Progress
Revision history for this message
Frankline Ngatia (fngatia) wrote :
Download full text (6.9 KiB)

20201 Wallaby
@>juju run-action --wait glance-simplestreams-sync/2 sync-images
unit-glance-simplestreams-sync-2:
  UnitId: glance-simplestreams-sync/2
  id: "69"
  message: exit status 1
  results:
    ReturnCode: 1
    Stderr: |
      /usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py:109: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
        confobj = yaml.load(f)
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
          cnx.do_handshake()
        File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
          self._raise_ssl_error(self._ssl, result)
        File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
          _raise_current_error()
        File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
          raise exception_type(errors)
      OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
          httplib_response = self._make_request(
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
          self._validate_conn(conn)
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
          conn.connect()
        File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
          self.sock = ssl_wrap_socket(
        File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
          return context.wrap_socket(sock)
        File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
          raise ssl.SSLError("bad handshake: %r" % e)
      ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
          resp = conn.urlopen(
        File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
          retries = retries.increment(
        File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
          raise MaxRetryError(_pool, url, error or ResponseError(cause))
      urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.0.0.35', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/...

Read more...

Revision history for this message
Nikolay Vinogradov (nikolay.vinogradov) wrote :

Just hit this bug as well. Using the current stable version of the g-s-s charm [1] with simplestreams snap. SSL CA comes over 'certificates' relation to vault. The problem seems to be that strictly confined 'simplestreams' snap can't access host's truststore where the CA certificate is installed by g-s-s charm.

Workaround is to reinstall the snap with --devmodel

Please fix.

Revision history for this message
Nikolay Vinogradov (nikolay.vinogradov) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.