Neutron server log shows: 2018-07-17 15:09:26.244 28 INFO neutron.wsgi [-] 172.16.81.58 "GET /v2.0/floatingips.json HTTP/1.1" status: 500 len: 380 time: 0.0040829 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors [-] An error occurred during processing the request: GET /v2.0/floatingips.json HTTP/1.0^M Accept: application/json^M Accept-Encoding: gzip, deflate^M Content-Type: text/plain^M Host: 10.87.1.100:13696^M User-Agent: python-neutronclient^M X-Auth-Token: ***** X-Forwarded-Proto: https: AttributeError: 'HTTPConnectionPool' object has no attribute 'key_file' 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors Traceback (most recent call last): 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__ 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors response = req.get_response(self.application) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1316, in send 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1280, in call_application 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__ 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **self.kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 334, in __call__ 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors response = self.process_request(req) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 633, in process_request 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = super(AuthProtocol, self).process_request(request) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 407, in process_request 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors allow_expired=allow_expired) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 440, in _do_fetch_token 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors data = self.fetch_token(token, **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 772, in fetch_token 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors allow_expired=allow_expired) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors auth_ref = self._request_strategy.verify_token( 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors strategy_class = self._get_strategy_class() 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors if self._adapter.get_endpoint(version=klass.AUTH_VERSION): 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in get_endpoint 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors return self.session.get_endpoint(auth or self.auth, **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in get_endpoint 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors return auth.get_endpoint(self, **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 379, in get_endpoint 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors allow_version_hack=allow_version_hack, **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 270, in get_endpoint_data 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors service_catalog = self.get_access(session).service_catalog 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors self.auth_ref = self.get_auth_ref(session) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors self._plugin = self._do_create_plugin(session) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors authenticated=False) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 556, in get_discovery 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors authenticated=authenticated) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 1171, in get_discovery 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors disc = Discover(session, url, authenticated=authenticated) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 401, in __init__ 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors authenticated=authenticated) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 99, in get_version_data 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = session.get(url, headers=headers, authenticated=authenticated) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 840, in get 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors return self.request(url, 'GET', **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 698, in request 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = send(**kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 756, in _send_request 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = self.session.request(method, url, **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = self.send(prep, **send_kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors r = adapter.send(request, **kwargs) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 405, in send 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors self.cert_verify(conn, request.url, verify, cert) 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 247, in cert_verify 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors if conn.key_file and not os.path.exists(conn.key_file): 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors AttributeError: 'HTTPConnectionPool' object has no attribute 'key_file' 2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors this was tracked back to find that /etc/neutron/plugins/opencontrail/ContrailPlugin.ini in the container (/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/opencontrail/ContrailPlugin.ini on the host) has set: [KEYSTONE] ... cafile=/etc/pki/tls/private/overcloud_endpoint.pem certfile=/etc/pki/tls/private/overcloud_endpoint.pem [keystone_authtoken] ... cafile=/etc/pki/tls/private/overcloud_endpoint.pem certfile=/etc/pki/tls/private/overcloud_endpoint.pem So we can see that 'certfile' is set but not 'keyfile'. The requests library expects that the 'keyfile' is set when the 'certfile' is set. If we remove the 'certfile' and restart the neutron_api container then the neutron service is functioning correctly. Root cause: the /usr/share/openstack-puppet/modules/tripleo/manifests/network/contrail/neutron_plugin.pp on the undercloud is the puppet manifest that generates this content. This file should be adjusted to not set the 'certfile'. This manifest is delivered by the puppet-tripleo-8.3.2-8.el7ost.noarch package of the rhel-7-server-openstack-13-rpms repository (i.e. it comes from upstream).