ConnectFailure: Unable to establish connection to gnocchi internal endpoint

Bug #1750095 reported by bhujay kumar
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloudkitty
In Progress
Undecided
bhujay kumar

Bug Description

Despite having interface=internalURL or endpoint_type =internalURL mentioned in relevant sections of cloudkitty.conf , during cloudkitty-storage-init the connection fails and tries to connect public interface.

The cloudkitty.ini sections tired are as follows:
[ks_auth]
memcached_servers = 10.0.0.171:11211,10.0.0.139:11211,10.0.0.250:11211
auth_uri = http://10.0.0.14:5000
project_domain_name = default
project_name = service
user_domain_name = default
password =
username = cloudkitty
auth_url = http://10.0.0.14:35357/v3
identity_uri = http://10.0.0.14:35357/v3
auth_type = v3password
interface = internalURL
endpoint_type = internalURL
debug = True
#endpoint_override = internalURL
#valid_interfaces = internalURL

[storage_gnocchi]
auth_section = ks_auth

[gnocchi_collector]
auth_section = ks_auth

The error during storage init is as follows

(cloudkitty-16.0.4) root@infra2-cloudkitty-container-6a50490b:~# cloudkitty-storage-init

2018-01-29 15:57:13.648 32617 CRITICAL cloudkitty [-] Unhandled error: ConnectFailure: Unable to establish connection to https://14.142.104.140:8041/v1/archive_policy/rating: HTTPSConnectionPool(host='14.142.104.140', port=8041): Max retries exceeded with url: /v1/archive_policy/rating (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6bc1efa890>: Failed to establish a new connection: [Errno 110] Connection timed out',))
2018-01-29 15:57:13.648 32617 ERROR cloudkitty Traceback (most recent call last):
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/bin/cloudkitty-storage-init", line 11, in <module>
2018-01-29 15:57:13.648 32617 ERROR cloudkitty sys.exit(main())
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 29, in main
2018-01-29 15:57:13.648 32617 ERROR cloudkitty init_storage_backend()
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 24, in init_storage_backend
2018-01-29 15:57:13.648 32617 ERROR cloudkitty backend.init()
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/cloudkitty/storage/gnocchi/__init__.py", line 94, in init
2018-01-29 15:57:13.648 32617 ERROR cloudkitty self._conn.archive_policy.get(self._archive_policy_name)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/gnocchiclient/v1/archive_policy.py", line 35, in get
2018-01-29 15:57:13.648 32617 ERROR cloudkitty return self._get(self.url + name).json()
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line 37, in _get
2018-01-29 15:57:13.648 32617 ERROR cloudkitty return self.client.api.get(*args, **kwargs)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 288, in get
2018-01-29 15:57:13.648 32617 ERROR cloudkitty return self.request(url, 'GET', **kwargs)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/gnocchiclient/client.py", line 35, in request
2018-01-29 15:57:13.648 32617 ERROR cloudkitty **kwargs)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 192, in request
2018-01-29 15:57:13.648 32617 ERROR cloudkitty return self.session.request(url, method, **kwargs)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/positional/__init__.py", line 108, in inner
2018-01-29 15:57:13.648 32617 ERROR cloudkitty return wrapped(*args, **kwargs)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/keystoneauth1/session.py", line 703, in request
2018-01-29 15:57:13.648 32617 ERROR cloudkitty resp = send(**kwargs)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty File "/openstack/venvs/cloudkitty-16.0.4/lib/python2.7/site-packages/keystoneauth1/session.py", line 777, in _send_request
2018-01-29 15:57:13.648 32617 ERROR cloudkitty raise exceptions.ConnectFailure(msg)
2018-01-29 15:57:13.648 32617 ERROR cloudkitty ConnectFailure: Unable to establish connection to https://14.142.104.140:8041/v1/archive_policy/rating: HTTPSConnectionPool(host='14.142.104.140', port=8041): Max retries exceeded with url: /v1/archive_policy/rating (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6bc1efa890>: Failed to establish a new connection: [Errno 110] Connection timed out',))
2018-01-29 15:57:13.648 32617 ERROR cloudkitty

The problem was solved after changing the code to explicitly specifying adapter_options to include interface option parameters in gnocchi client while the connection was initiated by the following files :

1)cloudkitty/storage/gnocchi/__init__.py
2)cloudkitty/collector/gnocchi.py

A patch request has been submitted in gerrit

bhujay kumar (bhatta)
Changed in cloudkitty:
assignee: nobody → bhujay kumar (bhatta)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cloudkitty (master)

Fix proposed to branch: master
Review: https://review.openstack.org/545512

Changed in cloudkitty:
status: New → In Progress
bhujay kumar (bhatta)
description: updated
bhujay kumar (bhatta)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cloudkitty 8.0.0

This issue was fixed in the openstack/cloudkitty 8.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.