Juniper Openstack

ssl_adapter.py doesn't work on RDO

Series r3.0
Bug #1528851

Bug #1528851 reported by Martin Millnert on 2015-12-23

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.0	Fix Committed	Undecided	Unassigned	Juniper Openstack r3.0.3.0
Trunk	Fix Committed	Undecided	Unassigned	Juniper Openstack r3.1.0.0-fcs

Bug Description

A system running RDO Kilo (on EL 7) has the following two relevant packages installed:
python-requests --> /usr/lib/python2.7/site-packages/requests
python-urllib3 --> /usr/lib/python2.7/site-packages/urllib3

The SSL Adapter in contrail-controller/src/config/common/ssl_adapter.py gets installed as /usr/lib/python2.7/site-packages/cfgm_common/ssl_adapter.py .
When using SSL against Keystone v3 from the contrail-api, that uses this code, on a system such as the above, there will be a class loader type bug, due to a incorrect order of imports in ssl_adapter.py.

First, the bug, using the attached test_case.py, with some added debugging within /usr/lib/python2.7/site-packages/urllib3/connectionpool.py (also attached in next comment):

[root@dev14 ~]# ./testcase.py
('1: ', <bound method apa.openks of <__main__.apa object at 0x261ba50>>)
('minapa._ks: ', None)
('2: ', <bound method apa.openks of <__main__.apa object at 0x261ba50>>)
('minapa._ks: ', None)
'openks definition'
('retval: ', None)
('minapa._ks: ', <requests.sessions.Session object at 0x261ba90>)
HTTP get
xxx: connectionpool: timeout: Timeout(connect=3, read=3, total=None)
xxx: connectionpool TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
xxx: connectionpool: timeout: Timeout(connect=3, read=3, total=None)
xxx: connectionpool TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=3, read=3, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
<Response [200]>
HTTPS get
xxx: connectionpool: timeout: Timeout(connect=7, read=7, total=None)
xxx: connectionpool TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout: timeout: Timeout(connect=7, read=7, total=None)
_get_timeout TYPE: timeout: <class 'requests.packages.urllib3.util.timeout.Timeout'>
_get_timeout TYPE: testtimeout: <class 'urllib3.util.timeout.Timeout'>
Traceback (most recent call last):
  File "./testcase.py", line 46, in <module>
    main()
  File "./testcase.py", line 41, in main
    resp = minapa._ks.get("https://google.com", timeout=7, verify=False)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 478, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 466, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 574, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 542, in urlopen
    timeout_obj = self._get_timeout(timeout)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 304, in _get_timeout
    return Timeout.from_float(timeout)
  File "/usr/lib/python2.7/site-packages/urllib3/util/timeout.py", line 153, in from_float
    return Timeout(read=timeout, connect=timeout)
  File "/usr/lib/python2.7/site-packages/urllib3/util/timeout.py", line 95, in __init__
    self._connect = self._validate_timeout(connect, 'connect')
  File "/usr/lib/python2.7/site-packages/urllib3/util/timeout.py", line 126, in _validate_timeout
    "int or float." % (name, value))
ValueError: Timeout value connect was Timeout(connect=7, read=7, total=None), but it must be an int or float.

From the debugging, we see that the code around https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L297 is evaluating a locally instantiated Timeout object, per the import on https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L47 , to be loaded via the /usr/lib/python2.7/site-packages/urllib3 path rather than through the symlink within python-requests ( /usr/lib/python2.7/site-packages/requests/packages/urllib3 ).

This happens due to incorrect order of import statements in contrail-controller/src/config/common/ssl_adapter.py
This is unfixed in master (thereby, by policy, in all versions?)

A patch will be submitted shortly.

Tags:

Revision history for this message

Martin Millnert (r-martin-5) wrote on 2015-12-23:

simple standalone test case for the bug Edit (1.1 KiB, text/x-python)

information type:

Proprietary → Public

Revision history for this message

Martin Millnert (r-martin-5) wrote on 2015-12-23:

/usr/lib/python2.7/site-packages/urllib3/connectionpool.py with debugging Edit (30.0 KiB, text/x-python)

/usr/lib/python2.7/site-packages/urllib3/connectionpool.py with some simple debug lines to demonstrate the error more clearly

Revision history for this message

Martin Millnert (r-martin-5) wrote on 2015-12-23:

fix for contrail-controller/src/config/common/ssl_adapter.py Edit (1.2 KiB, text/x-python)

The fix is the following ssl_adapter.py, also being submitted in git-review.
Additionally, PEP8 fixed the source code.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-12-23: [Review update] master

Review in progress for https://review.opencontrail.org/15996
Submitter: Martin Millnert (<email address hidden>)

Revision history for this message

Martin Millnert (r-martin-5) wrote on 2015-12-23:

This may be moot considering contrail-controller/src/config/vnc_openstack/vnc_openstack/__init__.py has moved away from using python-requests natively for keystone v3, into using keystone client, which alleviates this issue.

The fix however is applicable for R2.22-dev, any other branches that intends to support SSL but don't use the keystoneclient for the keystone v3 client operations.

In that case it should also be noted, that the code at https://github.com/Juniper/contrail-controller/blob/R2.22-dev/src/config/vnc_openstack/vnc_openstack/__init__.py#L360 is incorrect python code, as per the below error:

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vnc_openstack/__init__.py", line 398, in _ksv3_process_request
resp = self._ks.get(uri, headers, verify=False)
TypeError: get() takes exactly 2 arguments (4 given)

The code snippet:
resp = self._ks.get(uri,headers,verify=False)
becomes a call as:
self._ks.get(self, uri, headers, verify=False)
And this does not match requests.get: http://docs.python-requests.org/en/latest/api/#main-interface
A quick fix is for example:
resp = self._ks.get(url=uri, params=headers, verify=False)

Nischal Sheth (nsheth) on 2015-12-31

tags:	added: config
tags:	removed: contrail-control

Revision history for this message

Siva Gurumurthy (sgurumurthy) wrote on 2016-05-17: FW: SMLite job https://jenkins.opencontrail.org/job/ci-contrail-fabric-utils-systest-ubuntu14-kilo/12/ : END rc=-1

Hi Sachin,
Need your help to nail down this issue.
A lot of reviews like this are failing at the ‘verify contrail’ stage with the error ‘No BGP configuration for self’.

== Contrail Control ==
supervisor-control: active
supervisor-control: ['contrail-control active pid 1984, uptime 0:11:44', 'contrail-control-nodemgr active pid 1983, uptime 0:11:44', 'contrail-dns active pid 1985, uptime 0:11:44', 'contrail-named active pid 1986, uptime 0:11:44', '']
contrail-control: DEFAULT/S.http_server_port not present
contrail-control initializing (No BGP configuration for self)pid 1984, uptime 0:11:44
contrail-control-nodemgr: DEFAULT/S.http_server_port not present

This is for the smlite kilo job.
Can you please take look at the failure?

Thanks,
Siva.

On 5/17/16, 6:59 AM, "opencontrail-ci-admin" <email address hidden> wrote:

>SMLite provisioning job has completed
>exit_code=-1
>
>Slave VM: ci-oc-slave-ubuntu14-10-84-34-150
>Workspace: /home/jenkins/workspace/ci-contrail-fabric-utils-systest-ubuntu14-kilo
>Review: https://review.opencontrail.org/20277
>Job: https://jenkins.opencontrail.org/job/ci-contrail-fabric-utils-systest-ubuntu14-kilo/12/
>
>

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-22: A change has been merged

Reviewed: https://review.opencontrail.org/15996
Committed: http://github.org/Juniper/contrail-controller/commit/cf6c7fa6f0092115bff9dff465a320255ca6b0b5
Submitter: Zuul
Branch: master

commit cf6c7fa6f0092115bff9dff465a320255ca6b0b5
Author: Martin Millnert <email address hidden>
Date: Wed Dec 23 14:19:48 2015 +0100

ssl_adapter.py: Properly fix order of imports

A system running RDO Kilo (on EL 7) has the following two relevant
packages installed:
python-requests --> /usr/lib/python2.7/site-packages/requests
python-urllib3 --> /usr/lib/python2.7/site-packages/urllib3

The SSL Adapter in contrail-controller/src/config/common/ssl_adapter.py
gets installed as
/usr/lib/python2.7/site-packages/cfgm_common/ssl_adapter.py .

When using SSL against Keystone v3 from the contrail-api, that uses this
code, on a system such as the above, there will be a class loader type
bug, due to a incorrect order of imports in ssl_adapter.py.

The intended ImportError doesn't happen, since the native urllib3
actually exists, which leads to a mixup of name space between
the Pool Manager and the requests code, when executing.

The proper fix is to reverse the order of the imports, and have the
fallback become the base case.
Any non-RDO system will trigger the ImportError, and load the module
the regular way. And an RDO-based system works, as well.

Change-Id: Iba275c3521b1cdde5595fa443b449fdd962286fb
Closes-Bug: 1528851

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-23: [Review update] R3.0

Review in progress for https://review.opencontrail.org/21384
Submitter: Senthilnathan Murugappan (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-06-24: A change has been merged

#11

Reviewed: https://review.opencontrail.org/21384
Committed: http://github.org/Juniper/contrail-controller/commit/d7898b6033030be3f6b206ac6875f1cd6e60a9a8
Submitter: Zuul
Branch: R3.0

commit d7898b6033030be3f6b206ac6875f1cd6e60a9a8
Author: Martin Millnert <email address hidden>
Date: Wed Dec 23 14:19:48 2015 +0100

ssl_adapter.py: Properly fix order of imports

A system running RDO Kilo (on EL 7) has the following two relevant
packages installed:
python-requests --> /usr/lib/python2.7/site-packages/requests
python-urllib3 --> /usr/lib/python2.7/site-packages/urllib3

The SSL Adapter in contrail-controller/src/config/common/ssl_adapter.py
gets installed as
/usr/lib/python2.7/site-packages/cfgm_common/ssl_adapter.py .

The intended ImportError doesn't happen, since the native urllib3
actually exists, which leads to a mixup of name space between
the Pool Manager and the requests code, when executing.

Change-Id: Iba275c3521b1cdde5595fa443b449fdd962286fb
Closes-Bug: 1528851

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.