GET /os-quota-sets/{tenant_id} API is failing with SSL exception

Bug #1704798 reported by prashkre
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
jichenjc
Pike
Fix Committed
Medium
jichenjc

Bug Description

In the flow of GET /os-quota-sets/{tenant_id} API, when project_id/tenant_id is being verified by communicating with keystone through secure(https)connection at https://github.com/openstack/nova/blob/master/nova/api/openstack/identity.py#L32, it is failing in certificate validation error as below.

2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity Traceback (most recent call last):
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/nova/api/openstack/identity.py", line 42, in verify_project_id
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity raise_exc=False)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 758, in get
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity return self.request(url, 'GET', **kwargs)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity return wrapped(*args, **kwargs)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity resp = send(**kwargs)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 678, in _send_request
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity raise exceptions.SSLError(msg)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://xxx.xxx.xxx.xxx:5000/v3/projects/0fe761dc32934fc88c390d244acb6971: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)
2017-07-06 01:13:28.134 21365 ERROR nova.api.openstack.identity

prashkre (prashkre)
summary: - GET /os-quota-sets/{tenant_id} API is failing with SSL execption
+ GET /os-quota-sets/{tenant_id} API is failing with SSL exception
Revision history for this message
Matthew Edmonds (edmondsw) wrote :

The problem is that https://github.com/openstack/nova/blob/f6fbfc7ff07b790ef052a759552c69429b3d79c7/nova/api/openstack/identity.py#L32 simply relies on the default for the Session constructor's verify argument (True). We need to provide a configuration option to allow an operator to specify verify=False and verify=<path>.

Revision history for this message
Matthew Edmonds (edmondsw) wrote :

this presumably also impacts POST /flavors/{flavor_id}/action with addTenantAccess or removeTenantAccess actions, since they also call verify_project_id. E.g.: https://github.com/openstack/nova/blob/4d552d6bf61955fa121feedbb4469f4ed0b07cff/nova/api/openstack/compute/flavor_access.py#L99

jichenjc (jichenjc)
Changed in nova:
assignee: nobody → jichenjc (jichenjc)
Revision history for this message
jichenjc (jichenjc) wrote :

we rely on this for nova to communicate with keystone, and this bug also occurs when nova api talking to keystone, so the cred should be able to be reused?

[keystone_authtoken]
auth_type = v3password
auth_url = xxxx
auth_uri = xxx
username = nova
project_name = service
user_domain_id = default
project_domain_id = default
cafile = /data/PKIcerts/certs/cacert.pem
password = xxxx

so if we pass cafile in this definition into code from
sess = session.Session(auth=context.get_auth_plugin())
to
sess = session.Session(auth=context.get_auth_plugin(), cert=CONF.keystone_authtoken.cafile)

should be fine?

Revision history for this message
Matthew Edmonds (edmondsw) wrote :

it's not what that conf section is for, but yeah, I like that idea. We should avoid creating more conf options than we have to, and the cert needed for this verify_project_id would always match the one needed in that section, so creating a new conf option would just be duplicating things unnecessarily.

Revision history for this message
jichenjc (jichenjc) wrote :

ok, maybe need some testing, I remember we can't create https in devstack env, can we ? or could you please help to verify if this change could solve the problem you mentioned?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/485121

Changed in nova:
status: New → In Progress
Changed in nova:
assignee: jichenjc (jichenjc) → Matthew Edmonds (edmondsw)
Changed in nova:
assignee: Matthew Edmonds (edmondsw) → jichenjc (jichenjc)
Changed in nova:
assignee: jichenjc (jichenjc) → Sean Dague (sdague)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/485121
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=905d31dd9715505599b0a2ad123eebef37f606f5
Submitter: Jenkins
Branch: master

commit 905d31dd9715505599b0a2ad123eebef37f606f5
Author: jichenjc <email address hidden>
Date: Sun Jul 9 10:35:23 2017 +0800

    Enable custom certificates for keystone communication

    Nova creates a session back to keystone to verify project ids for
    quota and flavor access APIs. The session that was being created was
    not based on conf options, so it only worked in simple default
    scenarios. This updates the session by using the newly added keystone
    section to utilize keystoneauth1 to manage the session creation, which
    allows for specifying custom site certificates to secure the link
    between Nova and Keystone.

    Change-Id: Ice4b226fdabdfb66e60b61de05ac8f3b37610661
    Closes-Bug: 1704798

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/505546

Matt Riedemann (mriedem)
Changed in nova:
assignee: Sean Dague (sdague) → jichenjc (jichenjc)
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/505546
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c0ff5b33ca42a417a5fa08e5133a27397af08c3d
Submitter: Zuul
Branch: stable/pike

commit c0ff5b33ca42a417a5fa08e5133a27397af08c3d
Author: jichenjc <email address hidden>
Date: Sun Jul 9 10:35:23 2017 +0800

    Enable custom certificates for keystone communication

    Nova creates a session back to keystone to verify project ids for
    quota and flavor access APIs. The session that was being created was
    not based on conf options, so it only worked in simple default
    scenarios. This updates the session by using the newly added keystone
    section to utilize keystoneauth1 to manage the session creation, which
    allows for specifying custom site certificates to secure the link
    between Nova and Keystone.

    Change-Id: Ice4b226fdabdfb66e60b61de05ac8f3b37610661
    Closes-Bug: 1704798
    (cherry picked from commit 905d31dd9715505599b0a2ad123eebef37f606f5)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b1

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.2

This issue was fixed in the openstack/nova 16.0.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.