OS::Heat::SoftwareDeployment failed due SSL certificate verification error

Bug #1482510 reported by neil nie
34
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Triaged
Medium
Unassigned
os-collect-config
Triaged
Medium
Unassigned

Bug Description

When SSL is enforced, the resource OS::Heat::SoftwareDeployment time_out and caused the stack creation failure.

#follow is the log
Aug 5 12:46:37 nnie-example-script-template-server-masn56hswa2n os-collect-config: /opt/stack/venvs/os-collect-config/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Aug 5 12:46:37 nnie-example-script-template-server-masn56hswa2n os-collect-config: InsecurePlatformWarning
Aug 5 12:46:37 nnie-example-script-template-server-masn56hswa2n os-collect-config: 2015-08-05 12:46:37.344 1106 WARNING os_collect_config.cfn [-] [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

In /etc/os-collec-config.conf, the https is set for cfn, like following.
[DEFAULT]
command = os-refresh-config

[cfn]
metadata_url = https://xx.xxx.xxx.xxx:8000/v1/
stack_name = nnie-example-script-template
secret_access_key = 42ee9b138aaa47ea92fec5034ae4eeda
access_key_id = fdba6c55f1944ac08873a0622523de2c
path = server.Metadata

Root cause: in python module requests, the default value for “verify” is set to True, which means to enforce the certificate check. But there is no certificate info in user_data, which is set by heat.

Question: is it expected that the certificate should be installed in VM beforehand? If so, heat needs to add certificate into user_data. Or is it reasonable to relax the certificate check for this type cases?

#source of heat/engine/resources/server.py
        elif self.transport_poll_server_cfn():
            meta['os-collect-config'] = {'cfn': {
                'metadata_url': '%s/v1/' % cfg.CONF.heat_metadata_server_url,
                'access_key_id': self.access_key,
                'secret_access_key': self.secret_key,
                'stack_name': self.stack.name,
                'path': '%s.Metadata' % self.name}
            }
#source of os_collect_config/cfn.py
            try:
                content = self._session.get(
                    url, params=params, headers=headers,
                    verify=CONF.cfn.ca_certificate)

Regards,
Neil

Rico Lin (rico-lin)
Changed in heat:
assignee: nobody → Rico Lin (rico-lin)
Changed in heat:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

This likely requires some coordination between heat and os-collect-config.

The absolute bare minimum would be for os-collect-config.conf [cfn] and [heat] to gain an 'insecure' option, and for heat to populate that from its own /etc/heat/heat.conf [clients_heat] insecure.

Beyond that, /etc/heat/heat.conf [clients_heat] also has options ca_file, cert_file, key_file. We would need security expert input on whether it is appropriate to populate boot user_data with the contents of these files to allow similar options to be set in os-collec-config.conf.

If this is appropriate then the heat user_data cloud-init items can include the contents of ca_file, cert_file, key_file and heat can also populate os-collect-config.conf with the resulting paths.

If this is not appropriate then os-collect-config will need to check for cert paths by convention, and the image building process needs to copy in those cert files.

Revision history for this message
neil nie (neil-nie) wrote :

Steve,

Thanks for the response. I did private change to relax the certificate check for "cfn" to work around this issue. To support other transport types, looks like more changes are required.

About how to add the certificate on VM, I agree we need security experts'comment. Another option is to add the certificate when creating nova server, which is similar to add SSH key.

Regards,
Neil

Changed in os-collect-config:
status: New → Triaged
importance: Undecided → Medium
Rico Lin (rico-lin)
Changed in heat:
assignee: Rico Lin (rico-lin) → nobody
Revision history for this message
Anant Patil (ananta) wrote :

To start with, we atleast need the insecure option, which should be configurable from heat.conf in [cfn] and [heat]. The os-collect-config's conf should then be configured accordingly. A lot of tests are failing downstream simply because of this.

Revision history for this message
Miika Kankare (kuula) wrote :

Any good workarounds or options for this?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/285959

Changed in heat:
assignee: nobody → Ishant Tyagi (ishant-tyagi)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/285962

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (stable/kilo)

Change abandoned by Dave Walker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/285962
Reason:
stable/kilo closed for 2015.1.4

This release is now pending its final release and no freeze exception has
been seen for this changeset. Therefore, I am now abandoning this change.

If this is not correct, please urgently raise a thread on openstack-dev.

More details at: https://wiki.openstack.org/wiki/StableBranch

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/360025

Changed in heat:
assignee: Ishant Tyagi (ishant-tyagi) → Abhishek Chanda (abhishek-i)
Changed in heat:
assignee: Abhishek Chanda (abhishek-i) → Steve Baker (steve-stevebaker)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Steve Baker (<email address hidden>) on branch: master
Review: https://review.openstack.org/360025
Reason: lets do this here https://review.openstack.org/#/c/285959/

Changed in heat:
assignee: Steve Baker (steve-stevebaker) → Rico Lin (rico-lin)
Zane Bitter (zaneb)
Changed in heat:
status: In Progress → Triaged
Revision history for this message
Zane Bitter (zaneb) wrote :

The proposed patch https://review.openstack.org/285959 lacks a way to get the *content* of the CA file to the booted server. The patches to o-c-c https://review.openstack.org/284725 and tripleo-image-elements https://review.openstack.org/285957 got abandoned for want of updates from the author (who no longer works on OpenStack).

All of this only affects the cfn collector, which is probably not the most interesting one anyway.

Zane Bitter (zaneb)
Changed in heat:
assignee: Rico Lin (rico-lin) → nobody
Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Zane Bitter (<email address hidden>) on branch: master
Review: https://review.opendev.org/285959
Reason: This is not close to a workable patch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.