Artifact creation in GLARE using public endpoint that is under SSL failed with CommunicationError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:765)

Related fix proposed to branch: master
Review: https://review.openstack.org/327621

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/327622

Update:
We are found, that GLARE can't handle SSL termination.

Proof:

root@node-1:~# curl -k https://public.fuel.local:9494

{"versions": [{"status": "EXPERIMENTAL", "id": "v0.1", "links": [{"href": "http://public.fuel.local:9494/v0.1/", "rel": "self"}]}]}

In response over https should be a valid https endpoint, not http.

relevant patches for other services:
https://review.openstack.org/#/c/64142/ heat
https://review.openstack.org/#/c/234134/ murano

Glare team, could you please prepare the fix for the issue based on fixes for Murano and Heat?

Folks, we have new update.
After digging an environment, we found, that SSL termination - this is not exactly true problem.

Finally, we found, that following option in 081-glance-glare.cfg of HAPROXY 'option httpclose' leads to errors. After commenting it, we are able to see passed OSTF for Murano/Glare.

What does this option do?

@Timur, we were investigating what influence will be in case of this option usage / non usage. In fact this option leads to session closing in case of timeouts, etc. - so it's important. The thing is that this issue cannot be easily reproduced, and exactly the same scenario run manually is working perfectly. Probably there is an issue in the either in openstack clients or their usage - this is something Victor is trying to find this out.

Change abandoned by tatyana-leontovich (<email address hidden>) on branch: master
Review: https://review.openstack.org/327621

Fix proposed to branch: master
Review: https://review.openstack.org/329641

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/327622
Reason: Incorrect root cause for a bug

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/329645

During debugging, we are found, that not only GLARE affected here.
If we are change 'dummy_data' string with generated 'data' variable (put into image smth significant) in this line https://github.com/openstack/fuel-ostf/blob/master/fuel_health/glancemanager.py#L77 , we catch the same error.

Also, it looks like that only glance using PUT method in these tests. So, we are can't verify, that this is global problem.

Possible solution with HTTP_PROXY variable will not work (we are tried to verify it, but bvt was failed. Without HTTPS_PROXY it will not work).

I still think, that this happens due httpclose parameter in haproxy, but I've talked with Alex Shultz about it, and he voted against this change.

I can't find, what exactly component of the chain lead to this problem.

Release notes:
Murano OSTF tests fail for deployments with enabled TLS feature. It doesn't affect Murano functionality, it affects only OSTF tests because these tests use proxy service on OpenStack controller nodes to work with OpenStack API.

Sustaining team, can you please take a look on this issue?

Summary:
* problem is not in murano and murano-tests. Not in glance.
* possible problem in haproxy, ostf-framework or glance client.

Feel free to reassign to other teams if reason not in your area.

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: master
Review: https://review.openstack.org/329641
Reason: Incorrect root cause

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/329645
Reason: Incorrect root cause

Victor, could you please update the status of the issue.

Max, status if the issue wasn't changed.

Request that fails:

glanceclient.common.http: DEBUG: curl -g -i -X PUT -H 'Accept-Encoding: gzip, deflate' -H 'Accept: ⁠⁠⁠/⁠⁠⁠' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}8741e14bac7d38329763df01c0b20a2b23a4542c' -H 'Content-Type: application/octet-stream' -k --cert None --key None https://public.fuel.local:9494/v0.1/artifacts/murano/v1/e5f00dbc-3cd6-41da-90ea-ca114b30c1e6/archive

CommunicationError: Error finding address for https://public.fuel.local:9494/v0.1/artifacts/murano/v1/e5f00dbc-3cd6-41da-90ea-ca114b30c1e6/archive: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:765)

I have tried to catch this request using tcpdump on controller, but it never comes to port 9494.

Glance team, can you check?

Fix proposed to branch: master
Review: https://review.openstack.org/378475

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/378475

SSL tunings for haproxy did not help. The only solution seems is to set option server-http-close for this backend.

Blocked by https://bugs.launchpad.net/fuel/+bug/1634550

*Probably blocked.
I'll prepare a patch without murano envolved.

Fix proposed to branch: master
Review: https://review.openstack.org/408735

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/409149

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/409150

Reviewed: https://review.openstack.org/408735
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0331b59169b773bca6f184119f02e08b89515c11
Submitter: Jenkins
Branch: master

commit 0331b59169b773bca6f184119f02e08b89515c11
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633

Reviewed: https://review.openstack.org/409149
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9f85768baeb4c64bcd22fa8b522324030baa1892
Submitter: Jenkins
Branch: stable/newton

commit 9f85768baeb4c64bcd22fa8b522324030baa1892
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633
    (cherry-picked from 0331b59169b773bca6f184119f02e08b89515c11)

Reviewed: https://review.openstack.org/409150
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=047e301a5c9b3239b3679207289bb105cd8e71c8
Submitter: Jenkins
Branch: stable/mitaka

commit 047e301a5c9b3239b3679207289bb105cd8e71c8
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633
    (cherry-picked from 0331b59169b773bca6f184119f02e08b89515c11)

All OSTFs are passed. Verified on:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-nailgun-9.0.0-1.mos8924.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-notify-9.0.0-1.mos8680.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 python-packetary-9.0.0-1.mos160.noarch
 nailgun-mcagents-9.0.0-1.mos784.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-utils-9.0.0-1.mos8680.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8924.noarch
 fuel-mirror-9.0.0-1.mos160.noarch
 fuel-openstack-metadata-9.0.0-1.mos8924.noarch
 rubygem-astute-9.0.0-1.mos784.noarch
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-misc-9.0.0-1.mos8680.noarch
 fuel-ui-9.0.0-1.mos2854.noarch
 fuel-library9.0-9.0.0-1.mos8680.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-migrate-9.0.0-1.mos8680.noarch
 python-fuelclient-9.0.0-1.mos364.noarch
 fuel-9.0.0-1.mos6359.noarch

Related fix proposed to branch: master
Change author: Mariia Zlatkova <email address hidden>
Review: https://review.fuel-infra.org/30303

Reviewed: https://review.fuel-infra.org/30303
Submitter: Olena Logvinova <email address hidden>
Branch: master

Commit: dc4cfe1141c0237b04d32ddea8c33b09ac0f854d
Author: Mariia Zlatkova <email address hidden>
Date: Thu Feb 2 13:50:37 2017

[RN-9.2] Fuel resolved and known issues

Change-Id: Idb919f92b981eee0f2cb48618dde243e4582ee5b
Related-Bug: #1590633
Related-Bug: #1625293
Related-Bug: #1561092
Related-Bug: #1619341
Related-Bug: #1563465
Related-Bug: #1628500
Related-Bug: #1593277
Related-Bug: #1628940
Related-Bug: #1658952

Related fix proposed to branch: stable/9.2
Change author: Mariia Zlatkova <email address hidden>
Review: https://review.fuel-infra.org/30423

Reviewed: https://review.fuel-infra.org/30423
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.2

Commit: c040581a57fac1dfeaed44952359f21963216d62
Author: Mariia Zlatkova <email address hidden>
Date: Thu Feb 2 14:03:50 2017

[RN-9.2] Fuel resolved and known issues

Change-Id: Idb919f92b981eee0f2cb48618dde243e4582ee5b
Related-Bug: #1590633
Related-Bug: #1625293
Related-Bug: #1561092
Related-Bug: #1619341
Related-Bug: #1563465
Related-Bug: #1628500
Related-Bug: #1593277
Related-Bug: #1628940
Related-Bug: #1658952
(cherry picked from commit dc4cfe1141c0237b04d32ddea8c33b09ac0f854d)

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

Can't verify due to https://bugs.launchpad.net/fuel/+bug/1653073