[Tempest] [SSL] 2 tests for object storage failed with ssl configuration

Priority changed to High because two Tempest tests failed for any configurations with SSL and it looks like we don't have workaround for the issue.

Assigned to MOS Ceph team because we reproduced the issue but we don't know the root of the issue yet. It looks like we need someone with experience in object storage API and Swift.

Need more time to understand the root cause. This may even be a problem of Tempest configuration. Moving to MU.

Hello all,

Please let me to provide some explanation, related with the points listed in this bug report.

1) The following tempest tests for object storage testing failed in configuration with Swift:
             test_web_listing_css
             test_web_index
    Please find example of the trace here [1]
2) All tempest test for object storage testing failed in configuration with Ceph(RGW)
    Please find example of the trace here [2]

For more details please take a look into tempest reports:
1) swift https://drive.google.com/a/mirantis.com/file/d/0B68GLJA1XY9CMWFaaWFxVUpzWEE/view?usp=sharing
2) ceph https://drive.google.com/a/mirantis.com/file/d/0B68GLJA1XY9CNmx3SkJ3amxPd1E/view?usp=sharing

[1] http://paste.openstack.org/show/485936/
[2] http://paste.openstack.org/show/485939/

Any updates?

You can override the 'shouldfail' tests in /home/developer/mos-tempest-runner/shouldfail/shouldfail.yaml
/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/__init__.py:28: DeprecationWarning: tempest-lib is deprecated for future bug-fixes and code changes in favor of tempest. Please change your imports from tempest_lib to tempest.lib
  DeprecationWarning)
Non-zero exit code (2) from test listing.
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
--- import errors ---
Failed to import test module: tempest.test_discover.test_discover
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
    __import__(name)
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/test_discover.py", line 18, in <module>
    from tempest.test_discover import plugins
  File "/home/developer/mos-tempest-runner/tempest/tempest/test_discover/plugins.py", line 20, in <module>
    from tempest_lib.common.utils import misc
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/utils/misc.py", line 19, in <module>
    from oslo_log import log as logging
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/log.py", line 49, in <module>
    from oslo_log import _options
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_log/_options.py", line 38, in <module>
    help='If set to true, the logging level will be set to '
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/oslo_config/cfg.py", line 965, in __init__
    super(BoolOpt, self).__init__(name, type=types.Boolean(), **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mutable'

I stop a test right before request using IPython and execute same request from console, using swift client:

swift --os-project-name tempest-StaticWebTest-1175394924 --os-username tempest-StaticWebTest-777055592 --os-password 'K8*KRpQh1jOyBVc' list tempest-TestContainer-951268031

this gives me expected object name "tempest-TestObject-284701657".

Same time request from tempest fail. Before fail it print a lot of warnings:

/home/rally/.rally/tempest/for-deployment-83d24637-32ce-4576-9ba8-e54ec2068072/.venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

which might be a reason for issue. In any case the reason is somewhere in tempest/rally.

I check this issue on my env with ssl.
I stop a test before fail and execute request from console, using swift client and curl:
http://paste.openstack.org/show/508362/
So that the root cause - swift container doesn't available via https

on env without ssl curl return "test" - expected result for test

Moving to 9.0-updates after communication with Konstantin. It's too late trying fix it in 9.0

I looked into a situation where these two tests, test_web_index and test_web_listing_css, fail when running through HAproxy that terminates SSL. Here's what I found.

At a certain point, Tempest does GET on /v1/AUTH_fa4d5382f43b41d59f0fb51ef330077b/tempest-TestContainer-2075955620 without a trailing slash. The staticweb returns a 301 redirect with Location: http://10.0.0.101:13808/v1/AUTH_fa4d5382f43b41d59f0fb51ef330077b/tempest-TestContainer-2075955620/

Basically all it wants is to add the trailing slash. It does so by returning HTTPMovedPermanently(location=env['PATH_INFO']+'/'). The path_info at this point is just a path. Before the Response.__call__() returns, swob prepends the netloc to the location, and mistakenly uses the http: scheme.

The urllib3 then uses the Location from the redirect and attempts to talk HTTP to HAproxy that is listening with SSL. HApxoxy resets the connectin. The process repeats until urllib3 connection exhausts its retries.

The right fix needs the scheme to be correct in responses. It can either be set by staticweb (tempauth does that), or extracted from wsgi.url_scheme (which requires magic and may be impossible).

I discussed this on #openstack-swift and surprisingly, Clay et.al. were in favour of additional configuration. I came around to this approach too, when I found that my stunnel does not provide any kind of header at all and thus the automatic guessing is impossible.

Here's a proposed patch:
 https://review.openstack.org/372809

Note that this still needs OOO or other orchestration to set url_base. With the patch above, one can set url_base to "https://", in which case hostname is taken from Host: header as usual.

ETA: 08/03/2016

AFAIK this bug doesn't feet into MU requirements. We have never see it in production, the only failing code, is a unit-test. So I put 'wont-fix' for 8.0 and 7.0

Fix Committed for 9.2, the fix is https://review.fuel-infra.org/#/c/29499/

if you need access to env and steps to reproduce - contact me
reproduced again on MOS 9.2 snapshot #744

More time is needed to get this issue fixed - retargeted to 9.3

sla1 for 9.0-updates

we need configure base_url in /etc/swift/proxy-server.conf for mos 10.
If 9.x release (stable\mitaka) had all releated patches - need create the same patch(If not - create backports before)

I can't reproduce this error on latest 9.x deploy.(test failed with another error)
After debug issue we will prepare fix for 9.2

fixes for master, ocata, newton - https://review.openstack.org/#/q/topic:bug/1537071

https://review.openstack.org/#/c/442600/ - fix for mitaka

Verified on 9.2 + mu1 updates.

Environment:
3 controllers + 1 compute node with enabled TLS/HTTPS.

Actual results:
The following tempest tests are passed after the fix (http://paste.openstack.org/show/602792/):
test_web_index
test_web_listing_css

This issue was fixed in the openstack/fuel-library ocata-eol release.