Fuel for OpenStack

[ostf] Use proxy on primary controller for keystone client

  1. Series 5.1.x
  2. Bug #1355897
Bug #1355897 reported by Artem Panchenko
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Tatyanka
Fuel for OpenStack 6.1
5.1.x
Fix Committed
Medium
Tatyanka
Fuel for OpenStack 5.1.1-updates
6.0.x
Fix Committed
Medium
Tatyanka
Fuel for OpenStack 6.0-updates

Bug Description

In case of network connectivity problems on some controller other controller nodes in HA cluster mark it as offline, but if admin network works fine there, OSTF still uses proxy started on that node and returns 500 error:

AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request. (1047, 'Unknown command') (HTTP 500)

This issue can be reproduced in the following way:

1. Deploy HA environment with neutron
2. Login on controller which has the smallest id
3. Shut down management interface (emulate network connectivity failure e.g.: ovs-vsctl del-port br-mgmt br-mgmt--br-eth0)
4. Run health checks. All checks will fail with authorization failure

I propose to add an additional step there:

https://github.com/stackforge/fuel-ostf/blob/stable/5.0/fuel_health/config.py#L639

if len(self.compute.online_controllers) > 1 then check out which controller is primary and use it as proxy server. IMHO a method which should detect an id of primary controller can execute 'pcs status' command on each controller node and return ID of node, which has partition with quorum on most of controllers.

Artem Panchenko (apanchenko-8)
Changed in fuel:
milestone: none → 5.1
Nastya Urlapova (aurlapova)
Changed in fuel:
importance: Undecided → High
Revision history for this message
Dima Shulyak (dshulyak) wrote :

I strongly disagree with provided approach..
Ostf should not go to each node and execute pcs status or any other shell command to find out primary controller.

If we need this kind of functionality lets register blueprint for 6.0, where we will provide a way for node
to report this kind of data to nailgun. And ostf will fetch this data from nailgun api.

Nastya Urlapova (aurlapova)
Changed in fuel:
milestone: 5.1 → 6.0
importance: High → Medium
Nastya Urlapova (aurlapova)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Dima Shulyak (dshulyak) wrote :

Artem, what if we will use another method for detecting working controller,
we can try to authorize on each of those controllers, and stick with one that is working

Will it cover your case?

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

Dima,

yeah, I believe it will be enough to avoid the issue I reported. But I'm not sure

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 6.0 → 6.1
Mike Scherbakov (mihgen)
tags: added: release-notes
Revision history for this message
Mike Scherbakov (mihgen) wrote :

Folks,
it seems to me that this issues blocks proper automated testing of HA failover. Because of this, I think it has be higher priority issue than it is now.

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Confirmed on CI for 5.1.1: http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.ha_neutron_destructive/48/ , system test 'ha_disconnect_controllers'

{"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "auth_required": true, "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}}}, "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}

Changed in fuel:
milestone: 6.1 → none
no longer affects: fuel
Dennis Dmitriev (ddmitriev)
Changed in fuel:
milestone: none → 5.1.2
milestone: 5.1.2 → 6.1
assignee: nobody → Fuel Python Team (fuel-python)
importance: Undecided → Medium
status: New → Confirmed
no longer affects: fuel/6.1.x
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ostf (master)

Fix proposed to branch: master
Review: https://review.openstack.org/145016

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Tatyanka (tatyana-leontovich)
status: Confirmed → In Progress
Tatyanka (tatyana-leontovich)
Changed in fuel:
milestone: 6.0.1 → 6.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ostf (master)

Reviewed: https://review.openstack.org/145016
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=0b75defeb24e301e412ef28220c9b0ff50c5be16
Submitter: Jenkins
Branch: master

commit 0b75defeb24e301e412ef28220c9b0ff50c5be16
Author: Tatyana Leontovich <email address hidden>
Date: Mon Jan 5 17:32:58 2015 +0200

    Add verification if proxy works

    In case if controller is online, but connectivity
    on it is broken we receive false positive
    message like "Can not set proxy". To fix this:
    * Import of keystoneclient was added into config.py
    * find_proxy method was added
    * set proxy changed

    Change-Id: I77d3c0ec12bc071a7e490ddb3ea24cd2f7fea543
    Closes-Bug: #1355897

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ostf (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/161233

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ostf (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/161251

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ostf (stable/6.0)

Reviewed: https://review.openstack.org/161233
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=e15d4ebc7da62cd6793e1fcd7ce9e8f2504f788e
Submitter: Jenkins
Branch: stable/6.0

commit e15d4ebc7da62cd6793e1fcd7ce9e8f2504f788e
Author: Tatyana Leontovich <email address hidden>
Date: Wed Mar 4 16:30:04 2015 +0200

    6.0 Add verification if proxy works

    In case if controller is online, but connectivity
    on it is broken we receive false positive
    message like "Can not set proxy". To fix this:
    * Import of keystoneclient was added into config.py
    * find_proxy method was added
    * set proxy changed

    Change-Id: I77d3c0ec12bc071a7e490ddb3ea24cd2f7fea543
    Closes-Bug: #1355897

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ostf (stable/5.1)

Reviewed: https://review.openstack.org/161251
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=9d4392fb97252c84dee7d29ee33c706d78b5bbef
Submitter: Jenkins
Branch: stable/5.1

commit 9d4392fb97252c84dee7d29ee33c706d78b5bbef
Author: Tatyana Leontovich <email address hidden>
Date: Wed Mar 4 16:30:04 2015 +0200

    5.1 Add verification if proxy works

    In case if controller is online, but connectivity
    on it is broken we receive false positive
    message like "Can not set proxy". To fix this:
    * Import of keystoneclient was added into config.py
    * find_proxy method was added
    * set proxy changed

    Change-Id: I77d3c0ec12bc071a7e490ddb3ea24cd2f7fea543
    Closes-Bug: #1355897

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

{"build_id": "2015-03-26_09-08-29", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "231", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-26_09-08-29", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "231", "api": "1.0", "nailgun_sha": "7f0e0af1f54db840230745ee4f7aec6824dac9b9", "production": "docker", "python-fuelclient_sha": "e5e8389d8d481561a4d7107a99daae07c6ec5177", "astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "345a98b34dd0cd450a45d405ac47a6a9fa48b6d8"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "7f0e0af1f54db840230745ee4f7aec6824dac9b9", "production": "docker", "python-fuelclient_sha": "e5e8389d8d481561a4d7107a99daae07c6ec5177", "astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "345a98b34dd0cd450a45d405ac47a6a9fa48b6d8"} verified for 6.1

for other version move to fix committed according to comments above

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.