[Upgrade] Rollback of 5.1.1 upgrade wasn't done right - ostf and fuelclient failed with authorization failure

Bug #1392393 reported by Andrey Sledzinskiy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Andrey Sledzinskiy
5.1.x
Fix Released
Medium
Andrey Sledzinskiy

Bug Description

{

    "build_id": "2014-09-17_21-40-34",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "11",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d",
    "production": "docker",
    "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
    "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "release_versions": {
        "2014.1.1-5.1": {
            "VERSION": {
                "build_id": "2014-09-17_21-40-34",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "11",
                "api": "1.0",
                "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d",
                "production": "docker",
                "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
                "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1",
                "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"
            }
        }
    },
    "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"

}

Upgrade script version
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1.1"
  api: "1.0"
  build_number: "3"
  build_id: "2014-11-12_21-56-33"
  astute_sha: "702af3db6f5bca92525bc8322d7d5d7675ec857e"
  fuellib_sha: "7d711bb3d2572d55463cd38b9df3ad7a20a34262"
  ostf_sha: "64cb59c681658a7a55cc2c09d079072a41beb346"
  nailgun_sha: "bbc9dfe78a0c33040dcd16de9a40a3491788719c"
  fuelmain_sha: "e5e534ade6f3765a87feee3d44d39df68ae28f80"

Steps:
1. Deploy new 5.1.1 env - Simple, Neutron Gre, CentOS, 1 controller, 2 compute+cinder
2. Modify 5.1.1 upgrade script to fail
3. Run upgrade

Expected - rollback was successfull, everything is operational
Actual - rollback passed, but there are problems with authorization in fuel client and ostf server is unavailable

[root@nailgun ~]# fuel nodes
Traceback (most recent call last):
  File "/usr/bin/fuel", line 9, in <module>
    load_entry_point('fuelclient==0.2', 'console_scripts', 'fuel')()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 105, in wrapper
    handle_exceptions(exc)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 177, in main
    parser.parse()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 93, in parse
    actions[parsed_params.action].action_func(parsed_params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/base.py", line 58, in action_func
    method(params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/node.py", line 243, in list
    node_collection = NodeCollection.get_all()
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/node.py", line 152, in get_all
    return cls(Node.get_all())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 68, in get_all
    return map(cls.init_with_data, cls.get_all_data())
  File "/usr/lib/python2.6/site-packages/fuelclient/objects/base.py", line 64, in get_all_data
    return cls.connection.get_request(cls.class_api_path)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 151, in get_request
    request.add_header('X-Auth-Token', self.auth_token)
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 67, in auth_token
    if self.auth_required:
  File "/usr/lib/python2.6/site-packages/fuelclient/client.py", line 76, in auth_required
    request = urllib2.urlopen(''.join([self.api_root, 'version']))
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1163, in do_open
    r = h.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine

But after some period of time fuel client started working

Logs are attached

Tags: upgrade
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Igor Kalnitsky (ikalnitsky)
Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

Very strange and not enough info. It looks like there's some delay between fuel_upgrade script is done and all services are running. Need to investigate, but it's definitely not a Critical, because some timeout (1min) is enough to avoid this issue.

Changed in fuel:
importance: Critical → Medium
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

Need to modify test to wait until all services are ready

Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Andrey Sledzinskiy (asledzinskiy)
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 5.1.1 → 6.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/135310

Changed in fuel:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-main (stable/5.1)

Reviewed: https://review.openstack.org/135310
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=a6d5f5ae9a9c2a0adaed527b729eb0cab6c2a2d5
Submitter: Jenkins
Branch: stable/5.1

commit a6d5f5ae9a9c2a0adaed527b729eb0cab6c2a2d5
Author: asledzinskiy <email address hidden>
Date: Tue Nov 18 16:37:18 2014 +0200

    Add wait until nailgun will be operable to rollback tests

    -In case of rollback some services need time to start
    so we need to wait in tests for these services to start

    Change-Id: I34ae6388fe023cad929c1c039d4f7643658548be
    Closes-Bug: #1392393

Revision history for this message
Mike Scherbakov (mihgen) wrote :

Colleagues,
I see a big issue with this bug report. It doesn't target 5.1.1, but fix was merged to stable/5.1. More over, there was no patch to master first, so it's in violation of https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Backport_bugfixes_to_stable_release_series. Please fix.

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

Sorry , I put wrong closes-bug in commit to master - it's already merged to master
https://review.openstack.org/#/c/136265/

Changed in fuel:
status: Confirmed → Fix Committed
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.