[Upgrade] Rollback of 5.1.1 upgrade wasn't done right - ostf and fuelclient failed with authorization failure
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Medium
|
Andrey Sledzinskiy | ||
5.1.x |
Fix Released
|
Medium
|
Andrey Sledzinskiy |
Bug Description
{
"build_id": "2014-09-
"ostf_sha": "64cb59c681658a
"build_number": "11",
"auth_
"api": "1.0",
"nailgun_sha": "eb8f2b358ea4bb
"production": "docker",
"fuelmain_sha": "8ef433e939425e
"astute_sha": "f5fbd89d1e0e1f
"feature_
"mirantis"
],
"release": "5.1",
"release_
],
}
}
},
"fuellib_sha": "d9b16846e54f76
}
Upgrade script version
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "5.1.1"
api: "1.0"
build_number: "3"
build_id: "2014-11-
astute_sha: "702af3db6f5bca
fuellib_sha: "7d711bb3d2572d
ostf_sha: "64cb59c681658a
nailgun_sha: "bbc9dfe78a0c33
fuelmain_sha: "e5e534ade6f376
Steps:
1. Deploy new 5.1.1 env - Simple, Neutron Gre, CentOS, 1 controller, 2 compute+cinder
2. Modify 5.1.1 upgrade script to fail
3. Run upgrade
Expected - rollback was successfull, everything is operational
Actual - rollback passed, but there are problems with authorization in fuel client and ostf server is unavailable
[root@nailgun ~]# fuel nodes
Traceback (most recent call last):
File "/usr/bin/fuel", line 9, in <module>
load_
File "/usr/lib/
handle_
File "/usr/lib/
return func(*args, **kwargs)
File "/usr/lib/
parser.parse()
File "/usr/lib/
actions[
File "/usr/lib/
method(params)
File "/usr/lib/
node_collection = NodeCollection.
File "/usr/lib/
return cls(Node.get_all())
File "/usr/lib/
return map(cls.
File "/usr/lib/
return cls.connection.
File "/usr/lib/
request.
File "/usr/lib/
if self.auth_required:
File "/usr/lib/
request = urllib2.
File "/usr/lib64/
return _opener.open(url, data, timeout)
File "/usr/lib64/
response = self._open(req, data)
File "/usr/lib64/
'_open', req)
File "/usr/lib64/
result = func(*args)
File "/usr/lib64/
return self.do_
File "/usr/lib64/
r = h.getresponse()
File "/usr/lib64/
response.
File "/usr/lib64/
version, status, reason = self._read_status()
File "/usr/lib64/
raise BadStatusLine(line)
httplib.
But after some period of time fuel client started working
Logs are attached
Changed in fuel: | |
assignee: | Fuel Python Team (fuel-python) → Igor Kalnitsky (ikalnitsky) |
Changed in fuel: | |
milestone: | 5.1.1 → 6.0 |
Changed in fuel: | |
status: | New → Confirmed |
Changed in fuel: | |
status: | Confirmed → Fix Committed |
Changed in fuel: | |
status: | Fix Committed → Fix Released |
Very strange and not enough info. It looks like there's some delay between fuel_upgrade script is done and all services are running. Need to investigate, but it's definitely not a Critical, because some timeout (1min) is enough to avoid this issue.