[bionic][ussuri] functest failing due to connection reset during setup
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Keystone Charm |
Triaged
|
High
|
Unassigned |
Bug Description
The section it got to:
2020-10-05 13:38:37 [INFO] Using keystone API V3 (or later) for overcloud auth
2020-10-05 13:38:43 [INFO] Downloading image bionic
2020-10-05 13:40:10 [INFO] active
2020-10-05 13:40:10 [INFO] Setting image properties: None
2020-10-05 13:41:34 [INFO] Using keystone API V3 (or later) for overcloud auth
2020-10-05 13:42:39 [ERROR] {'default_alias': 'zaza-ab0a67dae
2020-10-05 13:42:39 [ERROR] Model default_alias (zaza-ab0a67dae570)
i.e. it didn't finish setting up.
The model looks stable, but it could be due to a hook still running (i.e. the check for stability wasn't true).
The traceback:
Traceback (most recent call last):
File "/tmp/tmp.
config_
File "/tmp/tmp.
run_
File "/tmp/tmp.
utils.
File "/tmp/tmp.
_v3()
File "/tmp/tmp.
enabled=True)
File "/tmp/tmp.
**kwargs)
File "/tmp/tmp.
return f(*args, **new_kwargs)
File "/tmp/tmp.
self.key)
File "/tmp/tmp.
resp, body = self.client.
File "/tmp/tmp.
return self.request(url, 'POST', **kwargs)
File "/tmp/tmp.
resp = super(LegacyJso
File "/tmp/tmp.
return self.session.
File "/tmp/tmp.
resp = send(**kwargs)
File "/tmp/tmp.
raise exceptions.
keystoneauth1.
tags: | added: unstable-test |
Changed in charm-keystone: | |
status: | New → Triaged |
importance: | Undecided → High |
"('Connection aborted.', ConnectionReset Error(104, 'Connection reset by peer')" is almost certainly caused by haproxy timing out and dropping the connection.
It is likely due to a busy undercloud causing connections to pile up on keystone units and eventually timeout. This is difficult to prove without having a process list and cpu usage metrics.
Potential path forward:
Since keystone is central to the entire cloud we should give it the resources to handle the barrage of requests.
* Bump the constraints for keystone units so they have more RAM
* Up the number of threads keystone runs with worker-multiplier (if necessary, the bump may do this for us)
Note: we have seen similar with neutron-api and other API charms. It is possible we should do this for all API charms.