Environment
RHOSP STEIN in HA environment
- 3x controllers
- 4x computes
- 5x ceph
While running openstack scenario full_l2.yaml shaker failed here:
Jun 30 17:00:34 u20 shaker[65181]: 2020-06-30 17:00:34.679 65181 WARNING shaker.engine.quorum [-] Lost agents: {'shaker_dtfirk_slave_1', 'shaker_dtfirk_slave_0', 'shaker_dtfirk_master_1', 'sha
ker_dtfirk_master_0'}
Jun 30 17:00:34 u20 shaker[65181]: 2020-06-30 17:00:34.679 65181 INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.JoinOperation object at 0x7f91ffc16e80> Jun 30 17:00:34 u20 shaker[65185]: 2020-06-30 17:00:34.682 65185 DEBUG shaker.agent.agent [-] Received: {'operation': 'none'} poll_task /home/skhan/.local/lib/python3.8/site-packages/shaker/a$
ent/agent.py:43
Jun 30 17:00:34 u20 shaker[65181]: 2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server [-] Agents failed to join: {'shaker_dtfirk_slave_1': 'lost', 'shaker_dtfirk_slave_0': 'lost', 'shak$r_dtfirk_master_0': 'lost', 'shaker_dtfirk_master_1': 'lost'}: Exception: Agents failed to join: {'shaker_dtfirk_slave_1': 'lost', 'shaker_dtfirk_slave_0': 'lost', 'shaker_dtfirk_master_0': '$
ost', 'shaker_dtfirk_master_1': 'lost'} 2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server Traceback (most recent call last):
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server File "/home/skhan/.local/lib/python3.8/site-packages/shaker/engine/server.py", line 188, in play_$
cenario 2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server quorum = quorum_pkg.make_quorum(
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server File "/home/skhan/.local/lib/python3.8/site-packages/shaker/engine/quorum.py", line 244, in make_$uorum
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server raise Exception('Agents failed to join: %s' % failed)
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server Exception: Agents failed to join: {'shaker_dtfirk_slave_1': 'lost', 'shaker_dtfirk_slave_0': 'lost'$
'shaker_dtfirk_master_0': 'lost', 'shaker_dtfirk_master_1': 'lost'}
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server
whereas from the keystone/heat-engine logs found the policy related exception, that is:
keystone.log
------------
2020-06-30 16:48:51.849 30 WARNING keystone.auth.plugins.core [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] User is disabled: 01c67033fdee4073a0432c5c007264b5: AssertionError: User is disabled: 01c67033fdee4073a0432c5c007264b5
2020-06-30 16:48:51.851 30 WARNING keystone.server.flask.application [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] Authorization failed. The request you have made requires authentication. from 10.133.128.143: keystone.exception.Unauthorized: The request you have made requires authentication.
heat-engine.log
---------------
2020-06-30 16:48:51.853 23 ERROR heat.engine.clients.keystoneclient [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Domain admin client authentication failed: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a7c5e591-5a10-4587-9061-e51e5f4ca07d)
2020-06-30 16:48:51.854 23 INFO heat.engine.stack [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Stack CREATE FAILED (shaker_dtfirk): Authorization failed.
Seems like the shaker code is not ready for the changes made in the new identity policy.
nice to have a work around from the experts here.. Cheers.. :-)
Couple of hints to proceed: /opendev. org/performa/ shaker/ src/branch/ master/ shaker/ openstack/ clients/ openstack. py#L67). Is there a message 'Connection to OpenStack is initialized' in Shaker logs? auth.plugins. core [req-a7c5e591- 5a10-4587- 9061-e51e5f4ca0 7d - - - - -] User is disabled:' -- is it expected that the user is disabled? Could it be that the user does not have some of permissions (e.g. to create Heat stack)?
1. It's strange that authentication fails so late when Heat stack is being created. During the initialization Shaker verifies credentials by requesting a token from Keystone (https:/
2. There is a message from keystone log '2020-06-30 16:48:51.849 30 WARNING keystone.