| 2020-06-30 13:13:43 |
SK |
bug |
|
|
added bug |
| 2020-06-30 13:41:25 |
SK |
description |
Environment
RHOSP STEIN in HA environment
- 3x controllers
- 4x computes
- 5x ceph
While running openstack scenario full_l2.yaml shaker failed here:
2020-06-30 15:44:24.903 65015 DEBUG shaker.engine.messaging [-] Received request: {'operation': 'poll', 'agent_id': '__heartbeat'} __iter__ /home/skhan/.local/lib/python3.8/site-packages/shake
r/engine/messaging.py:62
2020-06-30 15:44:24.904 65019 DEBUG shaker.agent.agent [-] Received: {'operation': 'none'} poll_task /home/skhan/.local/lib/python3.8/site-packages/shaker/agent/agent.py:43 2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.messaging [-] Sent reply: {'operation': 'none'} reply_handler /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/messaging.py:66
2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.quorum [-] Lost agents: {'__cleaner'} _run /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/quorum.py:185
2020-06-30 15:44:24.906 65015 INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.CleanOperation object at 0x7f0e245db460> 2020-06-30 15:44:24.906 65015 DEBUG shaker.engine.deploy [-] Cleaning up the test stack: shaker_orktda with id: 2e95855a-99bf-4fac-98c4-f09de796e177 cleanup /home/skhan/.local/lib/python3.8/s$
te-packages/shaker/engine/deploy.py:466
whereas from the keystone/heat-engine logs found the policy related exception, that is:
keystone.log
------------
2020-06-30 15:33:03.400 25 WARNING keystone.server.flask.application [req-d348ba00-f89a-4b20-987b-6108480185e4 8f5644850ee64d7a96f751b64a49a416 - - default -] Authorization failed. The request you have made requires authentication. from 10.133.128.143: keystone.exception.Unauthorized: The request you have made requires authentication.
heat-engine.log
---------------
/var/log/containers/heat/heat-engine.log:2020-06-30 15:32:44.107 28 ERROR heat.engine.clients.keystoneclient [req-3f3cfc88-656b-49e3-9580-a7786d1aaae3 - sProject - default default] Domain admin client authentication failed: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-b14d1fbd-17f1-40a7-aa55-a30856ba55fc)/var/log/containers/heat/heat-engine.log:2020-06-30 15:32:44.108 28 INFO heat.engine.stack [req-3f3cfc88-656b-49e3-9580-a7786d1aaae3 - sProject - default default] Stack CREATE FAILED (shaker_o
rktda): Authorization failed.
Seems like the shaker code is not ready for the changes made in the new identity policy.
nice to have a work around from the experts here.. Cheers.. :-) |
Environment
RHOSP STEIN in HA environment
- 3x controllers
- 4x computes
- 5x ceph
While running openstack scenario full_l2.yaml shaker failed here:
2020-06-30 15:44:24.903 65015 DEBUG shaker.engine.messaging [-] Received request: {'operation': 'poll', 'agent_id': '__heartbeat'} __iter__ /home/skhan/.local/lib/python3.8/site-packages/shake
r/engine/messaging.py:62
2020-06-30 15:44:24.904 65019 DEBUG shaker.agent.agent [-] Received: {'operation': 'none'} poll_task /home/skhan/.local/lib/python3.8/site-packages/shaker/agent/agent.py:43 2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.messaging [-] Sent reply: {'operation': 'none'} reply_handler /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/messaging.py:66
2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.quorum [-] Lost agents: {'__cleaner'} _run /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/quorum.py:185
2020-06-30 15:44:24.906 65015 INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.CleanOperation object at 0x7f0e245db460> 2020-06-30 15:44:24.906 65015 DEBUG shaker.engine.deploy [-] Cleaning up the test stack: shaker_orktda with id: 2e95855a-99bf-4fac-98c4-f09de796e177 cleanup /home/skhan/.local/lib/python3.8/s$
te-packages/shaker/engine/deploy.py:466
whereas from the keystone/heat-engine logs found the policy related exception, that is:
keystone.log
------------
2020-06-30 15:23:15.805 24 WARNING py.warnings [req-717ef91e-9cc8-4d61-89ce-75079023cbf4 8f5644850ee64d7a96f751b64a49a416 a7878ba89d7a498abd3998860819b644 - default default] /usr/lib/python3.6/site-packages/oslo_policy/policy.py:964: UserWarning: Policy identity:list_role_assignments failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required
warnings.warn(msg)
heat-engine.log
---------------
/var/log/containers/heat/heat-engine.log:2020-06-30 15:32:44.107 28 ERROR heat.engine.clients.keystoneclient [req-3f3cfc88-656b-49e3-9580-a7786d1aaae3 - sProject - default default] Domain admin client authentication failed: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-b14d1fbd-17f1-40a7-aa55-a30856ba55fc)/var/log/containers/heat/heat-engine.log:2020-06-30 15:32:44.108 28 INFO heat.engine.stack [req-3f3cfc88-656b-49e3-9580-a7786d1aaae3 - sProject - default default] Stack CREATE FAILED (shaker_o
rktda): Authorization failed.
Seems like the shaker code is not ready for the changes made in the new identity policy.
nice to have a work around from the experts here.. Cheers.. :-) |
|
| 2020-06-30 13:50:55 |
SK |
description |
Environment
RHOSP STEIN in HA environment
- 3x controllers
- 4x computes
- 5x ceph
While running openstack scenario full_l2.yaml shaker failed here:
2020-06-30 15:44:24.903 65015 DEBUG shaker.engine.messaging [-] Received request: {'operation': 'poll', 'agent_id': '__heartbeat'} __iter__ /home/skhan/.local/lib/python3.8/site-packages/shake
r/engine/messaging.py:62
2020-06-30 15:44:24.904 65019 DEBUG shaker.agent.agent [-] Received: {'operation': 'none'} poll_task /home/skhan/.local/lib/python3.8/site-packages/shaker/agent/agent.py:43 2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.messaging [-] Sent reply: {'operation': 'none'} reply_handler /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/messaging.py:66
2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.quorum [-] Lost agents: {'__cleaner'} _run /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/quorum.py:185
2020-06-30 15:44:24.906 65015 INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.CleanOperation object at 0x7f0e245db460> 2020-06-30 15:44:24.906 65015 DEBUG shaker.engine.deploy [-] Cleaning up the test stack: shaker_orktda with id: 2e95855a-99bf-4fac-98c4-f09de796e177 cleanup /home/skhan/.local/lib/python3.8/s$
te-packages/shaker/engine/deploy.py:466
whereas from the keystone/heat-engine logs found the policy related exception, that is:
keystone.log
------------
2020-06-30 15:23:15.805 24 WARNING py.warnings [req-717ef91e-9cc8-4d61-89ce-75079023cbf4 8f5644850ee64d7a96f751b64a49a416 a7878ba89d7a498abd3998860819b644 - default default] /usr/lib/python3.6/site-packages/oslo_policy/policy.py:964: UserWarning: Policy identity:list_role_assignments failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required
warnings.warn(msg)
heat-engine.log
---------------
/var/log/containers/heat/heat-engine.log:2020-06-30 15:32:44.107 28 ERROR heat.engine.clients.keystoneclient [req-3f3cfc88-656b-49e3-9580-a7786d1aaae3 - sProject - default default] Domain admin client authentication failed: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-b14d1fbd-17f1-40a7-aa55-a30856ba55fc)/var/log/containers/heat/heat-engine.log:2020-06-30 15:32:44.108 28 INFO heat.engine.stack [req-3f3cfc88-656b-49e3-9580-a7786d1aaae3 - sProject - default default] Stack CREATE FAILED (shaker_o
rktda): Authorization failed.
Seems like the shaker code is not ready for the changes made in the new identity policy.
nice to have a work around from the experts here.. Cheers.. :-) |
Environment
RHOSP STEIN in HA environment
- 3x controllers
- 4x computes
- 5x ceph
While running openstack scenario full_l2.yaml shaker failed here:
2020-06-30 15:44:24.903 65015 DEBUG shaker.engine.messaging [-] Received request: {'operation': 'poll', 'agent_id': '__heartbeat'} __iter__ /home/skhan/.local/lib/python3.8/site-packages/shake
r/engine/messaging.py:62
2020-06-30 15:44:24.904 65019 DEBUG shaker.agent.agent [-] Received: {'operation': 'none'} poll_task /home/skhan/.local/lib/python3.8/site-packages/shaker/agent/agent.py:43 2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.messaging [-] Sent reply: {'operation': 'none'} reply_handler /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/messaging.py:66
2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.quorum [-] Lost agents: {'__cleaner'} _run /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/quorum.py:185
2020-06-30 15:44:24.906 65015 INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.CleanOperation object at 0x7f0e245db460> 2020-06-30 15:44:24.906 65015 DEBUG shaker.engine.deploy [-] Cleaning up the test stack: shaker_orktda with id: 2e95855a-99bf-4fac-98c4-f09de796e177 cleanup /home/skhan/.local/lib/python3.8/s$
te-packages/shaker/engine/deploy.py:466
whereas from the keystone/heat-engine logs found the policy related exception, that is:
keystone.log
------------
2020-06-30 16:48:51.849 30 WARNING keystone.auth.plugins.core [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] User is disabled: 01c67033fdee4073a0432c5c007264b5: AssertionError: User is disabled: 01c67033fdee4073a0432c5c007264b5
2020-06-30 16:48:51.851 30 WARNING keystone.server.flask.application [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] Authorization failed. The request you have made requires authentication. from 10.133.128.143: keystone.exception.Unauthorized: The request you have made requires authentication.
heat-engine.log
---------------
2020-06-30 16:48:51.853 23 ERROR heat.engine.clients.keystoneclient [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Domain admin client authentication failed: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a7c5e591-5a10-4587-9061-e51e5f4ca07d)
2020-06-30 16:48:51.854 23 INFO heat.engine.stack [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Stack CREATE FAILED (shaker_dtfirk): Authorization failed.
Seems like the shaker code is not ready for the changes made in the new identity policy.
nice to have a work around from the experts here.. Cheers.. :-) |
|
| 2020-06-30 14:02:31 |
SK |
description |
Environment
RHOSP STEIN in HA environment
- 3x controllers
- 4x computes
- 5x ceph
While running openstack scenario full_l2.yaml shaker failed here:
2020-06-30 15:44:24.903 65015 DEBUG shaker.engine.messaging [-] Received request: {'operation': 'poll', 'agent_id': '__heartbeat'} __iter__ /home/skhan/.local/lib/python3.8/site-packages/shake
r/engine/messaging.py:62
2020-06-30 15:44:24.904 65019 DEBUG shaker.agent.agent [-] Received: {'operation': 'none'} poll_task /home/skhan/.local/lib/python3.8/site-packages/shaker/agent/agent.py:43 2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.messaging [-] Sent reply: {'operation': 'none'} reply_handler /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/messaging.py:66
2020-06-30 15:44:24.905 65015 DEBUG shaker.engine.quorum [-] Lost agents: {'__cleaner'} _run /home/skhan/.local/lib/python3.8/site-packages/shaker/engine/quorum.py:185
2020-06-30 15:44:24.906 65015 INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.CleanOperation object at 0x7f0e245db460> 2020-06-30 15:44:24.906 65015 DEBUG shaker.engine.deploy [-] Cleaning up the test stack: shaker_orktda with id: 2e95855a-99bf-4fac-98c4-f09de796e177 cleanup /home/skhan/.local/lib/python3.8/s$
te-packages/shaker/engine/deploy.py:466
whereas from the keystone/heat-engine logs found the policy related exception, that is:
keystone.log
------------
2020-06-30 16:48:51.849 30 WARNING keystone.auth.plugins.core [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] User is disabled: 01c67033fdee4073a0432c5c007264b5: AssertionError: User is disabled: 01c67033fdee4073a0432c5c007264b5
2020-06-30 16:48:51.851 30 WARNING keystone.server.flask.application [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] Authorization failed. The request you have made requires authentication. from 10.133.128.143: keystone.exception.Unauthorized: The request you have made requires authentication.
heat-engine.log
---------------
2020-06-30 16:48:51.853 23 ERROR heat.engine.clients.keystoneclient [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Domain admin client authentication failed: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a7c5e591-5a10-4587-9061-e51e5f4ca07d)
2020-06-30 16:48:51.854 23 INFO heat.engine.stack [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Stack CREATE FAILED (shaker_dtfirk): Authorization failed.
Seems like the shaker code is not ready for the changes made in the new identity policy.
nice to have a work around from the experts here.. Cheers.. :-) |
Environment
RHOSP STEIN in HA environment
- 3x controllers
- 4x computes
- 5x ceph
While running openstack scenario full_l2.yaml shaker failed here:
Jun 30 17:00:34 u20 shaker[65181]: 2020-06-30 17:00:34.679 65181 WARNING shaker.engine.quorum [-] Lost agents: {'shaker_dtfirk_slave_1', 'shaker_dtfirk_slave_0', 'shaker_dtfirk_master_1', 'sha
ker_dtfirk_master_0'}
Jun 30 17:00:34 u20 shaker[65181]: 2020-06-30 17:00:34.679 65181 INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.JoinOperation object at 0x7f91ffc16e80> Jun 30 17:00:34 u20 shaker[65185]: 2020-06-30 17:00:34.682 65185 DEBUG shaker.agent.agent [-] Received: {'operation': 'none'} poll_task /home/skhan/.local/lib/python3.8/site-packages/shaker/a$
ent/agent.py:43
Jun 30 17:00:34 u20 shaker[65181]: 2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server [-] Agents failed to join: {'shaker_dtfirk_slave_1': 'lost', 'shaker_dtfirk_slave_0': 'lost', 'shak$r_dtfirk_master_0': 'lost', 'shaker_dtfirk_master_1': 'lost'}: Exception: Agents failed to join: {'shaker_dtfirk_slave_1': 'lost', 'shaker_dtfirk_slave_0': 'lost', 'shaker_dtfirk_master_0': '$
ost', 'shaker_dtfirk_master_1': 'lost'} 2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server Traceback (most recent call last):
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server File "/home/skhan/.local/lib/python3.8/site-packages/shaker/engine/server.py", line 188, in play_$
cenario 2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server quorum = quorum_pkg.make_quorum(
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server File "/home/skhan/.local/lib/python3.8/site-packages/shaker/engine/quorum.py", line 244, in make_$uorum
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server raise Exception('Agents failed to join: %s' % failed)
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server Exception: Agents failed to join: {'shaker_dtfirk_slave_1': 'lost', 'shaker_dtfirk_slave_0': 'lost'$
'shaker_dtfirk_master_0': 'lost', 'shaker_dtfirk_master_1': 'lost'}
2020-06-30 17:00:34.680 65181 ERROR shaker.engine.server
whereas from the keystone/heat-engine logs found the policy related exception, that is:
keystone.log
------------
2020-06-30 16:48:51.849 30 WARNING keystone.auth.plugins.core [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] User is disabled: 01c67033fdee4073a0432c5c007264b5: AssertionError: User is disabled: 01c67033fdee4073a0432c5c007264b5
2020-06-30 16:48:51.851 30 WARNING keystone.server.flask.application [req-a7c5e591-5a10-4587-9061-e51e5f4ca07d - - - - -] Authorization failed. The request you have made requires authentication. from 10.133.128.143: keystone.exception.Unauthorized: The request you have made requires authentication.
heat-engine.log
---------------
2020-06-30 16:48:51.853 23 ERROR heat.engine.clients.keystoneclient [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Domain admin client authentication failed: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a7c5e591-5a10-4587-9061-e51e5f4ca07d)
2020-06-30 16:48:51.854 23 INFO heat.engine.stack [req-a7e4c70a-8732-4a85-9782-e913e4f2d7fc - sProject - default default] Stack CREATE FAILED (shaker_dtfirk): Authorization failed.
Seems like the shaker code is not ready for the changes made in the new identity policy.
nice to have a work around from the experts here.. Cheers.. :-) |
|
