I newly deployed a yoga version of the cluster with 3 control nodes and 10 computing nodes（with ceph）, and everything looked fine at the beginning. When I shut down a control node, the dhcp agent is still on the shut down node, and other nodes need to restart the neutron_dhcp_agent container to take over the dhcp service. An error also occurred when creating an instance. Can you help diagnose which component is causing the problem?
################################
launch instance Error
消息
MessagingTimeout
编码
500
详情
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get return self._queues[msg_id].get(block=True, timeout=timeout) File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 322, in get return waiter.wait() File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 141, in wait return get_hub().switch() File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 313, in switch return self.greenlet.switch() queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1549, in schedule_and_build_instances instance_uuids, return_alternates=True) File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 910, in _schedule_instances return_alternates=return_alternates) File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations instance_uuids, return_objects, return_alternates) File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192, in call retry=self.retry, transport_options=self.transport_options) File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send transport_options=transport_options) File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in send transport_options=transport_options) File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 679, in _send call_monitor_timeout) File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait message = self.waiters.get(msg_id, timeout=timeout) File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 445, in get 'to message ID %s' % msg_id) oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID e81563a1da4c4baa8bf7350f82737bce
已创建
2023年3月31日 00:37
##########################################
DHCP-agent log
2023-03-31 07:42:19.902 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [c18a1bc9-3fbd-49b3-afb1-03d603d577a9] AMQP server on 172.18.218.
34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionErr
or: <RecoverableConnectionError: unknown error>
2023-03-31 07:42:19.929 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [8dae00b8-bfbe-44ab-bac2-45632d68ae22] AMQP server on 172.18.218.
34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionErr
or: <RecoverableConnectionError: unknown error>
2023-03-31 07:42:25.904 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [c18a1bc9-3fbd-49b3-afb1-03d603d577a9] AMQP server on 172.18.218.
34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionErr
or: <RecoverableConnectionError: unknown error>
2023-03-31 07:42:25.930 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [8dae00b8-bfbe-44ab-bac2-45632d68ae22] AMQP server on 172.18.218.
34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionErr
or: <RecoverableConnectionError: unknown error>
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent [req-706f824e-106c-4e84-9a3b-92b77e8eb6e5 - - - - -] Failed reporting state!: os
lo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 0c7263bb71294201b8b720413043cd52
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 441, in get
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent return self._queues[msg_id].get(block=True, timeout=timeout)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 322, in get
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent return waiter.wait()
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 141, in wait
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent return get_hub().switch()
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 313, in swi
tch
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent return self.greenlet.switch()
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent queue.Empty
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent During handling of the above exception, another exception occurred:
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 1049
, in _report_state
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent ctx, self.agent_state, True)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/rpc.py", line 104, in rep
ort_state
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent return method(context, 'report_state', **kwargs)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 192
, in call
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent retry=self.retry, transport_options=self.transport_options)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128,
in _send
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent transport_options=transport_options)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 691, in send
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent transport_options=transport_options)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 679, in _send
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent call_monitor_timeout)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 567, in wait
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 445, in get
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id)
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to mes
sage ID 0c7263bb71294201b8b720413043cd52
2023-03-31 07:42:26.121 7 ERROR neutron.agent.dhcp.agent
2023-03-31 07:42:31.906 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [c18a1bc9-3fbd-49b3-afb1-03d603d577a9] AMQP server on 172.18.218.
34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionErr
or: <RecoverableConnectionError: unknown error>
2023-03-31 07:42:31.932 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [8dae00b8-bfbe-44ab-bac2-45632d68ae22] AMQP server on 172.18.218.
34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionErr
or: <RecoverableConnectionError: unknown error>
2023-03-31 07:42:36.404 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [8dae00b8-bfbe-44ab-bac2-45632d68ae22] AMQP server on 172.18.218.
34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionErr
or: <RecoverableConnectionError: unknown error>
#############################
rabbitmq log
2023-03-30 16:29:49.025129+08:00 [info] <0.232.0> Successfully synced tables from a peer
2023-03-30 16:29:49.025161+08:00 [warning] <0.232.0> Feature flags: the previous instance of this node must have failed to write the `fea
ture_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@L12D09Ctrl35-feature_flags`:
2023-03-30 16:29:49.025210+08:00 [warning] <0.232.0> Feature flags: - list of previously enabled feature flags now marked as such: [mai
ntenance_mode_status,
2023-03-30 16:29:49.025210+08:00 [warning] <0.232.0> quo
rum_queue,
2023-03-30 16:29:49.025210+08:00 [warning] <0.232.0>
2023-03-31 07:41:51.002530+08:00 [error] <0.18744.7> Channel error on connection <0.18712.7> (172.18.218.36:47340 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.002530+08:00 [error] <0.18744.7> operation queue.declare caused a channel exception not_found: queue 'engine' in vhos
t '/' process is stopped by supervisor
2023-03-31 07:41:51.002603+08:00 [error] <0.18745.7> Channel error on connection <0.18708.7> (172.18.218.36:47326 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.002603+08:00 [error] <0.18745.7> operation queue.declare caused a channel exception not_found: queue 'engine' in vhos
t '/' process is stopped by supervisor
2023-03-31 07:41:51.002782+08:00 [info] <0.18768.7> accepting AMQP connection <0.18768.7> (172.18.218.35:37102 -> 172.18.218.35:5672)
2023-03-31 07:41:51.003177+08:00 [info] <0.18754.7> Connection <0.18754.7> (172.18.218.35:37090 -> 172.18.218.35:5672) has a client-provi
ded name: neutron-server:32:2b02fc80-206e-4555-80fa-04eb52d18db4
2023-03-31 07:41:51.003920+08:00 [error] <0.18766.7> Channel error on connection <0.18735.7> (172.18.218.35:37082 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.003920+08:00 [error] <0.18766.7> operation queue.declare caused a channel exception not_found: queue 'engine' in vhos
t '/' process is stopped by supervisor
2023-03-31 07:41:51.003945+08:00 [info] <0.18768.7> Connection <0.18768.7> (172.18.218.35:37102 -> 172.18.218.35:5672) has a client-provi
ded name: neutron-server:31:c47e3227-ceec-4615-af6e-beb74f58817a
2023-03-31 07:41:51.004112+08:00 [info] <0.18754.7> connection <0.18754.7> (172.18.218.35:37090 -> 172.18.218.35:5672 - neutron-server:32
:2b02fc80-206e-4555-80fa-04eb52d18db4): user 'openstack' authenticated and granted access to vhost '/'
2023-03-31 07:41:51.004542+08:00 [info] <0.18768.7> connection <0.18768.7> (172.18.218.35:37102 -> 172.18.218.35:5672 - neutron-server:31
:c47e3227-ceec-4615-af6e-beb74f58817a): user 'openstack' authenticated and granted access to vhost '/'
2023-03-31 07:41:51.004854+08:00 [error] <0.18779.7> Channel error on connection <0.18708.7> (172.18.218.36:47326 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.004854+08:00 [error] <0.18779.7> operation queue.declare caused a channel exception not_found: queue 'engine' in vhos
t '/' process is stopped by supervisor
2023-03-31 07:41:51.004879+08:00 [error] <0.18782.7> Channel error on connection <0.18712.7> (172.18.218.36:47340 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.004879+08:00 [error] <0.18782.7> operation queue.declare caused a channel exception not_found: queue 'engine' in vhos
t '/' process is stopped by supervisor
2023-03-31 07:41:51.005774+08:00 [info] <0.18828.7> accepting AMQP connection <0.18828.7> (172.18.218.36:47356 -> 172.18.218.35:5672)
2023-03-31 07:41:51.005799+08:00 [error] <0.18805.7> Channel error on connection <0.18735.7> (172.18.218.35:37082 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.005799+08:00 [error] <0.18805.7> operation queue.declare caused a channel exception not_found: queue 'engine' in vhos
t '/' process is stopped by supervisor
2023-03-31 07:41:51.006162+08:00 [error] <0.18804.7> Channel error on connection <0.18754.7> (172.18.218.35:37090 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.006162+08:00 [error] <0.18804.7> operation queue.declare caused a channel exception not_found: queue 'q-reports-plugi
n' in vhost '/' process is stopped by supervisor
2023-03-31 07:41:51.006846+08:00 [info] <0.18828.7> Connection <0.18828.7> (172.18.218.36:47356 -> 172.18.218.35:5672) has a client-provi
ded name: neutron-server:30:4d90dd95-94a4-4493-980c-e9b0696d2fa6
2023-03-31 07:41:51.006729+08:00 [error] <0.18813.7> Channel error on connection <0.18768.7> (172.18.218.35:37102 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.006729+08:00 [error] <0.18813.7> operation queue.declare caused a channel exception not_found: queue 'q-reports-plugi
n' in vhost '/' process is stopped by supervisor
2023-03-31 07:41:51.007117+08:00 [error] <0.18824.7> Channel error on connection <0.18712.7> (172.18.218.36:47340 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
2023-03-31 07:41:51.007117+08:00 [error] <0.18824.7> operation queue.declare caused a channel exception not_found: queue 'engine' in vhos
t '/' process is stopped by supervisor
2023-03-31 07:41:51.007290+08:00 [info] <0.18828.7> connection <0.18828.7> (172.18.218.36:47356 -> 172.18.218.35:5672 - neutron-server:30
:4d90dd95-94a4-4493-980c-e9b0696d2fa6): user 'openstack' authenticated and granted access to vhost '/'
2023-03-31 07:41:51.007578+08:00 [error] <0.18827.7> Channel error on connection <0.18708.7> (172.18.218.36:47326 -> 172.18.218.35:5672,
vhost: '/', user: 'openstack'), channel 1:
#################
nova-scheduler log
2023-03-31 07:41:49.881 20 ERROR oslo.messaging._drivers.impl_rabbit [-] [5488d979-c94c-4345-9928-618ff11ce67b] AMQP server on 172.18.218
.34:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection r
eset by peer
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines [req-7c633b84-0242-438b-8fd9-477cc5d9b59b - - - - -] Database connection was
found disconnected; reconnecting: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL se
rver during query')
[SQL: SELECT 1]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1
803, in _execute_context
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines cursor, statement, parameters, context
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", lin
e 732, in do_execute
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in ex
ecute
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _q
uery
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, i
n query
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, i
n _read_query_result
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines result.read()
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069,
in read
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 646, i
n _read_packet
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, i
n _read_bytes
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during
query')
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines The above exception was the direct cause of the following exception:
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line
73, in _connect_ping_listener
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines connection.scalar(select(1))
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1
171, in scalar
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines return self.execute(object_, *multiparams, **params).scalar()
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1
289, in execute
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line
326, in _execute_on_connection
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines self, multiparams, params, execution_options
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1
491, in _execute_clauseelement
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines cache_hit=cache_hit,
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1
846, in _execute_context
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines e, statement, parameters, cursor, context
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2
024, in _handle_dbapi_exception
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 2
07, in raise_
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines raise exception
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1
803, in _execute_context
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines cursor, statement, parameters, context
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", lin
e 732, in do_execute
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in ex
ecute
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _q
uery
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, i
n query
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, i
n _read_query_result
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines result.read()
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069,
in read
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 646, i
n _read_packet
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, i
n _read_bytes
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'L
ost connection to MySQL server during query')
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines [SQL: SELECT 1]
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines (Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-03-31 07:41:50.769 22 ERROR oslo_db.sqlalchemy.engines
2023-03-31 07:41:50.883 20 ERROR oslo.messaging._drivers.impl_rabbit [-] [5488d979-c94c-4345-9928-618ff11ce67b] AMQP server on 172.18.218
.34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionEr
ror: <RecoverableConnectionError: unknown error>
2023-03-31 07:41:51.885 20 ERROR oslo.messaging._drivers.impl_rabbit [-] [5488d979-c94c-4345-9928-618ff11ce67b] AMQP server on 172.18.218
.34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionEr
ror: <RecoverableConnectionError: unknown error>
2023-03-31 07:41:52.887 20 ERROR oslo.messaging._drivers.impl_rabbit [-] [5488d979-c94c-4345-9928-618ff11ce67b] AMQP server on 172.18.218
.34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionEr
ror: <RecoverableConnectionError: unknown error>
2023-03-31 07:41:53.889 20 ERROR oslo.messaging._drivers.impl_rabbit [-] [5488d979-c94c-4345-9928-618ff11ce67b] AMQP server on 172.18.218
.34:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionEr
ror: <RecoverableConnectionError: unknown error>
2023-03-31 07:41:54.125 24 ERROR oslo_db.sqlalchemy.engines [req-af08f0df-2b3a-4cdc-b939-99e38d9d3672 - - - - -] Database connection was
found disconnected; reconnecting: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL se
rver during query')
[SQL: SELECT 1]
#############
cinder log
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines [req-80ecf80a-5520-47a0-8c12-dd503f6f699d - - - - -] Database connection was f
ound disconnected; reconnecting: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL ser
ver during query')
[SQL: SELECT 1]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 18
03, in _execute_context
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines cursor, statement, parameters, context
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line
732, in do_execute
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in exe
cute
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _qu
ery
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, in
query
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, in
_read_query_result
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines result.read()
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069, i
n read
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 646, in
_read_packet
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, in
_read_bytes
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during q
uery')
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines The above exception was the direct cause of the following exception:
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line
73, in _connect_ping_listener
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines connection.scalar(select(1))
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 11
71, in scalar
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines return self.execute(object_, *multiparams, **params).scalar()
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 12
89, in execute
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 3
26, in _execute_on_connection
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines self, multiparams, params, execution_options
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 14
91, in _execute_clauseelement
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines cache_hit=cache_hit,
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 18
46, in _execute_context
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines e, statement, parameters, cursor, context
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 20
24, in _handle_dbapi_exception
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 20
7, in raise_
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines raise exception
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 18
03, in _execute_context
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines cursor, statement, parameters, context
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line
732, in do_execute
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in exe
cute
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _qu
ery
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 505, in
query
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 724, in
_read_query_result
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines result.read()
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1069, i
n read
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 646, in
_read_packet
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, in
_read_bytes
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lo
st connection to MySQL server during query')
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines [SQL: SELECT 1]
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines (Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-03-31 07:41:58.230 7 ERROR oslo_db.sqlalchemy.engines
2023-03-31 07:42:04.244 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconne
ct: [Errno 104] Connection reset by peer
2023-03-31 07:42:09.247 7 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: timed out (retrying in 0 seconds): socket.time
out: timed out
2023-03-31 07:44:11.421 7 WARNING oslo_messaging._drivers.amqpdriver [req-8918629b-4aef-47cf-b1fd-1c1e47121842 fa9756342ebe41979972bda128
e051d0 035c6fc635dc4894a786ab834eebd398 - - -] reply_dd28ce899345422198ca719341b11e06 doesn't exist, drop reply to 19c221d8acbd428697dbdd
0ef918bd27: oslo_messaging.exceptions.MessageUndeliverable
2023-03-31 07:44:11.421 7 ERROR oslo_messaging._drivers.amqpdriver [req-8918629b-4aef-47cf-b1fd-1c1e47121842 fa9756342ebe41979972bda128e0
51d0 035c6fc635dc4894a786ab834eebd398 - - -] The reply 19c221d8acbd428697dbdd0ef918bd27 failed to send after 60 seconds due to a missing
queue (reply_dd28ce899345422198ca719341b11e06). Abandoning...: oslo_messaging.exceptions.MessageUndeliverable
2023-03-31 07:44:48.509 7 WARNING oslo_messaging._drivers.amqpdriver [req-523ccf74-1909-42ba-92d9-a9d4ecfc7388 fa9756342ebe41979972bda128
e051d0 035c6fc635dc4894a786ab834eebd398 - - -] reply_ee705547a439408aa0874847aafd480b doesn't exist, drop reply to 0fa48d57d417499b984f16
9c5fa660b7: oslo_messaging.exceptions.MessageUndeliverable
2023-03-31 07:44:48.509 7 ERROR oslo_messaging._drivers.amqpdriv
