MariaDB: database backup cause DB Deadlock

Bug #2034018 reported by Dongcan Ye

This bug report was marked for expiration 368 days ago. (find out why)

12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla
Incomplete
Undecided
Unassigned

Bug Description

Using kolla-ansible to backup database based on MariaDB, it will cause some OpenStack projects DB deadlock,
especially for Octavia, Octavia health-manager uses amphora_health table as the basis for failover. If amphora_health table can't update, it will cause failover Amphora.

2023-08-05 03:06:28.789 49 ERROR octavia.amphorae.drivers.health.heartbeat_udp [-] Health update for amphora 467e8b0b-8314-4936-ae85-a039643e26c8 encountered error (pymysql.err.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
[SQL: UPDATE amphora_health SET last_update=%(last_update)s WHERE amphora_health.amphora_id = %(amphora_id_1)s]
[parameters: {'last_update': datetime.datetime(2023, 8, 4, 19, 5, 38, 784220), 'amphora_id_1': '467e8b0b-8314-4936-ae85-a039643e26c8'}]
(Background on this error at: https://sqlalche.me/e/14/e3q8). Skipping health update.: oslo_db.exception.DBDeadlock: (pymysql.err.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
[SQL: UPDATE amphora_health SET last_update=%(last_update)s WHERE amphora_health.amphora_id = %(amphora_id_1)s]
[parameters: {'last_update': datetime.datetime(2023, 8, 4, 19, 5, 38, 784220), 'amphora_id_1': '467e8b0b-8314-4936-ae85-a039643e26c8'}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

Same problem can breaks Nova/Neutron/Cinder Service heartbeat.

Revision history for this message
Michal Nasiadka (mnasiadka) wrote :

Can you please post:
Kolla-Ansible version
MariaDB version
MariaDB log with relevant entries

Changed in kolla:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.