MySQL OCF RA action monitor must check if a seed node is running the most recent of known GTIDs
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Fuel for OpenStack |
Fix Committed
|
Medium
|
Bogdan Dobrelya | ||
| 7.0.x |
Fix Released
|
Critical
|
Denis Puchkin | ||
| 8.0.x |
Fix Released
|
Critical
|
Denis Puchkin | ||
| Mitaka |
Fix Released
|
Critical
|
Sergii Golovatiuk | ||
Bug Description
This bug looks not easy to catch up.
I caught it only a couple of times while was running jepsen tests for few days.
Details:
When the seed (aka master) node was started a long time ago, and later the OCF RA reports "MySQL lost quorum or uninitialized" on majority of the rest DB nodes, it ends up with either a *very* long auto-recovery time, or fails to recover at all.
Only the seed node keeps running, even if it has an obsolete GTID, which is not the most recent across the rest of the nodes. This requires a manual recovery of the DB cluster nodes. For example, one may "nuke" all mysqld on the nodes and allow the OCF RA to pick the most recent node. This provides sad UX, although should not be a big deal.
Example snippet (4/5 nodes was affected): http://
To fix that, perhaps monitor must check if the current seed node (aka master) is running with a bad GTID, which is not the most recent across the nodes, and report failure.
| Bogdan Dobrelya (bogdando) wrote | #1 |
| Changed in fuel: | |
| importance: | Undecided → Medium |
| milestone: | none → 10.0 |
| tags: | added: galera |
| summary: |
- MySQL OCF RA action monitor must check if a node is running the most - recent of known GTIDs + MySQL OCF RA action monitor must check if a seed node is running the + most recent of known GTIDs |
| Changed in fuel: | |
| status: | New → Triaged |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to fuel-library (master) | #2 |
| Changed in fuel: | |
| assignee: | nobody → Bogdan Dobrelya (bogdando) |
| status: | Triaged → In Progress |
| OpenStack Infra (hudson-openstack) wrote Fix merged to fuel-library (master) | #3 |
| Changed in fuel: | |
| status: | In Progress → Fix Committed |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to fuel-library (stable/mitaka) | #4 |
| Sergii Golovatiuk (sgolovatiuk) wrote | #5 |
| tags: | added: area-library |
| OpenStack Infra (hudson-openstack) wrote Fix merged to fuel-library (stable/mitaka) | #6 |
| tags: | added: ct2 customer-found sla1 support |
| tags: | added: on-verification |
| Dmitry Belyaninov (dbelyaninov) wrote | #7 |
| tags: | removed: on-verification |
| Alexander Rubtsov (arubtsov) wrote | #8 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to fuel-library (stable/7.0) | #9 |
| OpenStack Infra (hudson-openstack) wrote | #10 |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on fuel-library (stable/7.0) | #11 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to fuel-library (stable/7.0) | #12 |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on fuel-library (stable/7.0) | #13 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to fuel-library (stable/7.0) | #14 |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on fuel-library (stable/7.0) | #15 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to fuel-library (stable/7.0) | #16 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to fuel-library (stable/8.0) | #17 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to fuel-library (stable/8.0) | #18 |
| Dmitry (dtsapikov) wrote | #19 |
| tags: | added: on-verification |
| tags: | removed: on-verification |
| TatyanaGladysheva (tgladysheva) wrote | #20 |
Fix proposed to branch: master
Review: https:/