Fuel for OpenStack

MySQL OCF RA sst in progress logic is broken

Bug #1574747 reported by Bogdan Dobrelya
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Bogdan Dobrelya
Fuel for OpenStack 10.0
6.1.x
Won't Fix
High
MOS Maintenance
Fuel for OpenStack 6.1-updates
7.0.x
Won't Fix
High
MOS Maintenance
Fuel for OpenStack 7.0-updates
8.0.x
Won't Fix
High
MOS Maintenance
Fuel for OpenStack 8.0-updates
Mitaka
Fix Released
High
Bogdan Dobrelya
Fuel for OpenStack 9.0
Newton
Fix Committed
High
Bogdan Dobrelya
Fuel for OpenStack 10.0

Bug Description

If mysqld crashed, the file /var/lib/mysql/sst_in_progress isn't be removed. This makes the resource remain stopped for ever and logs
will show "SST is in progress" as if everything were OK:

Apr 25 16:24:54 n4 crmd[156]: notice: process_lrm_event: Operation p_mysql_start_0: ok (node=n4, call=611, rc=0, cib-update=861, confirmed=true)
Apr 25 16:25:25 n4 ocf-mysql-wss: INFO: p_mysql: check_if_sst(): SST is in progress
Apr 25 16:25:25 n4 crmd[156]: notice: process_lrm_event: Operation p_mysql_monitor_10000: ok (node=n4, call=612, rc=0, cib-update=862, confirmed=false)
Apr 25 16:25:36 n4 ocf-mysql-wss: INFO: p_mysql: check_if_sst(): SST is in progress

See original description
Bogdan Dobrelya (bogdando)
Changed in fuel:
importance: Undecided → High
milestone: none → 10.0
tags: added: area-library galera
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/310020

Changed in fuel:
assignee: nobody → Bogdan Dobrelya (bogdando)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310110

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/310020
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7232dd03e16f448685066f360fec0b6b231d85e5
Submitter: Jenkins
Branch: master

commit 7232dd03e16f448685066f360fec0b6b231d85e5
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Apr 25 10:47:30 2016 +0200

    Fix mysql ocf sst in progress tracking

    Closes-bug: #1574747

    Change-Id: I070c97267cde7ed3cffde93ced334b6776079f4f
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/310110
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=19fe98b584a3e2916ce76ad61f4e6ebc6e5599ed
Submitter: Jenkins
Branch: stable/mitaka

commit 19fe98b584a3e2916ce76ad61f4e6ebc6e5599ed
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Apr 25 10:47:30 2016 +0200

    Fix mysql ocf sst in progress tracking

    Closes-bug: #1574747

    Change-Id: I070c97267cde7ed3cffde93ced334b6776079f4f
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/315989

Bogdan Dobrelya (bogdando)
tags: added: tech-debt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message
Alexander Rubtsov (arubtsov) wrote :

sla1 for 7.0-updates

tags: added: customer-found sla1
TatyanaGladysheva (tgladysheva)
tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.0 ISO #465.

Steps to verify:
1. Create an HA environment
2. Import 14Gb of data into the mysql database
3. Kill mysqld_safe and mysqld process and clear /var/lib/mysql folder (rm /var/lib/mysql/* -rf)
4. Check status of mysql process: ps axuf | grep mysql and pay attention on content of folder /var/lib/mysql/
5. Check state of mysql in pacemaker using 'crm status' command
6. After some time when full state transfer is completed, check /var/lib/mysql folder and observe logs (/var/log/daemon.log | grep ocf-mysql-wss)

Actual result:
If to kill mysql processes and clear /var/lib/mysql folder, full state transfer is started and 'sst_in_progress' file appears in /var/lib/mysql folder.
Mysql resourse has 'started' state on all controllers (step 5) in this case.
Log contains the following lines during transfer:
<30>Jun 16 10:24:18 node-29 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): SST is in progress

When full state transfer is completed, 'sst_in_progress' file disappears in folder. Log contains the following lines:
<30>Jun 16 10:31:20 node-29 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): No signs of SST found
<30>Jun 16 10:31:20 node-29 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL PID found
<30>Jun 16 10:31:20 node-29 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL PID found

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 465
cat /etc/fuel_build_number:
 465
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8454.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8454.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-notify-9.0.0-1.mos8454.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8454.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8742.noarch
 fuel-library9.0-9.0.0-1.mos8454.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos935.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8742.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 fuel-openstack-metadata-9.0.0-1.mos8742.noarch

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/8.0)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/317978

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/7.0)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/316802

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/6.1)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/315989

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Won't Fix for 6.1-, 7.0-, 8.0-updates as this is too big change to be accepted to stable branch

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/374219

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/7.0)

Reviewed: https://review.openstack.org/374219
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f9a2d479f3687157d2b17a927a09ce5f995522d6
Submitter: Jenkins
Branch: stable/7.0

commit f9a2d479f3687157d2b17a927a09ce5f995522d6
Author: Denis Puchkin <email address hidden>
Date: Wed Sep 21 17:38:54 2016 +0300

    Backport mysql OCF from stable/mitaka

    backport mysql ocf script from stable/mitaka

    Closes-bug: #1524826
    Closes-bug: #1542256
    Closes-bug: #1572239
    Closes-bug: #1572557
    Closes-bug: #1572601
    Closes-bug: #1574747
    Closes-bug: #1574497
    Closes-bug: #1576244
    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779
    Closes-bug: #1574999
    Closes-bug: #1576244
    Closes-bug: #1583173
    Closes-bug: #1585125

    Change-Id: I1cc6f95884a8fbd5c3418ede89bdf9ec6864bdc8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/377597

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/377597
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b3873f5f5a0bb1526b1269f163223ae48d6e21f5
Submitter: Jenkins
Branch: stable/8.0

commit b3873f5f5a0bb1526b1269f163223ae48d6e21f5
Author: Denis Puchkin <email address hidden>
Date: Tue Sep 27 13:20:25 2016 +0300

    Backport mysql OCF from stable/mitaka

    backport mysql ocf script from stable/mitaka

    Closes-bug: #1524826
    Closes-bug: #1542256
    Closes-bug: #1572239
    Closes-bug: #1572557
    Closes-bug: #1572601
    Closes-bug: #1574747
    Closes-bug: #1574497
    Closes-bug: #1576244
    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779
    Closes-bug: #1574999
    Closes-bug: #1576244
    Closes-bug: #1583173
    Closes-bug: #1585125

    Change-Id: I1cc6f95884a8fbd5c3418ede89bdf9ec6864bdc8

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.