Mysql die after deployment and can not be recovered near 40 minutes, that leads to 500 error from different services

Bug #1542256 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Sergii Golovatiuk
6.1.x
Won't Fix
High
MOS Maintenance
7.0.x
Won't Fix
High
MOS Maintenance
8.0.x
Won't Fix
High
MOS Maintenance

Bug Description

Problem appears om next test:
Deploy cluster in ha multirole mode with Ceilometer

Scenario:
1. Create cluster. Set install Ceilometer option
2. Add 3 node with controller and mongo roles
3. Add 1 nodes with compute role
4. Add 1 nodes with cinder
5. Deploy the cluster
6. Verify ceilometer api is running
7. Run OSTF

Expected Result:
All OSTF tests are passed

Actual:
Smoke, sanity, ha -tests are passed, but Ceilometer tests failed with 50x errors from neutron:
http://paste.openstack.org/show/486065/

in neutron log we can see that there is no connection to mysql
http://paste.openstack.org/show/486067/

in mysql/error log there was failure:

http://paste.openstack.org/show/486070/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "523"
  build_id: "523"
  fuel-nailgun_sha: "9cfaee884d5a2a7304905f29bde57adebffa7577"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "71d3a7f43effd457391a1f02558fba4f0ca90eec"
  fuel-ostf_sha: "7bcddf18020f2d94a553a441ff57dff9632865df"
  fuel-mirror_sha: "c25d8931e30322ecf43246c8a957e376259b685c"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

Revision history for this message
Sergii Golovatiuk (sgolovatiuk) wrote :

As stated in https://bugs.launchpad.net/fuel/+bug/1541338

There is a bug in MySQL OCF stop function. It doesn't kill the process in normal amount of time. However, this bug leads to issues only with one controller as in 3 controller environment haproxy will switch traffic to 'working' instance of mysql on different controller.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergii Golovatiuk (sgolovatiuk)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
status: New → Confirmed
tags: added: team-bugfix
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

We are in HCF state for 8.0. Only critical fixes are allowed to be merged. Targeting bug to maintenance update.

Changed in fuel:
milestone: 8.0 → 9.0
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Folks, the thing that I see here is that MySQL fails with segfault. We need low-level folks in here to look further.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/280247

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/280247
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=615a0428e3250cffadbf5ebf61e477e91858cb16
Submitter: Jenkins
Branch: master

commit 615a0428e3250cffadbf5ebf61e477e91858cb16
Author: Sergii Golovatiuk <email address hidden>
Date: Tue Feb 9 12:22:17 2016 +0100

    Refactor MySQL OCF script

    - Add LH/LL variables to all functions for extra logging
    - Fix mysql_status to report OCF_NOT_RUNNING when MySQL is stopped
      outside of pacemaker
    - Removing sst_in_progress verification from mysql_monitor as it's part
      of mysql_status
    - Added default loglevel to mysql_status
    - Removed unnecessary update_node_gtid && check_if_reelection_needed
      from mysql_start
    - Refactor mysql_stop. Removed custom stop function in favour of
      proc_stop() from ocf-fuel-funcs library

    Closes-Bug: 1542256
    Change-Id: I4e0d45b9c629c91643fec931be701e13ea5001bb
    Signed-off-by: Sergii Golovatiuk <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
tags: added: on-verification
Revision history for this message
ElenaRossokhina (esolomina) wrote :

Verified on iso#305
fuel info http://paste.openstack.org/show/496749/

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/316085

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/316803

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/317979

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/8.0)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/317979

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/7.0)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/316803

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/6.1)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/316085

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Won't Fix for 6.1-, 7.0- and 8.0-updates as this is too large change to be accepted to stable branch

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/374219

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/7.0)

Reviewed: https://review.openstack.org/374219
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f9a2d479f3687157d2b17a927a09ce5f995522d6
Submitter: Jenkins
Branch: stable/7.0

commit f9a2d479f3687157d2b17a927a09ce5f995522d6
Author: Denis Puchkin <email address hidden>
Date: Wed Sep 21 17:38:54 2016 +0300

    Backport mysql OCF from stable/mitaka

    backport mysql ocf script from stable/mitaka

    Closes-bug: #1524826
    Closes-bug: #1542256
    Closes-bug: #1572239
    Closes-bug: #1572557
    Closes-bug: #1572601
    Closes-bug: #1574747
    Closes-bug: #1574497
    Closes-bug: #1576244
    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779
    Closes-bug: #1574999
    Closes-bug: #1576244
    Closes-bug: #1583173
    Closes-bug: #1585125

    Change-Id: I1cc6f95884a8fbd5c3418ede89bdf9ec6864bdc8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/377597

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/377597
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b3873f5f5a0bb1526b1269f163223ae48d6e21f5
Submitter: Jenkins
Branch: stable/8.0

commit b3873f5f5a0bb1526b1269f163223ae48d6e21f5
Author: Denis Puchkin <email address hidden>
Date: Tue Sep 27 13:20:25 2016 +0300

    Backport mysql OCF from stable/mitaka

    backport mysql ocf script from stable/mitaka

    Closes-bug: #1524826
    Closes-bug: #1542256
    Closes-bug: #1572239
    Closes-bug: #1572557
    Closes-bug: #1572601
    Closes-bug: #1574747
    Closes-bug: #1574497
    Closes-bug: #1576244
    Closes-bug: #1574999
    Closes-bug: #1578278
    Closes-bug: #1388779
    Closes-bug: #1574999
    Closes-bug: #1576244
    Closes-bug: #1583173
    Closes-bug: #1585125

    Change-Id: I1cc6f95884a8fbd5c3418ede89bdf9ec6864bdc8

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.