Unwanted public vip failover at the end of deployment

Bug #1643510 reported by Michael Polenchuk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Michael Polenchuk
Mitaka
Fix Released
High
Unassigned
Newton
Fix Released
High
Michael Polenchuk

Bug Description

Detailed bug description:
 since ping_checker task has order post-deploy-start -> ... post-deploy-end, which leads to run ping_checker on controllers (not on primary) in some times and this in turn causes unwanted public vip failover.

Steps to reproduce:
 deploy HA environment

Expected results:
 failover shouldn't happen in the middle of deployment

Actual result:
 it happens sometimes

Workaround:
 Disable run_ping_checker option

Description of the environment:
 Fuel 9.x +

Tags: area-library
Changed in fuel:
milestone: none → 11.0
assignee: nobody → Michael Polenchuk (mpolenchuk)
importance: Undecided → High
status: New → Confirmed
tags: added: area-library
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/400203

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/400203
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8358ecc8875c57a58852fe86cea287ddcb5e4ec4
Submitter: Jenkins
Branch: master

commit 8358ecc8875c57a58852fe86cea287ddcb5e4ec4
Author: Michael Polenchuk <email address hidden>
Date: Mon Nov 21 14:24:04 2016 +0300

    Split public_vip_ping task out

    public_vip_ping task has been splitted out to avoid unwanted failover when
    ping_checker resource might be started up on controllers before a primary one.
    In addition pingd constraint has been changed to be based on score.
    If all the nodes failed to ping the default gw, the resource
    ain't stopped and remains on the last node where it was.

    Change-Id: Ic02fc33f323f75c0b47e78e0935842dd9ae5a27d
    Closes-Bug: #1643510

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/410589

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/410714

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/410589
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=827da22aee47bc8126ff2ecbc4868747f7d1f38b
Submitter: Jenkins
Branch: stable/newton

commit 827da22aee47bc8126ff2ecbc4868747f7d1f38b
Author: Michael Polenchuk <email address hidden>
Date: Mon Nov 21 14:24:04 2016 +0300

    Split public_vip_ping task out

    public_vip_ping task has been splitted out to avoid unwanted failover when
    ping_checker resource might be started up on controllers before a primary one.
    In addition pingd constraint has been changed to be based on score.
    If all the nodes failed to ping the default gw, the resource
    ain't stopped and remains on the last node where it was.

    Change-Id: Ic02fc33f323f75c0b47e78e0935842dd9ae5a27d
    Closes-Bug: #1643510
    (cherry picked from commit 8358ecc8875c57a58852fe86cea287ddcb5e4ec4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/410714
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=de4dcf305c48b857f7e41f0d259022a068ec082e
Submitter: Jenkins
Branch: stable/mitaka

commit de4dcf305c48b857f7e41f0d259022a068ec082e
Author: Michael Polenchuk <email address hidden>
Date: Mon Nov 21 14:24:04 2016 +0300

    Split public_vip_ping task out

    public_vip_ping task has been splitted out to avoid unwanted failover when
    ping_checker resource might be started up on controllers before a primary one.
    In addition pingd constraint has been changed to be based on score.
    If all the nodes failed to ping the default gw, the resource
    ain't stopped and remains on the last node where it was.

    Change-Id: Ic02fc33f323f75c0b47e78e0935842dd9ae5a27d
    Closes-Bug: #1643510
    (cherry picked from commit 8358ecc8875c57a58852fe86cea287ddcb5e4ec4)

tags: added: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-nailgun-9.0.0-1.mos8931.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 python-packetary-9.0.0-1.mos161.noarch
 nailgun-mcagents-9.0.0-1.mos784.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos364.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8931.noarch
 fuel-mirror-9.0.0-1.mos161.noarch
 fuel-notify-9.0.0-1.mos8685.noarch
 rubygem-astute-9.0.0-1.mos784.noarch
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-misc-9.0.0-1.mos8685.noarch
 fuel-openstack-metadata-9.0.0-1.mos8931.noarch
 fuel-migrate-9.0.0-1.mos8685.noarch
 fuel-library9.0-9.0.0-1.mos8685.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-ui-9.0.0-1.mos2854.noarch
 fuel-utils-9.0.0-1.mos8685.noarch
 fuel-9.0.0-1.mos6359.noarch

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc1

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.