Fuel for OpenStack

Deployment fails on Task[primary-cluster-haproxy/1] after 1800 seconds timeout

Bug #1615488 reported by Tatyana Kuterina on 2016-08-22

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Maksim Malchuk	Fuel for OpenStack 10.0
	Mitaka	Fix Released	High	Maksim Malchuk	Fuel for OpenStack 9.2

Bug Description

The same error with MOS packages on SWARM:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multirole/61/testReport/(root)/deploy_multiple_services_local_mirror/

=========================
Error with UCA packages:
https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.deploy_from_uca/4/console

Test: Deploy cluster in ha mode with UCA repo
Test Group: uca_neutron_ha

Steps to reproduce:
        1. Create cluster using UCA release
        2. Add 3 nodes with controller role
        3. Add 2 nodes with compute+cinder role
        4. Run network verification
        5. Deploy the cluster

Expected results:
Deploy finished without errors

Actual result:
Deployment has failed. All nodes are finished. Failed tasks: Task[primary-cluster-haproxy/1] Stopping the deployment process!

In log for Error node:
(/Stage[main]/Cluster::Haproxy/Service[haproxy]/ensure) change from stopped to running failed: Execution timeout after 1800 seconds!

Description of the environment:

9.1 snapshot #153

FUEL_QA_COMMIT=b525749639a1880f7663bee0513cc541e3a3c7cf
UBUNTU_MIRROR_ID=ubuntu-2016-08-03-174238
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-08-18-162321
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-08-18-162321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNMGUwdFd2VnoxZlU/view?usp=sharing

See original description

Tags:

Revision history for this message

Tatyana Kuterina (tkuterina) wrote on 2016-08-22:

Two other test are failed with the same errors:
Deploy cluster with NeutronTUN, Ceph and UCA repo
Deploy cluster with NeutronVlan, Ceilometer and UCA repo

There are errors in trace :
In syslog:
<27>Aug 19 13:06:31 node-1 ocf-ns_haproxy: ERROR: stdin: is not a tty [ALERT] 231/130631 (18193) : parsing [/etc/haproxy/haproxy.cfg:10] : unknown keyword 'ssl-default-bind-options' in 'global' section [ALERT] 231/130631 (18193) : parsing [/etc/haproxy/haproxy.cfg:12] : unknown keyword 'ssl-default-server-options' in 'global' section [ALERT] 231/130631 (18193) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 231/130631 (18193) : Fatal errors found in configuration.
<27>Aug 19 13:06:31 node-1 ocf-ns_haproxy: ERROR: Error. haproxy daemon returned error 0.

In osf-ns_haproxy.log:
2016-08-19T12:41:06.713839+00:00 err: ERROR: stdin: is not a tty [ALERT] 231/124106 (24183) : parsing [/etc/haproxy/haproxy.cfg:10] : unknown keyword 'ssl-default-bind-options' in 'global' section [ALERT] 231/124106 (24183) : parsing [/etc/haproxy/haproxy.cfg:12] : unknown keyword 'ssl-default-server-options' in 'global' section [ALERT] 231/124106 (24183) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 231/124106 (24183) : Fatal errors found in configuration.
2016-08-19T12:41:06.717476+00:00 err: ERROR: Error. haproxy daemon returned error 0.
2016-08-19T12:41:06.821142+00:00 info: INFO: haproxy daemon is not running

https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNWENBMmJRMEpzQTQ/view?usp=sharing
https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNT1BZdHNwekRRQU0/view?usp=sharing

Dmitry Guryanov (dguryanov) on 2016-08-22

Changed in fuel:
status:	New → Confirmed
importance:	Undecided → High
assignee:	nobody → Fuel Sustaining (fuel-sustaining-team)
milestone:	none → 10.0

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-08-22:

Tatyana, this deploy failed because installed old haproxy package, this is caused because -proposed repo not added.

Changed in fuel:
assignee:	Fuel Sustaining (fuel-sustaining-team) → Fuel QA Team (fuel-qa)
tags:	added: area-qa

Sergey Shevorakov (sshevorakov) on 2016-08-29

tags:

added: blocker-for-qa

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-08-29:

The issue in incorrect configuration of update repositories.

Could you please check that you use all required repositories for MOS 9.1 updates and move this issue to Confirmed if it will be reproduced again.

We saw the same issue when was trying to deploy 9.1 manually or with our custom scripts. Now it works fine with fuel-qa code from stable/mitaka branch.

Changed in fuel:
assignee:	Fuel QA Team (fuel-qa) → Tatyana Kuterina (tkuterina)
status:	Confirmed → Incomplete

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-08-29:

https://mirantis.jira.com/wiki/pages/viewpage.action?pageId=298160033

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-09-07:

Marked as invalid because we saw the same issue before and now it is fixed.

Could you please follow to the documentation:

https://mirantis.jira.com/wiki/pages/viewpage.action?pageId=298160033

and make sure that everything works fine now?

Please feel free to move it to Confirmed state if the issue is still reproduce.

Thank you!

Changed in fuel:
status:	Incomplete → Invalid

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-09-12:

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.deploy_from_uca/8/testReport/(root)/uca_neutron_tun_ceph/uca_neutron_tun_ceph/

Revision history for this message

Alexander Kurenyshev (akurenyshev) wrote on 2016-09-14:

Reopened this bug, because of reproduce above and local reproduce during a verification of https://bugs.launchpad.net/fuel/+bug/1594387

root@node-3:~# dpkg -l | grep haproxy
ii haproxy 1.5.3-1~u14.04+mos2 amd64 fast and reliable load balancing reverse proxy

repo used http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-09-14-110333/pool/main/h/haproxy/

Changed in fuel:
status:	Invalid → Confirmed
assignee:	Tatyana Kuterina (tkuterina) → Fuel Sustaining (fuel-sustaining-team)

Maksim Malchuk (mmalchuk) on 2016-09-14

Changed in fuel:
assignee:	Fuel Sustaining (fuel-sustaining-team) → Maksim Malchuk (mmalchuk)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-14: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/370124

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-14: Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/370126

Revision history for this message

Andrey Maximov (maximov) wrote on 2016-09-15:

#10

Moving to 9.2 since UCA is not supported as our primary use-case.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-15: Fix merged to fuel-library (master)

#11

Reviewed: https://review.openstack.org/370124
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3054fc4b532dcb132bb2214c448f8bea88bea9b0
Submitter: Jenkins
Branch: master

commit 3054fc4b532dcb132bb2214c448f8bea88bea9b0
Author: Maksim Malchuk <email address hidden>
Date: Wed Sep 14 16:31:53 2016 +0300

Pin to 1.6.3 mos haproxy in case of UCA

Fuel-library contain changes depends on 1.6+ version of the HAProxy
and this is also covers UCA case.

    Change-Id: I9cb3f8b3320f779aaa8f3fd69f62160771d82db3
    Closes-Bug: #1615488
    Signed-off-by: Maksim Malchuk <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

ElenaRossokhina (esolomina) wrote on 2016-09-16:

#12

swarm failure: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multirole/61/testReport/(root)/deploy_multiple_services_local_mirror/

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-09-16:

#13

Elena, test case should be removed from the swarm, due to the comment #10

Revision history for this message

Nastya Urlapova (aurlapova) wrote on 2016-09-16:

#14

@Maksim, I hope you can find that Elena's link is about case in SWARM. For SWARM testing we don't use the UCA packages!

description:

updated

Nastya Urlapova (aurlapova) on 2016-09-16

tags:

removed: area-qa

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2016-09-16:

#15

@Nastya, thanks but Elena already knows about the issue related to the fuel-mirror. Her link not related to this bug.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-16: Related fix proposed to fuel-qa (master)

#16

Related fix proposed to branch: master
Review: https://review.openstack.org/371506

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-19: Related fix merged to fuel-qa (stable/mitaka)

#17

Reviewed: https://review.openstack.org/370591
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7f5aa1a71b2d18ed13bccd50f47326379a70c5eb
Submitter: Jenkins
Branch: stable/mitaka

commit 7f5aa1a71b2d18ed13bccd50f47326379a70c5eb
Author: Maksim Malchuk <email address hidden>
Date: Thu Sep 15 10:54:50 2016 +0300

Fix replace_default_repos to use correct releases

    This change removes hardcoded unavailable releases and makes the
    replace_default_repos function more flexible to support only needed
    and available releases. The previous version didn't work with UCA
    environments.

    Change-Id: Iec53c695037b08ee26e8e6c07fbf1dfacfcadde8
    Closes-Bug: #1624295
    Related-Bug: #1615488
    Signed-off-by: Maksim Malchuk <email address hidden>

tags:

added: in-stable-mitaka

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-19: Related fix merged to fuel-qa (master)

#18

Reviewed: https://review.openstack.org/371506
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=b12ad32816ee822282735fb02d8f1db8fe3d629e
Submitter: Jenkins
Branch: master

commit b12ad32816ee822282735fb02d8f1db8fe3d629e
Author: Maksim Malchuk <email address hidden>
Date: Thu Sep 15 10:54:50 2016 +0300

Fix replace_default_repos to use correct releases

    Change-Id: Iec53c695037b08ee26e8e6c07fbf1dfacfcadde8
    Closes-Bug: #1624295
    Related-Bug: #1615488
    Signed-off-by: Maksim Malchuk <email address hidden>