Fuel for OpenStack

HA scalability doesn't work

Bug #1312627 reported by Nastya Urlapova
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Dmitry Ilyin
Fuel for OpenStack 5.1

Bug Description

{
build_id: "2014-04-24_15-01-23",
mirantis: "yes",
build_number: "135",
nailgun_sha: "dcaed120b34dd2c8cf817463a243eedfb844f096",
production: "prod",
ostf_sha: "134765fcb5a07dce0cd1bb399b2290c988c3c63b",
fuelmain_sha: "387d2e931ee14d5773f21f1e2860a13b50b37d04",
astute_sha: "6e8fa4cc12968d7b468fc590b2f06bb59bf74511",
release: "5.0",
fuellib_sha: "fa57b98344e7cdd342b7c08a5e692525b102af8b"
}

1) deploy 1xController
2) add 2xControllers or 5xController - redeploy changes

 (/Stage[main]/Openstack::Ha::Mysqld/Exec[wait-for-haproxy-mysql-backend]/returns) change from notrun to 0 failed: echo show stat | socat unix-connect:///var/lib/haproxy/stats stdio | grep -q '^mysqld,BACKEND,.*,UP,' returned 1 instead of one of [0]
2014-04-25 09:28:56 ERR
 /usr/bin/puppet:4
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `perform_changes'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `each'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:82:in `perform_changes'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:120:in `sync'
2014-04-25 09:28:56 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:97:in `fail'
2014-04-25 09:28:56 ERR
 echo show stat | socat unix-connect:///var/lib/haproxy/stats stdio | grep -q '^mysqld,BACKEND,.*,UP,' returned 1 instead of one of [0]

Revision history for this message
Nastya Urlapova (aurlapova) wrote :
summary: - HIGH scalability doesn't work
+ HA scalability doesn't work
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
milestone: none → 5.0
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

it looks like the problem is because of corosync restart due to config change because we are using unicast for corosync

Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergii Golovatiuk (sgolovatiuk)
Mike Scherbakov (mihgen)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Sergey Vasilenko (xenolog) wrote :

NOTE:
This check by socat should been executed only on the first deploing controller.
And shouldn't while redeploy on any.

Revision history for this message
Sergii Golovatiuk (sgolovatiuk) wrote :

Fuel implementation uses static corosync.conf where all nodes are described. In order to have dynamic node configuration there should be another layer of abstraction CMAN or pcsd. CMAN works with corosync 1.x and has been deprecated in corosync 2.x. pcsd has some problems with ubuntu and designed for corosync 2.0 which requires redesign of puppet manifests.

Changed in fuel:
status: Confirmed → Triaged
status: Triaged → Confirmed
Revision history for this message
Mike Scherbakov (mihgen) wrote :

> which requires redesign of puppet manifests.
looks like large change to have. Postponing to 5.1.

Changed in fuel:
milestone: 5.0 → 5.1
tags: added: release-notes
Revision history for this message
Meg McRoberts (dreidellhasa) wrote :

Added to Known Issues in 5.0 Release Notes.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

depends on https://blueprints.launchpad.net/fuel/+spec/ha-pacemaker-improvements

Bogdan Dobrelya (bogdando)
tags: added: to-be-covered-by-tests
Revision history for this message
Meg McRoberts (dreidellhasa) wrote :

Marked as "Known Issue" in 5.0.1 Release Notes.

Revision history for this message
Dmitry Ilyin (idv1985) wrote :

FIx commited with pacemaker improvements

Changed in fuel:
status: Confirmed → Fix Committed
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #347
"build_id": "2014-07-23_02-01-14",
"ostf_sha": "c1b60d4bcee7cd26823079a86e99f3f65414498e",
"build_number": "347",
"auth_required": false,
"api": "1.0",
"nailgun_sha": "f5775d6b7f5a3853b28096e8c502ace566e7041f",
"production": "docker",
"fuelmain_sha": "74b9200955201fe763526ceb51607592274929cd",
"astute_sha": "fd9b8e3b6f59b2727b1b037054f10e0dd7bd37f1",
"feature_groups": ["mirantis"],
"release": "5.1",
"fuellib_sha": "fb0e84c954a33c912584bf35054b60914d2a2360"

Changed in fuel:
status: Fix Committed → Fix Released
Vladimir Kuklin (vkuklin)
Changed in fuel:
status: Fix Released → In Progress
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
assignee: Sergii Golovatiuk (sgolovatiuk) → Dmitry Ilyin (idv1985)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

addressed by fixing https://bugs.launchpad.net/fuel/+bug/1340172

Changed in fuel:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.