HA cluster deployment failed with "Lost connection to MySQL server at 'reading initial communication packet', system error: 0"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Confirmed
|
High
|
Vladimir Kuklin |
Bug Description
{
"build_id": "2014-07-
"ostf_sha": "8c328521b1444f
"build_number": "366",
"auth_
"api": "1.0",
"nailgun_sha": "83cc9ed44ebc8d
"production": "docker",
"fuelmain_sha": "9adfbf5a52cedb
"astute_sha": "aa5aed61035a8d
"feature_
"mirantis"
],
"release": "5.1",
"fuellib_sha": "d1c7f67b3cf519
}
Steps:
1. Create next cluster - Ubuntu, HA, Neutron GRE, 3 controller, 2 compute
2. Deploy cluster
Expected - deployment passed
Actual - deployment failed with errors in neutron-server log on controller-1 (node-1):
neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
2014-07-28 11:55:31.404 23309 TRACE neutron.
Looks like issue with connection (logs are attached)
Changed in fuel: | |
importance: | Medium → High |
Changed in fuel: | |
status: | New → Incomplete |
Changed in fuel: | |
status: | Incomplete → Confirmed |
Changed in mos: | |
assignee: | nobody → Dmitry Mescheryakov (dmitrymex) |
milestone: | none → 5.1 |
status: | New → Confirmed |
importance: | Undecided → High |
no longer affects: | fuel |
This problem belongs to mysqldump which is used for SST for Galera. Here is the log
node-3: /var/log/ puppet. log main]/Galera/ Service[ mysql-service] (info): Starting to evaluate the resource
Mon Jul 28 11:49:54 +0000 2014 /Stage[
node-1: neutron-server.log 28T12:44: 15.293475+ 01:00 warning: 2014-07-28 11:44:15.292 23135 WARNING neutron. openstack. common. db.sqlalchemy. session [req-21cbcff2- c7e8-4352- 8784-3de095ee0d cc ] Database server has gone away: (2006, 'MySQL server has gone away')
2014-07-
Since node-1 was donor and non blocking SST (xtrabackup) was used this means that during SST MySQL was not available and was turned off by HAProxy clustercheck script. However, this didn't affect on neutron as it restored its operations after SST.