Galera bootstrap is broken when starting on multinode

Bug #1479970 reported by Sam Yaple
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Sam Yaple

Bug Description

All-In-One works
All-In-One scaled to Multinode works
Initial creations as Multinode is broken

There is a timing issue. This can be solved with a few options added to the bootstrap process

Sam Yaple (s8m)
Changed in kolla:
milestone: liberty-2 → liberty-3
milestone: liberty-3 → liberty-2
Revision history for this message
Harm Weites (harmw) wrote :

Not sure if it's related, but I saw this tonight on support02:

Result from run 10 is: {'msg': 'Traceback (most recent call last):\r\n File "/root/.ansible/tmp/ansible-tmp-1438368697.17-26433943977046/mysql_user", line 2253, in <module>\r\n main()\r\n File "/root/.ansible/tmp/ansible-tmp-1438368697.17-26433943977046/mysql_user", line 498, in main\r\n if user_exists(cursor, user, host):\r\n File "/root/.ansible/tmp/ansible-tmp-1438368697.17-26433943977046/mysql_user", line 175, in user_exists\r\n cursor.execute("SELECT count(*) FROM user WHERE user = %s AND host = %s", (user,host))\r\n File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute\r\n self.errorhandler(self, exc, value)\r\n File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\r\n raise errorclass, errorvalue\r\n_mysql_exceptions.OperationalError: (1047, \'WSREP has not yet prepared node for application use\')\r\nOpenSSH_6.4, OpenSSL 1.0.1e-fips 11 Feb 2013\r\ndebug1: Reading configuration data /home/vagrant/.ssh/config\r\ndebug1: /home/vagrant/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 51: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: mux_client_request_session: master session id: 2\r\nShared connection to support02 closed.\r\n', 'failed': True, 'attempts': 10, 'parsed': False}

Where support01 and 03 would show this:

Result from run 10 is: {'msg': 'unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials', 'failed': True, 'attempts': 10}

Apparently mariadb is only active on support02 (which may be on purpose).

Changed in kolla:
status: Triaged → In Progress
Revision history for this message
Sam Yaple (s8m) wrote :

harmw, this issue does present like that with multinode

Try the patch once it merges and see if it is still a problem.

Revision history for this message
Sam Yaple (s8m) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/208066
Committed: https://git.openstack.org/cgit/stackforge/kolla/commit/?id=d32ed35d13bbc4ab5f5e6a1a146e1bc8d111ac88
Submitter: Jenkins
Branch: master

commit d32ed35d13bbc4ab5f5e6a1a146e1bc8d111ac88
Author: Sam Yaple <email address hidden>
Date: Fri Jul 31 21:50:43 2015 +0000

    Use sleep during database bootstrap

    A small sleep is used to allow the database to initialize before
    starting the databases on the other hosts.

    Change-Id: I1ea9ee8be55f47ae917f9985431e1e919f64c101
    Closes-Bug: #1479970

Changed in kolla:
status: In Progress → Fix Committed
Sam Yaple (s8m)
Changed in kolla:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.