percona-cluster crashes during cluster formation

I can reproduce this issue in multiple labs.

Tried downloading the log mysql1.tar to review was unable to untar to trace the logs. Next time when I tried was not able to downlaod and old one mysql1.tar have deleted. Any way keep me posted as this on critical path for opnfv JOID project...

It looks like, from the logs, that PXC (Percona Xtradb Cluster Server) started ok on mysql0, but crashes on both mysql1 and mysql2.

More info from hosts hosting lxcs for mysql1 and mysql2 is required. It is not clear from the logs, but it could be that memory is constrained in a way that crashes mysql.

Have you tried redeploying with innodb-buffer-pool-size=16M and max-connections=200?

Also, inside mysql data directory (/var/lib/percona-xtradb-cluster/( there is innobackup.prepare.log, can we also get that?

Those are hosted on LXD container and most host system has 128 GB of RAM. Current requirement to deploy bundle with innodb-buffer-pool-size=16M and max-connections=200 might not work but let me give a try and see.

I have not found innobackup.backup.log please find the log requested. pastebinit innobackup.backup.log
http://paste.ubuntu.com/23994180/

with the small numbers innodb-buffer-pool-size=16M and max-connections=200 it seems HA bundle got deployed successfully. do i need to experiment and see what should be idle values? What kind of issues we might face?

Also i ran the deployment on two different labs. One filed and another pass and issue is. Definitely looks like percona-cluster bug.

170214 17:27:35 innobackupex: Executing LOCK BINLOG FOR BACKUP...
DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr//bin/innobackupex line 3059.
innobackupex: Error:
Error executing 'LOCK BINLOG FOR BACKUP': DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /usr//bin/innobackupex line 3059.
170214 17:27:35 innobackupex: Waiting for ibbackup (pid=28146) to finish

pastebinit < /var/lib/percona-xtradb-cluster//innobackup.backup.log
http://paste.ubuntu.com/23996333/

I also saw some crashes even with performance_schema=off, so LP: #1646795 and LP: #1401133 might be related.

My quick workaround was to set sst-method=rsync in charm config to avoid innobackupex route.

I also can confirm that with rsync route I am not seeing this issue.

Thanks and Regards,
Narinder Gupta (PMP) <email address hidden>
Canonical, Ltd. narindergupta [irc.freenode.net]
+1.281.736.5150 narindergupta2007[skype]

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Wed, Feb 15, 2017 at 12:38 AM, Nobuto Murata <email address hidden>
wrote:

> My quick workaround was to set sst-method=rsync in charm config to avoid
> innobackupex route.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1663245
>
> Title:
> percona-cluster crashes during cluster formation
>
> Status in OPNFV:
> New
> Status in percona-cluster package in Juju Charms Collection:
> New
>
> Bug description:
> When i ran the below bundle to deploy the percona cluster I am finding
> the percona cluster crashes and does not recover.
> http://paste.ubuntu.com/23943677/
>
> please find the logs attached as well from all three cluster units.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/opnfv/+bug/1663245/+subscriptions
>

I recently landed a change into the percona-cluster charm to cap the amount of ram that will be allocated to the innodb buffer to 512M (unless the end user provides a explicit configuration value for innodb-buffer-size).

I think this bug is related to consumption of 50% of RAM by default prior to this change - if the system is doing anything else, there is no guarantee that is even possible.

Narinder - I'm going to mark this as Fix Committed on the assumption that this is resolve by the improved default memory management introduced in the current development charm.

If you still have access to the systems it would be nice to re-try with cs:~openstack-charmers-next/percona-cluster to see if this improves things.

James I can confirm that after that fix I am not able to reproduce this issue anymore. And mysql does not crash and all services comes up.