Autopilot Log Analyser

Exception: Failed to start mysql (max retries reached)

Bug #1596737 reported by Andreas Hasenack on 2016-06-27

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	Autopilot Log Analyser	Fix Committed	Undecided	Andreas Hasenack
	percona-cluster (Juju Charms Collection)	Fix Released	Medium	David Ames	Juju Charms Collection 16.07

Bug Description

cs:trusty/percona-cluster-243
landscape CI run reference: https://ci.lscape.net/job/landscape-system-tests/2884/

Got this while doing a mitaka/trusty cloud deployment.

I couldn't find a particular reason for this failure. This just repeats until it gives up:
2016-06-22 09:39:26 WARNING juju-log cluster:2: Option 'dataset-size' has been deprecated, please useinnodb_buffer_pool_size option instead
2016-06-22 09:39:26 INFO juju-log cluster:2: Writing file /etc/mysql/my.cnf root:root 444
2016-06-22 09:39:27 INFO cluster-relation-changed * Stopping MySQL (Percona XtraDB Cluster) mysqld
2016-06-22 09:39:34 INFO cluster-relation-changed ...done.
2016-06-22 09:39:34 INFO cluster-relation-changed * Starting MySQL (Percona XtraDB Cluster) database server mysqld
2016-06-22 09:40:09 INFO cluster-relation-changed * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
2016-06-22 09:40:09 INFO cluster-relation-changed ...fail!
2016-06-22 09:40:09 WARNING juju-log cluster:2: Failed to start mysql - retrying in 1s
2016-06-22 09:40:10 INFO cluster-relation-changed * Stopping MySQL (Percona XtraDB Cluster) mysqld
2016-06-22 09:40:10 INFO cluster-relation-changed ...done.
2016-06-22 09:40:10 INFO cluster-relation-changed * Starting MySQL (Percona XtraDB Cluster) database server mysqld
2016-06-22 09:40:48 INFO cluster-relation-changed * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
2016-06-22 09:40:48 INFO cluster-relation-changed ...fail!
(...)
2016-06-22 09:43:45 INFO cluster-relation-changed Traceback (most recent call last):
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 686, in <module>
2016-06-22 09:43:45 INFO cluster-relation-changed main()
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 679, in main
2016-06-22 09:43:45 INFO cluster-relation-changed hooks.execute(sys.argv)
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-06-22 09:43:45 INFO cluster-relation-changed self._hooks[hook_name]()
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 340, in cluster_changed
2016-06-22 09:43:45 INFO cluster-relation-changed config_changed()
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 81, in _harden_inner2
2016-06-22 09:43:45 INFO cluster-relation-changed return f(*args, **kwargs)
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 274, in config_changed
2016-06-22 09:43:45 INFO cluster-relation-changed render_config_restart_on_changed(clustered, hosts)
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 194, in render_config_restart_on_changed
2016-06-22 09:43:45 INFO cluster-relation-changed raise Exception("Failed to start mysql (max retries "
2016-06-22 09:43:45 INFO cluster-relation-changed Exception: Failed to start mysql (max retries reached)
2016-06-22 09:43:45 ERROR juju.worker.uniter.operation runhook.go:107 hook "cluster-relation-changed" failed: exit status 1

full logs attached.

Tags:

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2016-06-27:

percona-units-logs.tar.bz2 Edit (578.7 KiB, application/x-tar)

🤖 Landscape Builder (landscape-builder) on 2016-06-28

tags:

removed: kanban-cross-team

Andreas Hasenack (ahasenack) on 2016-06-30

Changed in autopilot-log-analyser:
status:	New → In Progress
assignee:	nobody → Andreas Hasenack (ahasenack)

Andreas Hasenack (ahasenack) on 2016-07-01

Changed in autopilot-log-analyser:
status:	In Progress → Fix Committed

Revision history for this message

David Ames (thedac) wrote on 2016-07-12:

Andreas,

I was able to re-create this failure with the stable version of percona-cluster.

I was also able to work around it by setting min-cluster-size. Can you test with min-cluster-size set?

I will be checking if this is present in the next version which will be released in a couple of weeks.

Changed in percona-cluster (Juju Charms Collection):
status:	New → Triaged
importance:	Undecided → Medium

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2016-07-12:

Hm, we do set min-cluster-size:
    def _configure_percona_ha(self):
        """Configure the percona-cluster service for HA."""
        self.available_services["mysql"].config["min-cluster-size"] = str(
            HA_COUNT)

HA_COUNT is 3. Unfortunately I don't have a "juju get mysql" to confirm that, but this code hasn't changed in a long while.

David Ames (thedac) on 2016-07-13

Changed in percona-cluster (Juju Charms Collection):
status:	Triaged → In Progress
assignee:	nobody → David Ames (thedac)

Revision history for this message

David Ames (thedac) wrote on 2016-07-14:

I did loop testing to confirm this. Where I was able to recreate with issue with stable in 2 or 3 loops with or without setting min-cluster-size the master branch does not show this problem.
It will be released with 16.07.

Changed in percona-cluster (Juju Charms Collection):
milestone:	none → 16.07
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-07-14: Change abandoned on charm-percona-cluster (stable/16.04)

Change abandoned by David Ames (<email address hidden>) on branch: stable/16.04
Review: https://review.openstack.org/340613
Reason: Fix to this bug is in master and will be release in 16.07. Abandoning this PR.

Liam Young (gnuoy) on 2016-07-29