Juju Charms Collection
percona-cluster package

Exception: Failed to start mysql (max retries reached)

Bug #1596737 reported by Andreas Hasenack
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Autopilot Log Analyser
Fix Committed
Undecided
Andreas Hasenack
percona-cluster (Juju Charms Collection)
Fix Released
Medium
David Ames
Juju Charms Collection 16.07

Bug Description

cs:trusty/percona-cluster-243
landscape CI run reference: https://ci.lscape.net/job/landscape-system-tests/2884/

Got this while doing a mitaka/trusty cloud deployment.

I couldn't find a particular reason for this failure. This just repeats until it gives up:
2016-06-22 09:39:26 WARNING juju-log cluster:2: Option 'dataset-size' has been deprecated, please useinnodb_buffer_pool_size option instead
2016-06-22 09:39:26 INFO juju-log cluster:2: Writing file /etc/mysql/my.cnf root:root 444
2016-06-22 09:39:27 INFO cluster-relation-changed * Stopping MySQL (Percona XtraDB Cluster) mysqld
2016-06-22 09:39:34 INFO cluster-relation-changed ...done.
2016-06-22 09:39:34 INFO cluster-relation-changed * Starting MySQL (Percona XtraDB Cluster) database server mysqld
2016-06-22 09:40:09 INFO cluster-relation-changed * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
2016-06-22 09:40:09 INFO cluster-relation-changed ...fail!
2016-06-22 09:40:09 WARNING juju-log cluster:2: Failed to start mysql - retrying in 1s
2016-06-22 09:40:10 INFO cluster-relation-changed * Stopping MySQL (Percona XtraDB Cluster) mysqld
2016-06-22 09:40:10 INFO cluster-relation-changed ...done.
2016-06-22 09:40:10 INFO cluster-relation-changed * Starting MySQL (Percona XtraDB Cluster) database server mysqld
2016-06-22 09:40:48 INFO cluster-relation-changed * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
2016-06-22 09:40:48 INFO cluster-relation-changed ...fail!
(...)
2016-06-22 09:43:45 INFO cluster-relation-changed Traceback (most recent call last):
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 686, in <module>
2016-06-22 09:43:45 INFO cluster-relation-changed main()
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 679, in main
2016-06-22 09:43:45 INFO cluster-relation-changed hooks.execute(sys.argv)
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-06-22 09:43:45 INFO cluster-relation-changed self._hooks[hook_name]()
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 340, in cluster_changed
2016-06-22 09:43:45 INFO cluster-relation-changed config_changed()
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 81, in _harden_inner2
2016-06-22 09:43:45 INFO cluster-relation-changed return f(*args, **kwargs)
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 274, in config_changed
2016-06-22 09:43:45 INFO cluster-relation-changed render_config_restart_on_changed(clustered, hosts)
2016-06-22 09:43:45 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/cluster-relation-changed", line 194, in render_config_restart_on_changed
2016-06-22 09:43:45 INFO cluster-relation-changed raise Exception("Failed to start mysql (max retries "
2016-06-22 09:43:45 INFO cluster-relation-changed Exception: Failed to start mysql (max retries reached)
2016-06-22 09:43:45 ERROR juju.worker.uniter.operation runhook.go:107 hook "cluster-relation-changed" failed: exit status 1

full logs attached.

Tags: landscape
Revision history for this message
Andreas Hasenack (ahasenack) wrote :
🤖 Landscape Builder (landscape-builder)
tags: removed: kanban-cross-team
Andreas Hasenack (ahasenack)
Changed in autopilot-log-analyser:
status: New → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
Andreas Hasenack (ahasenack)
Changed in autopilot-log-analyser:
status: In Progress → Fix Committed
Revision history for this message
David Ames (thedac) wrote :

Andreas,

I was able to re-create this failure with the stable version of percona-cluster.

I was also able to work around it by setting min-cluster-size. Can you test with min-cluster-size set?

I will be checking if this is present in the next version which will be released in a couple of weeks.

Changed in percona-cluster (Juju Charms Collection):
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hm, we do set min-cluster-size:
    def _configure_percona_ha(self):
        """Configure the percona-cluster service for HA."""
        self.available_services["mysql"].config["min-cluster-size"] = str(
            HA_COUNT)

HA_COUNT is 3. Unfortunately I don't have a "juju get mysql" to confirm that, but this code hasn't changed in a long while.

David Ames (thedac)
Changed in percona-cluster (Juju Charms Collection):
status: Triaged → In Progress
assignee: nobody → David Ames (thedac)
Revision history for this message
David Ames (thedac) wrote :

I did loop testing to confirm this. Where I was able to recreate with issue with stable in 2 or 3 loops with or without setting min-cluster-size the master branch does not show this problem.
It will be released with 16.07.

Changed in percona-cluster (Juju Charms Collection):
milestone: none → 16.07
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-percona-cluster (stable/16.04)

Change abandoned by David Ames (<email address hidden>) on branch: stable/16.04
Review: https://review.openstack.org/340613
Reason: Fix to this bug is in master and will be release in 16.07. Abandoning this PR.

Liam Young (gnuoy)
Changed in percona-cluster (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.