During an upgrade from percona-cluster ~15.04 (r52, revision-id: <email address hidden>) to 17.11,the upgrade-charm hook failed as follows:
2018-02-08 03:01:44 INFO upgrade-charm _mysql_exceptions.OperationalError: (1045, "Access denied for user 'root'@'localhost' (using password: YES)")
I realized that all of the passwords for the various related services were in the peer relation:
stagingstack-is@wekufe:~$ alias jru='juju run --unit'
stagingstack-is@wekufe:~$ jru mysql/0 'relation-get -r $(relation-ids cluster) - mysql/0'
bootstrap-uuid: c5a4bbfa-0c79-11e8-9b00-bfea4352af21
mysql-cinder.passwd: 89S9qrK8nxGPrTYbsSW2J5P5zBZ5M2RS
mysql-glance.passwd: Fsw6NXfdnXjdMypGjSYhFRJPg9mdRBYp
mysql-keystone.passwd: t4H6YNn49hC45bnJ4NYrZNsmK3gP4fBb
mysql-neutron.passwd: LNFM4Xq48zBb8Mm5XShHGktHpTc2CVGm
mysql-nova.passwd: 74Vf7bVd6JLfhrrc97kYHp476ZN2tZHk
mysql-sstuser.passwd: notarealsstpassword
mysql.passwd: notarealrootpassword
private-address: 10.25.126.203
stagingstack-is@wekufe:~$ _
whereas the charm code was now looking at the leader settings, which had fewer and different passwords:
stagingstack-is@wekufe:~$ jru mysql/0 leader-get
bootstrap-uuid: c5a4bbfa-0c79-11e8-9b00-bfea4352af21
mysql.passwd: LtWcYLGWNT6tXpJyVb5WwHjzPMkjGZZt
sst-password: notarealsstpassword
stagingstack-is@wekufe:~$ _
Running the following before upgrading the charm avoids the problem:
jru mysql/0 'leader-set $(relation-get -r $(relation-ids cluster) - mysql/0 | grep passwd | sed -e "s/: /=/")'
Commit bda5d1508190286
That happened in March 2017, so I think the main issue here is the long jump between charm revisions for this specific upgrade. Automated testing will only ever upgrade from current stable to proposed stable so we don't cover this specific use-case.