On Artful x86_64, percona-cluster shared-db-relation-changed hooks frequently error out, but not always. Various db migrate operations fail (sometimes keystone, sometimes glance).
Juju unit logs indicate db migrate operations were underway on the api unit, and system logs indicate mysql (percona-cluster) crashed on the db unit.
App Version Status Scale Charm Store Rev OS Notes
ceph 12.2.0 active 3 ceph local 105 ubuntu
ceph-osd 12.2.0 active 1 ceph-osd jujucharms 297 ubuntu
cinder 11.0.0 waiting 1 cinder jujucharms 298 ubuntu
cinder-ceph waiting 0/1 cinder-ceph jujucharms 247 ubuntu
glance 15.0.0 waiting 1 glance jujucharms 292 ubuntu
keystone 12.0.0 waiting 1 keystone jujucharms 322 ubuntu
nova-compute 16.0.1 active 1 nova-compute jujucharms 336 ubuntu
percona-cluster 5.6.34-26.19 error 1 percona-cluster jujucharms 274 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 280 ubuntu
Unit Workload Agent Machine Public address Ports Message
ceph-osd/0* active idle 3 172.17.106.17 Unit is ready (2 OSD)
ceph/0* active idle 0 172.17.106.10 Unit is ready and clustered
ceph/1 active idle 1 172.17.106.5 Unit is ready and clustered
ceph/2 active idle 2 172.17.106.3 Unit is ready and clustered
cinder/0* waiting executing 4 172.17.106.20 8776/tcp Incomplete relations: messaging, identity, database
cinder-ceph/0* waiting allocating 172.17.106.20 agent initializing
glance/0* waiting executing 5 172.17.106.22 9292/tcp Incomplete relations: identity, database
keystone/0* waiting executing 6 172.17.106.21 5000/tcp Incomplete relations: database
nova-compute/0* active executing 7 172.17.106.4 Unit is ready
percona-cluster/0* error idle 8 172.17.106.18 3306/tcp hook failed: "shared-db-relation-changed" for glance:shared-db
rabbitmq-server/0* active idle 9 172.17.106.26 5672/tcp Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 172.17.106.10 cc716b8d-f4ff-450d-b088-a3742a5ffffd artful nova ACTIVE
1 started 172.17.106.5 e0498107-15fe-4a5e-93a2-a1f6cfd9523b artful nova ACTIVE
2 started 172.17.106.3 6a02e6cc-2fdb-4b81-b4ab-0467c5e5705d artful nova ACTIVE
3 started 172.17.106.17 cc5a2f76-73a6-4f50-a97e-ae391ed96e63 artful nova ACTIVE
4 started 172.17.106.20 4d42c847-9dd3-4f1a-a668-0b5eb7819b9a artful nova ACTIVE
5 started 172.17.106.22 8dd50778-396b-4c27-b063-34ff63ddb3f2 artful nova ACTIVE
6 started 172.17.106.21 58eac09b-88d5-4f1b-a97d-b22176cc231f artful nova ACTIVE
7 started 172.17.106.4 534d23a0-5c30-451a-8963-68057bac27f8 artful nova ACTIVE
8 started 172.17.106.18 091cb17d-5898-44b4-9f0a-3bffbe808d05 artful nova ACTIVE
9 started 172.17.106.26 eb03b662-332e-48d5-8b5d-c39e3deadb4a artful nova ACTIVE
On Artful x86_64, percona-cluster shared-db-relation-changed hooks frequently error out, but not always. Various db migrate operations fail (sometimes keystone, sometimes glance).
Juju unit logs indicate db migrate operations were underway on the api unit, and system logs indicate mysql (percona-cluster) crashed on the db unit.
Artifacts:/openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceph/508041/4/743/consoleText.test_charm_amulet_full_1111.txt
https:/
percona-cluster/0* error idle 8 172.17.106.18 3306/tcp hook failed: "shared-db-relation-changed" for glance:shared-db
# A simple percona-cluster exercise on Artful, can readily repro with this on serverstackcluster:shared-dbidentity-serviceidentity-serviceserver:amqpcharmers-next/percona-clusterconnections: 1000buffer-pool-size: 256Mcharmers-next/keystonepassword: openstackmultiplier: 0.25charmers-next/glancemultiplier: 0.25charmers-next/rabbitmq-server
series: artful
relations:
- - keystone:shared-db
- percona-
- - glance:
- keystone:
- - glance:amqp
- rabbitmq-
services:
percona-cluster:
charm: cs:~openstack-
num_units: 1
constraints: "cpu-cores=4 mem=4G"
options:
max-
innodb-
keystone:
charm: cs:~openstack-
num_units: 1
constraints: "cpu-cores=2 mem=2G"
options:
admin-
worker-
glance:
charm: cs:~openstack-
num_units: 1
constraints: "cpu-cores=2 mem=2G"
options:
worker-
rabbitmq-server:
charm: cs:~openstack-
num_units: 1
constraints: "cpu-cores=4 mem=4G"
DEBUG:bundletester.utils:Updating JUJU_MODEL: "auto-osci-sv06:admin/auto-osci-sv06" -> ""ter.fetchers:git rev-parse HEAD: d0d41964227bc1dc46887fa77ba4be3bc5065fae
DEBUG:bundletes
ERROR: InvocationError: '/var/lib/jenkins/checkout/0/ceph/.tox/func27/bin/bundletester -vl DEBUG -r json -o func-results.json --test-pattern gate-* --no-destroy'____________________________ summary ____________________________________serverstack 2.2.4 unsupported
_______
ERROR: func27: commands failed
! Amulet test failed.
Model Controller Cloud/Region Version SLA
auto-osci-sv06 auto-osci-sv06 serverstack/
App Version Status Scale Charm Store Rev OS Notes
ceph 12.2.0 active 3 ceph local 105 ubuntu
ceph-osd 12.2.0 active 1 ceph-osd jujucharms 297 ubuntu
cinder 11.0.0 waiting 1 cinder jujucharms 298 ubuntu
cinder-ceph waiting 0/1 cinder-ceph jujucharms 247 ubuntu
glance 15.0.0 waiting 1 glance jujucharms 292 ubuntu
keystone 12.0.0 waiting 1 keystone jujucharms 322 ubuntu
nova-compute 16.0.1 active 1 nova-compute jujucharms 336 ubuntu
percona-cluster 5.6.34-26.19 error 1 percona-cluster jujucharms 274 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 280 ubuntu
Unit Workload Agent Machine Public address Ports Messagedb-relation-changed" for glance:shared-db
ceph-osd/0* active idle 3 172.17.106.17 Unit is ready (2 OSD)
ceph/0* active idle 0 172.17.106.10 Unit is ready and clustered
ceph/1 active idle 1 172.17.106.5 Unit is ready and clustered
ceph/2 active idle 2 172.17.106.3 Unit is ready and clustered
cinder/0* waiting executing 4 172.17.106.20 8776/tcp Incomplete relations: messaging, identity, database
cinder-ceph/0* waiting allocating 172.17.106.20 agent initializing
glance/0* waiting executing 5 172.17.106.22 9292/tcp Incomplete relations: identity, database
keystone/0* waiting executing 6 172.17.106.21 5000/tcp Incomplete relations: database
nova-compute/0* active executing 7 172.17.106.4 Unit is ready
percona-cluster/0* error idle 8 172.17.106.18 3306/tcp hook failed: "shared-
rabbitmq-server/0* active idle 9 172.17.106.26 5672/tcp Unit is ready
Machine State DNS Inst id Series AZ Messagef4ff-450d-b088-a3742a5ffffd artful nova ACTIVE15fe-4a5e-93a2-a1f6cfd9523b artful nova ACTIVE2fdb-4b81-b4ab-0467c5e5705d artful nova ACTIVE73a6-4f50-a97e-ae391ed96e63 artful nova ACTIVE9dd3-4f1a-a668-0b5eb7819b9a artful nova ACTIVE396b-4c27-b063-34ff63ddb3f2 artful nova ACTIVE88d5-4f1b-a97d-b22176cc231f artful nova ACTIVE5c30-451a-8963-68057bac27f8 artful nova ACTIVE5898-44b4-9f0a-3bffbe808d05 artful nova ACTIVE332e-48d5-8b5d-c39e3deadb4a artful nova ACTIVE
0 started 172.17.106.10 cc716b8d-
1 started 172.17.106.5 e0498107-
2 started 172.17.106.3 6a02e6cc-
3 started 172.17.106.17 cc5a2f76-
4 started 172.17.106.20 4d42c847-
5 started 172.17.106.22 8dd50778-
6 started 172.17.106.21 58eac09b-
7 started 172.17.106.4 534d23a0-
8 started 172.17.106.18 091cb17d-
9 started 172.17.106.26 eb03b662-
Relation provider Requirer Interface Typeceph:storage-backend cinder:storage-backend cinder-backend subordinateimage-service cinder:image-service glance regularimage-service nova-compute:image-service glance regularidentity-service cinder:identity-service keystone regularidentity-service glance:identity-service keystone regularcompute-peer nova-compute:compute-peer nova peercluster:cluster percona-cluster:cluster percona-cluster peercluster:shared-db cinder:shared-db mysql-shared regularcluster:shared-db glance:shared-db mysql-shared regularcluster:shared-db keystone:shared-db mysql-shared regularcluster:shared-db nova-compute:shared-db mysql-shared regularserver:amqp cinder:amqp rabbitmq regularserver:amqp glance:amqp rabbitmq regularserver:amqp nova-compute:amqp rabbitmq regularserver:cluster rabbitmq-server:cluster rabbitmq-ha peer
ceph:client cinder-ceph:ceph ceph-client regular
ceph:client glance:ceph ceph-client regular
ceph:client nova-compute:ceph ceph-client regular
ceph:mon ceph:mon ceph peer
ceph:osd ceph-osd:mon ceph-osd regular
cinder-
cinder:cluster cinder:cluster cinder-ha peer
glance:cluster glance:cluster glance-ha peer
glance:
glance:
keystone:cluster keystone:cluster keystone-ha peer
keystone:
keystone:
nova-compute:
percona-
percona-
percona-
percona-
percona-
rabbitmq-
rabbitmq-
rabbitmq-
rabbitmq-