provide tuning configuration options (was Huawei RH2288H V3 mysql 3 HA units, ~100% ioutil on idle (ext4 journaling + mysqld))
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Percona Cluster Charm |
Fix Released
|
High
|
James Page | ||
percona-cluster (Juju Charms Collection) |
Invalid
|
High
|
Unassigned |
Bug Description
Manufacturer: Huawei RH2288H V3
Ubuntu 14.04.4 LTS
Ubuntu 4.2.0-38.
Juju 1.25.5, OS Charms 16.04; Liberty
percona-cluster deployed on 3 LXC units
lp:charms/trusty/percona-cluster;revno=99
hacluster charm used for VIP management.
lp:charms/trusty/hacluster;revno=56
OpenStack deployed on 33 compute-storage nodes.
"/var/lib/lxc" has been tested on:
1) sda1, together with rootfs (ext4)
2) sda2, separated from rootfs (ext4)
3) sdb1, on a different, non-RAID disk
3.1) sdb1 on ext4
3.2) sdb1 on xfs
On all cases, both iostat and iotop showed high %ioutil on an idle OpenStack environment. If 2 out of 3 mysql units get their mysql service stopped, only the remaining metal running MySQL shows high ioutil.
We have another environment running 13 compute-storage nodes, with same specifications, having a similar issue (high io on drives where mysql is being run).
I'll add a few mysql logs. Let me know if you would need further details.
Changed in percona-cluster (Juju Charms Collection): | |
importance: | Wishlist → High |
tags: | added: performance |
Changed in charm-percona-cluster: | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in percona-cluster (Juju Charms Collection): | |
status: | Triaged → Invalid |
Changed in charm-percona-cluster: | |
milestone: | none → 17.05 |
Changed in charm-percona-cluster: | |
status: | Triaged → In Progress |
assignee: | nobody → James Page (james-page) |
Changed in charm-percona-cluster: | |
milestone: | 17.05 → 17.08 |
Changed in charm-percona-cluster: | |
status: | Fix Committed → Fix Released |
We've been able to workaround this issue by
tuning mysql's some innodb parameters, notably:
# main improvement: from ~90% util to ~20%: flush_log_ at_trx_ commit= 0
innodb_
#UPDATES also (openstack does quite some): change_ buffering= all
innodb_
# ~90 iops -> 100%util as observed io_capacity= 100
innodb_
locally branched charm "cowboy" patch: paste.ubuntu. com/18724829/
http://
IMO this charm should expose a "config-flags" setting
(e.g. as openstack charms do).