tripleo

CS9 - Standalone: mysql_wait_bundle fails with "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'"

Bug #1964999 reported by Douglas Viroel
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
New
High
Unassigned
tripleo yoga-3

Bug Description

Standalone jobs have this intermittent failure, where mysql_wait_bundle containet fails to start. The followin errors are shown[1][2]:

Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'

NOTE: this happens more often in FIPS enabled jobs, considering this change [3], but the issue was seen in non-FIPS jobs too.

[1] https://2496c8b3721efd3eb4ab-5b369dc32177e920b38f658289f2b193.ssl.cf5.rackcdn.com/824479/18/check/tripleo-ci-centos-9-standalone/69a6eda/logs/undercloud/home/zuul/standalone_deploy.log
[2] https://1deec6edd105464d9773-b3be29898e9936ff543653f1540e5e4c.ssl.cf1.rackcdn.com/824479/18/check/tripleo-ci-centos-9-scenario012-standalone/98ede58/logs/undercloud/home/zuul/standalone_deploy.log
[3] https://review.opendev.org/c/openstack/tripleo-ci/+/824479/

Tags: ci fips
Douglas Viroel (dviroel)
tags: added: fips
Revision history for this message
Luca Miccini (lmiccini2) wrote :

the issue is:

Mar 11 20:30:15.866 standalone.localdomain pacemaker-controld [25187] (log_executor_event) error: Result of start operation for galera-bundle-podman-0 on standalone: Timed Out after 20s (Resource agent did not complete in time) | CIB update 88, graph action confirmed; call=28 key=galera-bundle-podman-0_start_0

Bundle: galera-bundle
  Podman: image=149.202.181.113:5001/tripleomastercentos9/openstack-mariadb:pcmklatest network=host options="--user=root --log-driver=k8s-file --log-opt path=/var/log/containers/stdouts/galera-bundle.log -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" promoted-max=1 replicas=1 run-command="/bin/bash /usr/local/bin/kolla_start"
  Network: control-port=3123
  Storage Mapping:
   options=ro source-dir=/var/lib/kolla/config_files/mysql.json target-dir=/var/lib/kolla/config_files/config.json (mysql-cfg-files)
   options=ro source-dir=/var/lib/config-data/puppet-generated/mysql/ target-dir=/var/lib/kolla/config_files/src (mysql-cfg-data)
   options=ro source-dir=/etc/hosts target-dir=/etc/hosts (mysql-hosts)
   options=ro source-dir=/etc/localtime target-dir=/etc/localtime (mysql-localtime)
   options=rw,z source-dir=/var/lib/mysql target-dir=/var/lib/mysql (mysql-lib)
   options=rw,z source-dir=/var/log/mariadb target-dir=/var/log/mariadb (mysql-log-mariadb)
   options=rw,z source-dir=/var/log/containers/mysql target-dir=/var/log/mysql (mysql-log)
   options=rw source-dir=/dev/log target-dir=/dev/log (mysql-dev-log)
  Resource: galera (class=ocf provider=heartbeat type=galera)
   Attributes: additional_parameters=--open-files-limit=16384 cluster_host_map=standalone:standalone.ctlplane.localdomain enable_creation=true log=/var/log/mysql/mysqld.log wsrep_cluster_address=gcomm://standalone.ctlplane.localdomain
   Meta Attrs: container-attribute-target=host master-max=1 ordered=true
   Operations: demote interval=0s timeout=120s (galera-demote-interval-0s)
               monitor interval=20s timeout=30s (galera-monitor-interval-20s)
               monitor interval=10s role=Promoted timeout=30s (galera-monitor-interval-10s)
               monitor interval=30s role=Unpromoted timeout=30s (galera-monitor-interval-30s)
               promote interval=0s on-fail=block timeout=300s (galera-promote-interval-0s)
               start interval=0s timeout=120s (galera-start-interval-0s)
               stop interval=0s timeout=120s (galera-stop-interval-0s)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.