When podman dropped the journald log-driver we rushed to move to the supported
k8s-file driver. This had the side effect of us losing the stdout logs of the
HA containers.
In fact previously we were easily able to troubleshoot haproxy startup failures
just by looking in the journal. These days instead if haproxy fails to start we
have no traces whatsoever in the logs, because when a container fails it gets
stopped by pacemaker (and consequently removed) and no logs on the system are
available any longer.
Tested as follows:
1) Redeploy a previously deployed overcloud that did not have the patch
and observe that we now log the startup of HA bundles in /var/log/containers/stdouts/*bundle.log
2) Deploy a composable HA overcloud from scratch with the patch above
and observe that we obtain the stdout on disk.
Note that most HA containers log to their usual on-host files just
fine, we are mainly missing haproxy logs and/or the kolla startup only
of the HA containers.
Reviewed: https:/ /review. opendev. org/719773 /git.openstack. org/cgit/ openstack/ puppet- tripleo/ commit/ ?id=06c4aa74460 73022b86c1f034a 0c5406f2675ddb
Committed: https:/
Submitter: Zuul
Branch: master
commit 06c4aa744607302 2b86c1f034a0c54 06f2675ddb
Author: Michele Baldessari <email address hidden>
Date: Tue Apr 14 11:14:22 2020 +0200
Log stdout of HA containers
When podman dropped the journald log-driver we rushed to move to the supported
k8s-file driver. This had the side effect of us losing the stdout logs of the
HA containers.
In fact previously we were easily able to troubleshoot haproxy startup failures
just by looking in the journal. These days instead if haproxy fails to start we
have no traces whatsoever in the logs, because when a container fails it gets
stopped by pacemaker (and consequently removed) and no logs on the system are
available any longer.
Tested as follows: containers/ stdouts/ *bundle. log
1) Redeploy a previously deployed overcloud that did not have the patch
and observe that we now log the startup of HA bundles in /var/log/
[root@ controller- 0 stdouts]# ls -l *bundle.log |grep -v -e init -e restart cinder- volume. log
-rw-------. 1 root root 16032 Apr 14 14:13 openstack-
-rw-------. 1 root root 19515 Apr 14 14:00 haproxy-bundle.log
-rw-------. 1 root root 10509 Apr 14 14:03 ovn-dbs-bundle.log
-rw-------. 1 root root 6451 Apr 14 14:00 redis-bundle.log
2) Deploy a composable HA overcloud from scratch with the patch above
and observe that we obtain the stdout on disk.
Note that most HA containers log to their usual on-host files just
fine, we are mainly missing haproxy logs and/or the kolla startup only
of the HA containers.
Closes-Bug: #1872734
Change-Id: I4270b398366e90 206adffe32f8126 32b50df615b