Here is the solution:
E.g. lets check puppet logs in docker for node-6 residing in the new env (node-1, node-2, etc were belonging to the deleted envs)
Look for puppet logs FD mappings in rsyslog container dockerctl shell rsyslog lsof | grep puppet rsyslogd 391 root 15w REG 253,2 1149077 138872644 /var/log/remote/node-3.test.domain.local/puppet-apply.log rsyslogd 391 root 24w REG 253,2 370285 11935 /var/log/remote/node-1.test.domain.local/puppet-apply.log rsyslogd 391 root 26w REG 253,2 874922 138887274 /var/log/remote/node-2.test.domain.local/puppet-apply.log Wrong mappings for old (reset/recreate) envs!
Fix is: should be done at master node running the docker container for rsyslog on each reset/recreate env and on logrotate postscript as well for pids in $(pidof rsyslogd); do echo kill -HUP $pids; done
Recheck now: dockerctl shell rsyslog lsof | grep puppet rsyslogd 391 root 22w REG 253,2 1281 75649827 /var/log/remote/node-6.test.domain.local/puppet-apply.log Looks ok now, logs present.
Here is the solution:
E.g. lets check puppet logs in docker for node-6 residing in the new env (node-1, node-2, etc were belonging to the deleted envs)
Look for puppet logs FD mappings in rsyslog container remote/ node-3. test.domain. local/puppet- apply.log remote/ node-1. test.domain. local/puppet- apply.log remote/ node-2. test.domain. local/puppet- apply.log
dockerctl shell rsyslog lsof | grep puppet
rsyslogd 391 root 15w REG 253,2 1149077 138872644 /var/log/
rsyslogd 391 root 24w REG 253,2 370285 11935 /var/log/
rsyslogd 391 root 26w REG 253,2 874922 138887274 /var/log/
Wrong mappings for old (reset/recreate) envs!
Fix is:
should be done at master node running the docker container for rsyslog on each reset/recreate env and on logrotate postscript as well
for pids in $(pidof rsyslogd); do echo kill -HUP $pids; done
Recheck now: remote/ node-6. test.domain. local/puppet- apply.log
dockerctl shell rsyslog lsof | grep puppet
rsyslogd 391 root 22w REG 253,2 1281 75649827 /var/log/
Looks ok now, logs present.