On further check to get what's common in all these found that all these are empty directories:-
$ missing=$(sudo chroot /mnt rpm -Va |grep missing|sed -e "s/ c / /"|awk '{print $2}')
# trimmed some .repo and .conf files from above output
Some observations:-
- Something is removing empty directories during overcloud image build.
- Removed directories are not related to a specific package, multiple packages affected whenver image has issues.
- Not all empty directories are removed, example in this image /var/log/pcsd empty directory was not removed.
With debug enabled following Error is reported:- ******* ******* ******* ******* ******* ****** controller- 0]: FAILED! => changed=false 562930669e09f80 53fcd47750 tls/private does not exist controller- 1]: FAILED! => changed=false 562930669e09f80 53fcd47750 tls/private does not exist controller- 2]: FAILED! => changed=false 562930669e09f80 53fcd47750 tls/private does not exist
2020-04-23 08:24:59 | TASK [push certificate content] *******
2020-04-23 08:24:59 | Thursday 23 April 2020 08:24:59 +0000 (0:00:00.116) 0:02:20.602 ********
2020-04-23 08:25:00 | fatal: [overcloud-
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/
2020-04-23 08:25:00 | fatal: [overcloud-
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/
2020-04-23 08:25:00 | fatal: [overcloud-
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/
logs:- https:/ /logserver. rdoproject. org/12/ 26712/1/ check/periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001-master/ bbd78ab/ logs/undercloud /home/zuul/ overcloud_ deploy. log.txt. gz
/etc/pki/ tls/private directory is part of openssl-libs package and should exist by default. The issue(deletion of directories) is same as https:/ /bugs.launchpad .net/tripleo/ +bug/1867602.
Collected following info from latest overcloud image which has issue:- /images. rdoproject. org/centos8/ master/ rdo_trunk/ 724c195e098a6a8 5afd12fffc11ffb c0/overcloud- full.tar BACKEND= direct guestmount -i -a overcloud- full.qcow2 /mnt
$ curl -O https:/
$ tar -xvf overcloud-full.tar
$ sudo LIBGUESTFS_
$ sudo chroot /mnt rpm -Va openssl-libs tls/private
missing /etc/pki/tls/misc
missing /etc/pki/
^^ these directories are removed somehow.
Also the issue is not specific to openssl-libs, it's affecting other directories as well:-
$ sudo chroot /mnt rpm -Va |grep missing iscsi/ifaces iscsi/nodes iscsi/send_ targets iscsi/static ga/fsfreeze- hook.d libxslt- plugins doc/python3- pycurl/ tests/tmp i18n/charmaps i18n/locales ager/conf. d ager/dispatcher .d/no-wait. d ager/dispatcher .d/pre- down.d ager/dispatcher .d/pre- up.d ager/dnsmasq- shared. d ager/dnsmasq. d ager/system- connections NetworkManager NetworkManager/ VPN NetworkManager/ conf.d NetworkManager/ system- connections NetworkManager systemd/ system- sleep sudo/lectured net-snmp/ cert_indexes net-snmp/ mib_indexes openldap repos.d/ CentOS- AppStream. repo repos.d/ CentOS- Base.repo repos.d/ CentOS- CR.repo repos.d/ CentOS- Debuginfo. repo repos.d/ CentOS- Devel.repo repos.d/ CentOS- Extras. repo repos.d/ CentOS- HA.repo repos.d/ CentOS- Media.repo repos.d/ CentOS- PowerTools. repo repos.d/ CentOS- Sources. repo repos.d/ CentOS- Vault.repo repos.d/ CentOS- centosplus. repo repos.d/ CentOS- fasttrack. repo doc/policycoreu tils recommend. d doc/python3- cryptography/ docs/_static ruby/racc/ rdoc tls/private systemd/ system- shutdown systemd/ system/ basic.target. wants systemd/ system/ dbus.target. wants systemd/ system/ default. target. wants systemd/ system/ remote- fs.target. wants systemd/ system/ runlevel1. target. wants systemd/ system/ runlevel2. target. wants systemd/ system/ runlevel3. target. wants systemd/ system/ runlevel4. target. wants systemd/ system/ runlevel5. target. wants systemd/ system/ syslog. target. wants systemd/ user-generators rubygems/ rubygems/ ssl_certs
missing /var/lib/
missing /var/lib/iscsi/isns
missing /var/lib/
missing /var/lib/
missing /var/lib/iscsi/slp
missing /var/lib/
missing /etc/qemu-
missing /var/log/qemu-ga
missing /usr/lib64/
missing /usr/share/kdump
missing /var/crash
missing /usr/share/
missing /usr/share/i18n
missing /usr/share/
missing /usr/share/
missing /etc/NetworkMan
missing /etc/NetworkMan
missing /etc/NetworkMan
missing /etc/NetworkMan
missing /etc/NetworkMan
missing /etc/NetworkMan
missing /etc/NetworkMan
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /var/lib/
missing /etc/modules-load.d
missing /usr/lib/
missing /var/empty/sshd
missing /etc/sudoers.d
missing /var/db/sudo
missing /var/db/
missing c /boot/grub2/grubenv
missing /var/lib/
missing /var/lib/
missing /etc/openldap/certs
missing /usr/libexec/
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing c /etc/yum.
missing /usr/share/
missing /etc/tuned/
missing /run/tuned
missing /var/lib/tuned
missing /var/log/tuned
missing /usr/share/
missing /usr/share/
missing /etc/pki/tls/misc
missing /etc/pki/
missing /etc/binfmt.d
missing /etc/systemd/user
missing /usr/lib/binfmt.d
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /usr/lib/
missing /run/cloud-init
missing /var/lib/cloud
missing /var/log/audit
missing /usr/lib/gems/ruby
missing /usr/share/
On further check to get what's common in all these found that all these are empty directories:-
$ missing=$(sudo chroot /mnt rpm -Va |grep missing|sed -e "s/ c / /"|awk '{print $2}')
$ for file in $missing; do rpm=$(sudo chroot /mnt rpm -qf --queryformat= %{NAME} $file); sudo chroot /mnt rpm -ql $rpm |grep $file ;done iscsi/ifaces iscsi/nodes iscsi/send_ targets iscsi/static ga/fsfreeze- hook.d libxslt- plugins doc/python3- pycurl/ tests/tmp i18n/charmaps i18n/locales i18n/charmaps i18n/locales ager/conf. d ager/dispatcher .d/no-wait. d ager/dispatcher .d/pre- down.d ager/dispatcher .d/pre- up.d ager/dnsmasq- shared. d ager/dnsmasq. d ager/system- connections NetworkManager NetworkManager/ VPN NetworkManager/ conf.d NetworkManager/ system- connections NetworkManager/ VPN NetworkManager/ conf.d NetworkManager/ system- connections NetworkManager systemd/ system- sleep sudo/lectured sudo/lectured net-snmp/ cert_indexes net-snmp/ mib_indexes openldap doc/policycoreu tils recommend. d doc/python3- cryptography/ docs/_static ruby/racc/ rdoc tls/private systemd/ system- shutdown systemd/ system/ basic.target. wants systemd/ system/ dbus.target. wants systemd/ system/ default. target. wants systemd/ system/ remote- fs.target. wants systemd/ system/ runlevel1. target. wants systemd/ system/ runlevel2. target. wants systemd/ system/ runlevel3. target. wants systemd/ system/ runlevel4. target. wants systemd/ system/ runlevel5. target. wants systemd/ system/ syslog. target. wants systemd/ user-generators rubygems/ rubygems/ ssl_certs
/var/lib/
/var/lib/iscsi/isns
/var/lib/
/var/lib/
/var/lib/iscsi/slp
/var/lib/
/etc/qemu-
/var/log/qemu-ga
/usr/lib64/
/usr/share/kdump
/var/crash
/usr/share/
/usr/share/i18n
/usr/share/
/usr/share/
/usr/share/
/usr/share/
/etc/NetworkMan
/etc/NetworkMan
/etc/NetworkMan
/etc/NetworkMan
/etc/NetworkMan
/etc/NetworkMan
/etc/NetworkMan
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/var/lib/
/etc/modules-load.d
/usr/lib/
/var/empty/sshd
/etc/sudoers.d
/var/db/sudo
/var/db/
/var/db/
/boot/grub2/grubenv
/var/lib/
/var/lib/
/etc/openldap/certs
/usr/libexec/
/usr/share/
/etc/tuned/
/run/tuned
/var/lib/tuned
/var/log/tuned
/usr/share/
/usr/share/
/etc/pki/tls/misc
/etc/pki/
/etc/binfmt.d
/etc/systemd/user
/usr/lib/binfmt.d
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/usr/lib/
/run/cloud-init
/var/lib/cloud
/var/log/audit
/usr/lib/gems/ruby
/usr/share/
# trimmed some .repo and .conf files from above output
Some observations:-
- Something is removing empty directories during overcloud image build.
- Removed directories are not related to a specific package, multiple packages affected whenver image has issues.
- Not all empty directories are removed, example in this image /var/log/pcsd empty directory was not removed.
For reference, good image which do not have any missing directory:- https:/ /images. rdoproject. org/centos8/ master/ rdo_trunk/ b3720367a6a0349 abcfb06939bed31 01/overcloud- full.tar