Bug #1873770 “OVB fs001 in centos8 master fails to push certific...” : Bugs : tripleo

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-20: Related fix proposed to tripleo-heat-templates (master)

#1

Related fix proposed to branch: master
Review: https://review.opendev.org/721247

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-20:

#2

Related fix proposed to branch: master
Review: https://review.opendev.org/721255

Revision history for this message

Gabriele Cerami (gcerami) wrote on 2020-04-20:

#3

Investigating with Cedric to offer complete logs in CI to debug this.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-22: Related fix merged to tripleo-heat-templates (master)

#4

Reviewed: https://review.opendev.org/721247
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cabed543fa74014d17a6ac603bc2239237864b21
Submitter: Zuul
Branch: master

commit cabed543fa74014d17a6ac603bc2239237864b21
Author: Cédric Jeanneret <email address hidden>
Date: Mon Apr 20 14:02:21 2020 +0200

Introduce new HideSensitiveLogs parameter

This one toggles the no_log parameter. Directly related to #1873770 in
order to allow a deeper debug within CI.

Change-Id: I27f677467263c0e6cc78d775edff55b3811fec1f
Related-Bug: #1873770

Revision history for this message

yatin (yatinkarel) wrote on 2020-04-23:

#5

Download full text (8.0 KiB)

With debug enabled following Error is reported:-
2020-04-23 08:24:59 | TASK [push certificate content] ************************************************
2020-04-23 08:24:59 | Thursday 23 April 2020 08:24:59 +0000 (0:00:00.116) 0:02:20.602 ********
2020-04-23 08:25:00 | fatal: [overcloud-controller-0]: FAILED! => changed=false
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/tls/private does not exist
2020-04-23 08:25:00 | fatal: [overcloud-controller-1]: FAILED! => changed=false
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/tls/private does not exist
2020-04-23 08:25:00 | fatal: [overcloud-controller-2]: FAILED! => changed=false
2020-04-23 08:25:00 | checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 | msg: Destination directory /etc/pki/tls/private does not exist

logs:- https://logserver.rdoproject.org/12/26712/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/bbd78ab/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

/etc/pki/tls/private directory is part of openssl-libs package and should exist by default. The issue(deletion of directories) is same as https://bugs.launchpad.net/tripleo/+bug/1867602.

Collected following info from latest overcloud image which has issue:-
$ curl -O https://images.rdoproject.org/centos8/master/rdo_trunk/724c195e098a6a85afd12fffc11ffbc0/overcloud-full.tar
$ tar -xvf overcloud-full.tar
$ sudo LIBGUESTFS_BACKEND=direct guestmount -i -a overcloud-full.qcow2 /mnt

$ sudo chroot /mnt rpm -Va openssl-libs
missing /etc/pki/tls/misc
missing /etc/pki/tls/private

^^ these directories are removed somehow.

Also the issue is not specific to openssl-libs, it's affecting other directories as well:-

$ sudo chroot /mnt rpm -Va |grep missing
missing /var/lib/iscsi/ifaces
missing /var/lib/iscsi/isns
missing /var/lib/iscsi/nodes
missing /var/lib/iscsi/send_targets
missing /var/lib/iscsi/slp
missing /var/lib/iscsi/static
missing /etc/qemu-ga/fsfreeze-hook.d
missing /var/log/qemu-ga
missing /usr/lib64/libxslt-plugins
missing /usr/share/kdump
missing /var/crash
missing /usr/share/doc/python3-pycurl/tests/tmp
missing /usr/share/i18n
missing /usr/share/i18n/charmaps
missing /usr/share/i18n/locales
missing /etc/NetworkManager/conf.d
missing /etc/NetworkManager/dispatcher.d/no-wait.d
missing /etc/NetworkManager/dispatcher.d/pre-down.d
missing /etc/NetworkManager/dispatcher.d/pre-up.d
missing /etc/NetworkManager/dnsmasq-shared.d
missing /etc/NetworkManager/dnsmasq.d
missing /etc/NetworkManager/system-connections
missing /usr/lib/NetworkManager
missing /usr/lib/NetworkManager/VPN
missing /usr/lib/NetworkManager/conf.d
missing /usr/lib/NetworkManager/system-connections
missing /var/lib/NetworkManager
missing /etc/modules-load.d
missing /usr/lib/systemd/system-sleep
missing /var/empty/sshd
missing /etc/sudoers.d
missing /var/db/sudo
missing /var/db/sudo/lectured
...

With debug enabled following Error is reported:-
2020-04-23 08:24:59 | TASK [push certificate content] ************************************************
2020-04-23 08:24:59 | Thursday 23 April 2020  08:24:59 +0000 (0:00:00.116)       0:02:20.602 ******** 
2020-04-23 08:25:00 | fatal: [overcloud-controller-0]: FAILED! => changed=false 
2020-04-23 08:25:00 |   checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 |   msg: Destination directory /etc/pki/tls/private does not exist
2020-04-23 08:25:00 | fatal: [overcloud-controller-1]: FAILED! => changed=false 
2020-04-23 08:25:00 |   checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 |   msg: Destination directory /etc/pki/tls/private does not exist
2020-04-23 08:25:00 | fatal: [overcloud-controller-2]: FAILED! => changed=false 
2020-04-23 08:25:00 |   checksum: 63c1ca987a881f8562930669e09f8053fcd47750
2020-04-23 08:25:00 |   msg: Destination directory /etc/pki/tls/private does not exist

logs:- https://logserver.rdoproject.org/12/26712/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/bbd78ab/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

/etc/pki/tls/private directory is part of openssl-libs package and should exist by default. The issue(deletion of directories) is same as https://bugs.launchpad.net/tripleo/+bug/1867602.

Collected following info from latest overcloud image which has issue:-
$ curl -O https://images.rdoproject.org/centos8/master/rdo_trunk/724c195e098a6a85afd12fffc11ffbc0/overcloud-full.tar
$ tar -xvf overcloud-full.tar
$ sudo LIBGUESTFS_BACKEND=direct guestmount -i -a  overcloud-full.qcow2 /mnt

$ sudo chroot /mnt rpm -Va openssl-libs
missing     /etc/pki/tls/misc
missing     /etc/pki/tls/private

^^ these directories are removed somehow.

Also the issue is not specific to openssl-libs, it's affecting other directories as well:-

$ sudo chroot /mnt rpm -Va |grep missing
missing     /var/lib/iscsi/ifaces
missing     /var/lib/iscsi/isns
missing     /var/lib/iscsi/nodes
missing     /var/lib/iscsi/send_targets
missing     /var/lib/iscsi/slp
missing     /var/lib/iscsi/static
missing     /etc/qemu-ga/fsfreeze-hook.d
missing     /var/log/qemu-ga
missing     /usr/lib64/libxslt-plugins
missing     /usr/share/kdump
missing     /var/crash
missing     /usr/share/doc/python3-pycurl/tests/tmp
missing     /usr/share/i18n
missing     /usr/share/i18n/charmaps
missing     /usr/share/i18n/locales
missing     /etc/NetworkManager/conf.d
missing     /etc/NetworkManager/dispatcher.d/no-wait.d
missing     /etc/NetworkManager/dispatcher.d/pre-down.d
missing     /etc/NetworkManager/dispatcher.d/pre-up.d
missing     /etc/NetworkManager/dnsmasq-shared.d
missing     /etc/NetworkManager/dnsmasq.d
missing     /etc/NetworkManager/system-connections
missing     /usr/lib/NetworkManager
missing     /usr/lib/NetworkManager/VPN
missing     /usr/lib/NetworkManager/conf.d
missing     /usr/lib/NetworkManager/system-connections
missing     /var/lib/NetworkManager
missing     /etc/modules-load.d
missing     /usr/lib/systemd/system-sleep
missing     /var/empty/sshd
missing     /etc/sudoers.d
missing     /var/db/sudo
missing     /var/db/sudo/lectured
missing   c /boot/grub2/grubenv
missing     /var/lib/net-snmp/cert_indexes
missing     /var/lib/net-snmp/mib_indexes
missing     /etc/openldap/certs
missing     /usr/libexec/openldap
missing   c /etc/yum.repos.d/CentOS-AppStream.repo
missing   c /etc/yum.repos.d/CentOS-Base.repo
missing   c /etc/yum.repos.d/CentOS-CR.repo
missing   c /etc/yum.repos.d/CentOS-Debuginfo.repo
missing   c /etc/yum.repos.d/CentOS-Devel.repo
missing   c /etc/yum.repos.d/CentOS-Extras.repo
missing   c /etc/yum.repos.d/CentOS-HA.repo
missing   c /etc/yum.repos.d/CentOS-Media.repo
missing   c /etc/yum.repos.d/CentOS-PowerTools.repo
missing   c /etc/yum.repos.d/CentOS-Sources.repo
missing   c /etc/yum.repos.d/CentOS-Vault.repo
missing   c /etc/yum.repos.d/CentOS-centosplus.repo
missing   c /etc/yum.repos.d/CentOS-fasttrack.repo
missing     /usr/share/doc/policycoreutils
missing     /etc/tuned/recommend.d
missing     /run/tuned
missing     /var/lib/tuned
missing     /var/log/tuned
missing     /usr/share/doc/python3-cryptography/docs/_static
missing     /usr/share/ruby/racc/rdoc
missing     /etc/pki/tls/misc
missing     /etc/pki/tls/private
missing     /etc/binfmt.d
missing     /etc/systemd/user
missing     /usr/lib/binfmt.d
missing     /usr/lib/systemd/system-shutdown
missing     /usr/lib/systemd/system/basic.target.wants
missing     /usr/lib/systemd/system/dbus.target.wants
missing     /usr/lib/systemd/system/default.target.wants
missing     /usr/lib/systemd/system/remote-fs.target.wants
missing     /usr/lib/systemd/system/runlevel1.target.wants
missing     /usr/lib/systemd/system/runlevel2.target.wants
missing     /usr/lib/systemd/system/runlevel3.target.wants
missing     /usr/lib/systemd/system/runlevel4.target.wants
missing     /usr/lib/systemd/system/runlevel5.target.wants
missing     /usr/lib/systemd/system/syslog.target.wants
missing     /usr/lib/systemd/user-generators
missing     /run/cloud-init
missing     /var/lib/cloud
missing     /var/log/audit
missing     /usr/lib/gems/ruby
missing     /usr/share/rubygems/rubygems/ssl_certs

On further check to get what's common in all these found that all these are empty directories:-
$ missing=$(sudo chroot /mnt rpm -Va |grep missing|sed -e "s/ c /   /"|awk '{print $2}')

$ for file in $missing; do rpm=$(sudo chroot /mnt rpm -qf --queryformat=%{NAME} $file); sudo chroot /mnt rpm -ql $rpm |grep $file ;done
/var/lib/iscsi/ifaces
/var/lib/iscsi/isns
/var/lib/iscsi/nodes
/var/lib/iscsi/send_targets
/var/lib/iscsi/slp
/var/lib/iscsi/static
/etc/qemu-ga/fsfreeze-hook.d
/var/log/qemu-ga
/usr/lib64/libxslt-plugins
/usr/share/kdump
/var/crash
/usr/share/doc/python3-pycurl/tests/tmp
/usr/share/i18n
/usr/share/i18n/charmaps
/usr/share/i18n/locales
/usr/share/i18n/charmaps
/usr/share/i18n/locales
/etc/NetworkManager/conf.d
/etc/NetworkManager/dispatcher.d/no-wait.d
/etc/NetworkManager/dispatcher.d/pre-down.d
/etc/NetworkManager/dispatcher.d/pre-up.d
/etc/NetworkManager/dnsmasq-shared.d
/etc/NetworkManager/dnsmasq.d
/etc/NetworkManager/system-connections
/usr/lib/NetworkManager
/usr/lib/NetworkManager/VPN
/usr/lib/NetworkManager/conf.d
/usr/lib/NetworkManager/system-connections
/usr/lib/NetworkManager/VPN
/usr/lib/NetworkManager/conf.d
/usr/lib/NetworkManager/system-connections
/var/lib/NetworkManager
/etc/modules-load.d
/usr/lib/systemd/system-sleep
/var/empty/sshd
/etc/sudoers.d
/var/db/sudo
/var/db/sudo/lectured
/var/db/sudo/lectured
/boot/grub2/grubenv
/var/lib/net-snmp/cert_indexes
/var/lib/net-snmp/mib_indexes
/etc/openldap/certs
/usr/libexec/openldap
/usr/share/doc/policycoreutils
/etc/tuned/recommend.d
/run/tuned
/var/lib/tuned
/var/log/tuned
/usr/share/doc/python3-cryptography/docs/_static
/usr/share/ruby/racc/rdoc
/etc/pki/tls/misc
/etc/pki/tls/private
/etc/binfmt.d
/etc/systemd/user
/usr/lib/binfmt.d
/usr/lib/systemd/system-shutdown
/usr/lib/systemd/system/basic.target.wants
/usr/lib/systemd/system/dbus.target.wants
/usr/lib/systemd/system/default.target.wants
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/runlevel1.target.wants
/usr/lib/systemd/system/runlevel2.target.wants
/usr/lib/systemd/system/runlevel3.target.wants
/usr/lib/systemd/system/runlevel4.target.wants
/usr/lib/systemd/system/runlevel5.target.wants
/usr/lib/systemd/system/syslog.target.wants
/usr/lib/systemd/user-generators
/run/cloud-init
/var/lib/cloud
/var/log/audit
/usr/lib/gems/ruby
/usr/share/rubygems/rubygems/ssl_certs

# trimmed some .repo and .conf files from above output

Some observations:-
- Something is removing empty directories during overcloud image build.
- Removed directories are not related to a specific package, multiple packages affected whenver image has issues.
- Not all empty directories are removed, example in this image /var/log/pcsd empty directory was not removed.

For reference, good image which do not have any missing directory:- https://images.rdoproject.org/centos8/master/rdo_trunk/b3720367a6a0349abcfb06939bed3101/overcloud-full.tar

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-24:

#6

Reviewed: https://review.opendev.org/721255
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5eeadb2114016f1844565b77cab5555b51679190
Submitter: Zuul
Branch: master

commit 5eeadb2114016f1844565b77cab5555b51679190
Author: Gabriele Cerami <email address hidden>
Date: Mon Apr 20 13:20:39 2020 +0100

Enable sensitive logs for OVB HA

to debug correctly http://bugs.launchpad.net/tripleo/+bug/1873770

Change-Id: I7f92b2a815b3fade3bdb933046ce611d0d68c020
Related-Bug: #1873770

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-04-29:

#7

let's make sure this is consistently reproducible.. there is a prior bug and escalation on pretty much the same issue.

Changed in tripleo:
status:	Triaged → Incomplete

wes hayutin (weshayutin) on 2020-05-11

Changed in tripleo:
milestone:	ussuri-rc1 → ussuri-rc3

wes hayutin (weshayutin) on 2020-05-26

Changed in tripleo:
milestone:	ussuri-rc3 → victoria-1

Emilien Macchi (emilienm) on 2020-07-28

Changed in tripleo:
milestone:	victoria-1 → victoria-3

wes hayutin (weshayutin) on 2020-08-07

tags:

removed: alert ci promotion-blocker

Kevin Carter (kevin-carter) on 2020-09-10

Changed in tripleo:
status:	Incomplete → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-04: Related fix proposed to tripleo-heat-templates (stable/train)

#8

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/803438

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-04: Related fix merged to tripleo-heat-templates (stable/train)

#9

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/803438
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/d2def42c539b78510ac8a6ba5c9850f91ef00d88
Submitter: "Zuul (22348)"
Branch: stable/train

commit d2def42c539b78510ac8a6ba5c9850f91ef00d88
Author: Cédric Jeanneret <email address hidden>
Date: Mon Apr 20 14:02:21 2020 +0200

Introduce new HideSensitiveLogs parameter

This one toggles the no_log parameter. Directly related to #1873770 in
order to allow a deeper debug within CI.

    Change-Id: I27f677467263c0e6cc78d775edff55b3811fec1f
    Related-Bug: #1873770
    (cherry picked from commit cabed543fa74014d17a6ac603bc2239237864b21)

tags:

added: in-stable-train

tripleo

OVB fs001 in centos8 master fails to push certificates contents to controllers

Bug Description

Other bug subscribers

Remote bug watches