Error log:
2020-09-30T02:32:51.382582920+00:00 stderr F + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,vs_config,exec"' /etc/config.pp
2020-09-30T02:32:55.185977218+00:00 stderr F <13>Sep 30 02:32:51 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
2020-09-30T02:32:55.185977218+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: (file: /etc/puppet/hiera.yaml)
2020-09-30T02:32:55.186248907+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Warning: Undefined variable '::deploy_config_name';
2020-09-30T02:32:55.186248907+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: (file & line not available)
2020-09-30T02:32:55.244780258+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/6.14/deprecated_language.html
2020-09-30T02:32:55.244780258+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: (file & line not available)
2020-09-30T02:32:55.465105657+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Notice: Compiled catalog for standalone.localdomain in environment production in 0.30 seconds
2020-09-30T02:32:55.540213621+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Error: Could not prefetch vs_config provider 'ovs': Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Changes:
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Events:
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Resources:
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Skipped: 3
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Total: 3
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Time:
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Config retrieval: 0.38
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Last run: 1601433175
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Total: 0.03
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Version:
2020-09-30T02:32:55.547239458+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Config: 1601433175
2020-09-30T02:32:55.547245572+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Puppet: 6.14.0
2020-09-30T02:32:55.617656987+00:00 stderr F + rc=1
Also impacting check job:- https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade
Logs link:
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/90bb089/job-output.txt
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/90bb089/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/90bb089/logs/undercloud/var/log/containers/stdouts/container-puppet-ovn_controller.log.txt.gz
This is happening because openvswitch service is in stopped state. And it went to stopped state after upgrade to openvswitch2.13-2.13. openvswitch needs to be in started state before upgrade get's triggered.
So, what is happening in the upgrade job is:
1. After the repos for master are configured in the job, a " sudo dnf update -y" is executed which updates all packages in the host. This includes upgrading from old "non-layered" openvswitch-2.12 to new layered openvswitch2.13 via rdo-openvswitch -2.13. After this upgrade, openvswitch service is stopped, this is expected and a design decision.
https:/ /logserver. rdoproject. org/51/ 29351/15/ check/periodic- tripleo- ci-centos- 8-standalone- upgrade- master/ 70ef433/ logs/undercloud /home/zuul/ repo_setup_ upgrade. log.txt. gz
2. In the upgrade part, when the logic for ovs is executed, no ovs upgrade is detected (it has been updated previously) so the tast to start openvswitch is skipped:
2020-09-30 04:47:33 | 2020-09-30 04:47:33.428539 | fa163eea- bbcd-be32- 770f-0000000008 2a | TASK | Special treatment for OpenvSwitch bbcd-be32- 770f-0000000008 2a | OK | Special treatment for OpenvSwitch | standalone bbcd-be32- 770f-0000000008 2a | TIMING | Special treatment for OpenvSwitch | standalone | 0:04:06.525630 | 1.05s bbcd-be32- 770f-0000000008 2b | TASK | Always ensure the openvswitch service is enabled and running after upgrades bbcd-be32- 770f-0000000008 2b | SKIPPED | Always ensure the openvswitch service is enabled and running after upgrades | standalone bbcd-be32- 770f-0000000008 2b | TIMING | Always ensure the openvswitch service is enabled and running after upgrades | standalone | 0:04:06.597170 | 0.02s
2020-09-30 04:47:34 | 2020-09-30 04:47:34.478348 | fa163eea-
2020-09-30 04:47:34 | 2020-09-30 04:47:34.479476 | fa163eea-
2020-09-30 04:47:34 | 2020-09-30 04:47:34.525550 | fa163eea-
2020-09-30 04:47:34 | 2020-09-30 04:47:34.550044 | fa163eea-
2020-09-30 04:47:34 | 2020-09-30 04:47:34.551015 | fa163eea-
https:/ /logserver. rdoproject. org/51/ 29351/15/ check/periodic- tripleo- ci-centos- 8-standalone- upgrade- master/ 70ef433/ logs/undercloud /home/zuul/ standalone_ upgrade. log.txt. gz
So openvswitch is stopped in the host and upgrade process fails.
Note that currently the tripleo_ ovs_upgrade. py module does not work with upstream wrapper as the package name is different, downstream is rhosp-openvswitch while upstream is rdo-openvswitch and also it seems it doesn't cover the upgrade from non-layered to layered.
https:/ /opendev. org/openstack/ tripleo- ansible/ raw/branch/ master/ tripleo_ ansible/ ansible_ plugins/ modules/ tripleo_ ovs_upgrade. py
Possible solutions I can see:
1. Remove the "dnf update -y" from repo_setup_ upgrade. log (iiuc there is some task later that would take care of updating host packages?) and fixing the tripleo_ ovs_upgrade. py to cover this upstream case.
2. Keep the dnf update -y in repo_setup_ upgrade. sh and add some task in the script or at a later stage to restart it before undercloud-upgrade tasks.
I think we need upgrades team here.