Hello,
We are doing wallaby branching preparation, and have created a periodic version of ovb job for wallaby branch.
Patch to create new job: https://review.rdoproject.org/r/c/rdo-jobs/+/32626
Testproject: https://review.rdoproject.org/r/c/testproject/+/28458
Deployment is failing during the Baremetal provisioning and metalsmith instances are going into error state.
https://logserver.rdoproject.org/58/28458/70/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/c9cb1ca/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
(undercloud) [zuul@undercloud ~]$ metalsmith list
+--------------------------------------+---------------------------+--------------------------------------+-------------------------+-------+--------------+
| UUID | Node Name | Allocation UUID | Hostname | State | IP Addresses |
+--------------------------------------+---------------------------+--------------------------------------+-------------------------+-------+--------------+
| 48b14792-1d55-43ba-aef2-13c29f27b6a4 | baremetal-28458-70-3819-0 | 0f620f82-5af0-4897-9836-f2c368f35177 | overcloud-controller-0 | ERROR | |
| 6174875e-3a06-4d38-9241-1a938a110aba | baremetal-28458-70-3819-1 | d45c2e1b-2b74-4691-b225-913542371a6b | overcloud-controller-1 | ERROR | |
| 454718e1-41dd-4a81-a126-b36b4bdd93ac | baremetal-28458-70-3819-2 | f429f182-9716-4082-aba4-c2cf476f4e8a | overcloud-controller-2 | ERROR | |
| 57728f8b-7b32-4e28-8565-6e4cf8048f27 | baremetal-28458-70-3819-3 | b332d9b2-ff95-4740-9615-897eb9a91f9c | overcloud-novacompute-0 | ERROR | |
+--------------------------------------+---------------------------+--------------------------------------+-------------------------+-------+--------------+
Neutron DHCP agent logs are full of errors:-
https://logserver.rdoproject.org/58/28458/70/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/c9cb1ca/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.1.gz
~~~
---------------------------------------------------------------------------
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 227, in call_driver
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 266, in enable
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 703, in wait_until_true
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent while not predicate():
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 274, in _enable
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent self.disable(retain_port=True, block=True)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 300, in disable
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent self._get_process_manager().disable()
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/external_process.py", line 114, in disable
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent self._kill_process(sig, pid)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/external_process.py", line 133, in _kill_process
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent run_as_root=self.run_as_root)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 135, in execute
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent cmd, process_input, addl_env)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 114, in execute_rootwrap_daemon
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent LOG.error("Rootwrap error running command: %s", cmd)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent self.force_reraise()
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent raise self.value
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 111, in execute_rootwrap_daemon
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent returncode, _stdout, _stderr = client.execute(cmd, process_input)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 169, in execute
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent res = self._run_one_command(proxy, cmd, stdin)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/client.py", line 154, in _run_one_command
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent res = proxy.run_one_command(cmd, stdin)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "<string>", line 2, in run_one_command
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent raise convert_to_error(kind, result)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent multiprocessing.managers.RemoteError:
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent ---------------------------------------------------------------------------
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent Unserializable message: Traceback (most recent call last):
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent send(msg)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 129, in send
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent s = self.dumps(obj)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 171, in dumps
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent **kw).encode(obj)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent chunks = self.iterencode(o, _one_shot=True)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent return _iterencode(o, 0)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent return super(RpcJSONEncoder, self).default(o)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent o.__class__.__name__)
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent TypeError: Object of type 'ValueError' is not JSON serializable
2021-03-26 10:32:52.522 82297 ERROR neutron.agent.dhcp.agent
~~~
