I can't see any major difference in the nodes between a good log [1] and a timeout out [2], except the timeout one has less free memory (but same total)
[1] * MemTotal: 8150828 kB
MemFree: 1240728 kB
[2] * MemTotal: 8150828 kB
MemFree: 293704 kB
Similarly the cpuinfo log looks the same good @ [3] bad at [4]
I see in the errors log an issue reaching rabbit on controller-1 with retries, I don't know if that is directly related
2021-08-05 18:18:42.101 ERROR /var/log/containers/nova/nova-api.log: 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [116db7bd-d0dd-4522-b666-8282a6acc71f] AMQP server on overcloud-controller-1.internalapi.localdomain:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
I can't see any major difference in the nodes between a good log [1] and a timeout out [2], except the timeout one has less free memory (but same total)
[1] * MemTotal: 8150828 kB
MemFree: 1240728 kB
[2] * MemTotal: 8150828 kB
MemFree: 293704 kB
Similarly the cpuinfo log looks the same good @ [3] bad at [4]
I see in the errors log an issue reaching rabbit on controller-1 with retries, I don't know if that is directly related
2021-08-05 18:18:42.101 ERROR /var/log/ containers/ nova/nova- api.log: 17 ERROR oslo.messaging. _drivers. impl_rabbit [-] [116db7bd- d0dd-4522- b666-8282a6acc7 1f] AMQP server on overcloud- controller- 1.internalapi. localdomain: 5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
[1] https:/ /logserver. rdoproject. org/openstack- periodic- integration- stable1/ opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-wallaby/ d385cc6/ logs/undercloud /var/log/ extra/meminfo. txt.gz /logserver. rdoproject. org/61/ 34861/1/ check/periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-wallaby/ 894cb7e/ logs/undercloud /var/log/ extra/meminfo. txt.gz /logserver. rdoproject. org/openstack- periodic- integration- stable1/ opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-wallaby/ d385cc6/ logs/undercloud /var/log/ extra/cpuinfo. txt.gz /logserver. rdoproject. org/61/ 34861/1/ check/periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-wallaby/ 894cb7e/ logs/undercloud /var/log/ extra/cpuinfo. txt.gz /logserver. rdoproject. org/61/ 34861/1/ check/periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-wallaby/ 894cb7e/ logs/overcloud- controller- 0/var/log/ extra/errors. txt.gz
[2] https:/
[3] https:/
[4] https:/
[5] https:/