openstack-infra jobs timing out due to slow connectivity to review.fuel-infra.org
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Committed
|
High
|
Fuel CI | ||
8.0.x |
Fix Committed
|
High
|
Vladimir Kuklin |
Bug Description
There appears to be connectivity issues. We're getting jobs taking longer than an hour to complete and failing (normal time is < 10 mins). Additionally we're getting timeouts when pulling down modules.
See the following reviews & their job logs:
https:/
http://
https:/
http://
https:/
http://
https:/
http://
http://
http://
https:/
http://
http://
https:/
http://
https:/
http://
https:/
http://
http://
http://
https:/
http://
https:/
http://
https:/
http://
http://
https:/
http://
https:/
http://
https:/
http://
https:/
http://
http://
https:/
http://
http://
http://
http://
https:/
http://
http://
https:/
http://
http://
http://
https:/
http://
http://
http://
https:/
http://
https:/
http://
http://
https:/
http://
http://
http://
http://
https:/
http://
https:/
http://
http://
https:/
http://
http://
https:/
http://
http://
Changed in fuel: | |
assignee: | nobody → Fuel DevOps (fuel-devops) |
Changed in fuel: | |
status: | In Progress → Fix Committed |
CI team, please enable dmesg gathering the same was as we're getting those logs.
As I got from OpenStack Infra folks there's a way to enable it in job configuration.
This will really help to get the real reason of those failure.
For now I have two suspicions:
- instance running our test is get out of some limits(should be reflected in dmesg)
- our Gerrit is out of it's RPS capacity(no clear sings of it on graphs, but it's possible)
Having dmesg logging will make us possible to get it.
Reassigning to CI team to enable logging.