openstack-infra jobs timing out due to slow connectivity to review.fuel-infra.org

Bug #1535832 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Fuel CI
8.0.x
Fix Committed
High
Vladimir Kuklin

Bug Description

There appears to be connectivity issues. We're getting jobs taking longer than an hour to complete and failing (normal time is < 10 mins). Additionally we're getting timeouts when pulling down modules.

See the following reviews & their job logs:

https://review.openstack.org/#/c/268195/
http://logs.openstack.org/95/268195/5/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/ab27488/

https://review.openstack.org/#/c/269507/
http://logs.openstack.org/07/269507/3/gate/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/423dab0/

https://review.openstack.org/#/c/269110/
http://logs.openstack.org/10/269110/2/gate/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/1c5e327/

https://review.openstack.org/#/c/268214/
http://logs.openstack.org/14/268214/7/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/4315ea9/
http://logs.openstack.org/14/268214/7/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/efc8667/
http://logs.openstack.org/14/268214/7/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/bbf7208/

https://review.openstack.org/#/c/267483/
http://logs.openstack.org/83/267483/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/40a967e/
http://logs.openstack.org/83/267483/2/check/gate-fuel-library-puppet-unit-4.0-dsvm-centos7/327982a/

https://review.openstack.org/#/c/269207/
http://logs.openstack.org/07/269207/2/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/e3ed5b5/

https://review.openstack.org/#/c/268147/
http://logs.openstack.org/47/268147/7/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/7a974d6/

https://review.openstack.org/#/c/269564/
http://logs.openstack.org/64/269564/1/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/1932871/
http://logs.openstack.org/64/269564/1/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/9f38565/
http://logs.openstack.org/64/269564/1/check/gate-fuel-library-puppet-unit-4.0-dsvm-centos7/d727dbe/

https://review.openstack.org/#/c/267569/
http://logs.openstack.org/69/267569/4/gate/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/eb37579/

https://review.openstack.org/#/c/269522/
http://logs.openstack.org/22/269522/1/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/4f1b729/

https://review.openstack.org/#/c/266512/
http://logs.openstack.org/12/266512/7/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/72b7def/
http://logs.openstack.org/12/266512/7/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/0131a4d/

https://review.openstack.org/#/c/268082/
http://logs.openstack.org/82/268082/5/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/316cb01/

https://review.openstack.org/#/c/267989/
http://logs.openstack.org/89/267989/4/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/ae06fa8/

https://review.openstack.org/#/c/269694/
http://logs.openstack.org/94/269694/1/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/4d2d2b2/

https://review.openstack.org/#/c/262587/
http://logs.openstack.org/87/262587/3/gate/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/e1f02c0/
http://logs.openstack.org/87/262587/3/gate/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/8e8a919/

https://review.openstack.org/#/c/263807/
http://logs.openstack.org/07/263807/2/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/6ee2e34/
http://logs.openstack.org/07/263807/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/9685b24/
http://logs.openstack.org/07/263807/2/gate/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/146239f/
http://logs.openstack.org/07/263807/2/gate/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/b27dae2/

https://review.openstack.org/#/c/268133/
http://logs.openstack.org/33/268133/2/gate/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/5be5781/
http://logs.openstack.org/33/268133/2/gate/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/5be6d33/

https://review.openstack.org/#/c/262681/
http://logs.openstack.org/81/262681/6/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/6370b0b/
http://logs.openstack.org/81/262681/6/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/2256497/
http://logs.openstack.org/81/262681/6/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/26503d8/

https://review.openstack.org/#/c/265945/
http://logs.openstack.org/45/265945/12/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/c7239a7/
http://logs.openstack.org/45/265945/12/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/33ee67d/
http://logs.openstack.org/45/265945/12/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/dca607a/

https://review.openstack.org/#/c/269527/
http://logs.openstack.org/27/269527/2/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/d40c636/

https://review.openstack.org/#/c/269749/
http://logs.openstack.org/49/269749/3/check/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/a372690/
http://logs.openstack.org/49/269749/3/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/e3aa520/

https://review.openstack.org/#/c/263426/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-3.4-dsvm-centos7/30669a9/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/f414290/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/0dfda41/
http://logs.openstack.org/26/263426/5/check/gate-fuel-library-puppet-unit-4.0-dsvm-centos7/c803b3a/

https://review.openstack.org/#/c/267623/
http://logs.openstack.org/23/267623/4/gate/gate-fuel-library-puppet-unit-3.3-dsvm-centos7/978c733/

https://review.openstack.org/#/c/269585/
http://logs.openstack.org/85/269585/2/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/d2befb7/
http://logs.openstack.org/85/269585/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/68d3eca/

https://review.openstack.org/#/c/269607/
http://logs.openstack.org/07/269607/1/check/gate-fuel-library-puppet-unit-3.6-dsvm-centos7/e4941ea/
http://logs.openstack.org/07/269607/1/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/0aa84e8/

https://review.openstack.org/#/c/269676/
http://logs.openstack.org/76/269676/2/check/gate-fuel-library-puppet-unit-3.7-dsvm-centos7/5d729b0/
http://logs.openstack.org/76/269676/2/check/gate-fuel-library-puppet-unit-3.8-dsvm-centos7/d9c2e1e/

Tags: area-devops
Changed in fuel:
assignee: nobody → Fuel DevOps (fuel-devops)
Revision history for this message
Igor Shishkin (teran) wrote :

CI team, please enable dmesg gathering the same was as we're getting those logs.
As I got from OpenStack Infra folks there's a way to enable it in job configuration.
This will really help to get the real reason of those failure.

For now I have two suspicions:
- instance running our test is get out of some limits(should be reflected in dmesg)
- our Gerrit is out of it's RPS capacity(no clear sings of it on graphs, but it's possible)

Having dmesg logging will make us possible to get it.

Reassigning to CI team to enable logging.

Changed in fuel:
assignee: Fuel DevOps (fuel-devops) → Fuel CI (fuel-ci)
Revision history for this message
Igor Belikov (ibelikov) wrote :

Igor, I created a separate bug for log gathering part: https://bugs.launchpad.net/fuel/+bug/1536160

Revision history for this message
Aleksandra Fedorova (bookwar) wrote :

Please proceed with git replication to GitHub workaround.

Changed in fuel:
assignee: Fuel CI (fuel-ci) → Fuel DevOps (fuel-devops)
tags: added: area-devops
Revision history for this message
Igor Shishkin (teran) wrote :

@Alexandra, we're researching what would be the fastest way to achieve to dedication those repos to github or dedicated VM whatever is faster.

Changed in fuel:
status: New → In Progress
assignee: Fuel DevOps (fuel-devops) → Andrey Nikitin (heos)
Revision history for this message
Andrey Nikitin (heos) wrote :

We've added those repositories on Github and enabled replication on them from Fuel Gerrit. The first replication of the projects on github works slowly.

List of projects are replication on Github: https://github.com/fuel-infra.

So, I think we can use them as workaround or to find a root cause of the problem.

Revision history for this message
Andrey Nikitin (heos) wrote :

I'm going to reduce Importance due to we have workaround for the problem.

Changed in fuel:
importance: Critical → High
assignee: Andrey Nikitin (heos) → Fuel CI (fuel-ci)
Revision history for this message
Andrey Nikitin (heos) wrote :

Fuel CI team, please take care about the bug, when replication will finished.

Revision history for this message
Andrey Nikitin (heos) wrote :

Replication to Github is finished, please replace review.fuel-infra.org on those jobs.

Changed in fuel:
status: In Progress → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/270277

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Tags were not propagated, please fix so we can switch them out.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/270313

Revision history for this message
Andrey Nikitin (heos) wrote :

All task of the projects were replicated too. Please check.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to fuel-infra/project-configs (master)

Related fix proposed to branch: master
Change author: Andrey Nikitin <email address hidden>
Review: https://review.fuel-infra.org/16341

Changed in fuel:
status: New → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to fuel-infra/project-configs (master)

Reviewed: https://review.fuel-infra.org/16341
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 1e6e8bdbdfa864cb40f166da59c9bb1f2afb3061
Author: Andrey Nikitin <email address hidden>
Date: Thu Jan 21 13:25:36 2016

Added 'read' permission to fuel-infra-github group

This permissions is needed to allow replication the projects
on github.

Change-Id: I4a85eb80aa779ad103e25e980c1b195d20f16f0b
Related-bug: #1535832

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/270313
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=abd991b2305fd0ba52c29211feff3325439489bb
Submitter: Jenkins
Branch: master

commit abd991b2305fd0ba52c29211feff3325439489bb
Author: Alex Schultz <email address hidden>
Date: Wed Jan 20 09:42:55 2016 -0700

    Update CI tasks and module fixtures

    This change updates the fixtures files for the modules within
    fuel-library to use the relative path to the upstream modules used by
    fuel-library. With this change we are also adding an initial prep task
    that is used by CI to pull down all the modules first and only one time
    to reduce the amount of repeated calls to the upstream git repositories.

    Included in this change is also an update to exclude all the upstream
    modules from the spec and lint tasks as those are maintained in the
    upstream modules and should not be done when testing fuel-library.

    This change also adds documentation to the README.md on how to run the
    tests from the root of fuel-library.

    Change-Id: I9cde21b94a8367fd45cf046af48d7618076a385c
    Related-Bug: #1535832

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/270277
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f26a6b23b08280d0ccdbb9b5252abbd9ba49b1da
Submitter: Jenkins
Branch: master

commit f26a6b23b08280d0ccdbb9b5252abbd9ba49b1da
Author: Igor Belikov <email address hidden>
Date: Wed Jan 20 18:57:22 2016 +0300

    Switch librarian git links to github mirror

    This change also updates the puppetfile validation script to allow for
    github.com/fuel-infra

    Change-Id: I274ec4c2f12b9effb76d59499d58ee713c78664d
    Related-Bug: #1535832
    Co-Authored-By: Alex Schultz <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/8.0)

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274712

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/274712
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0375f153b5e7d960e605e414e630525ce23bc3bb
Submitter: Jenkins
Branch: stable/8.0

commit 0375f153b5e7d960e605e414e630525ce23bc3bb
Author: Alex Schultz <email address hidden>
Date: Wed Jan 20 09:42:55 2016 -0700

    Update CI tasks and module fixtures

    This change updates the fixtures files for the modules within
    fuel-library to use the relative path to the upstream modules used by
    fuel-library. With this change we are also adding an initial prep task
    that is used by CI to pull down all the modules first and only one time
    to reduce the amount of repeated calls to the upstream git repositories.

    Included in this change is also an update to exclude all the upstream
    modules from the spec and lint tasks as those are maintained in the
    upstream modules and should not be done when testing fuel-library.

    This change also adds documentation to the README.md on how to run the
    tests from the root of fuel-library.

    Change-Id: I9cde21b94a8367fd45cf046af48d7618076a385c
    Related-Bug: #1535832

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.