rdo centos7 ovb 3 ctrl 1 comp featureset 001 timeout overcloud deploy

i am leaning towards closing this i haven't been able to find more examples of it

found some more examples there from yesterday [1][2] so i will not close, but this is not consistent it may be a race... from [3] today the fails are quite infrequent

[1] http://logs.rdoproject.org/93/675993/8/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/86267f9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2] http://logs.rdoproject.org/51/690151/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/cf913d8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[3] https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001

this just blocked the train promotion at [1]

         2019-10-24 00:32:18 | "[2019/10/24 12:31:10 AM] [INFO] Running ip route add 0.0.0.0/0 via 10.0.0.1 dev br-ex",
        2019-10-24 00:32:18 | "[2019/10/24 12:31:10 AM] [WARNING] Error in 'ip route add 0.0.0.0/0 via 10.0.0.1 dev br-ex', restarting br-ex:",
        2019-10-24 00:32:18 | "Unexpected error while running command.",
        2019-10-24 00:32:18 | "Command: /sbin/ip route add 0.0.0.0/0 via 10.0.0.1 dev br-ex",
        2019-10-24 00:32:18 | "Exit code: 1",
        2019-10-24 00:32:18 | "Stdout: u''",
        2019-10-24 00:32:18 | "Stderr: u'Cannot find device \"br-ex\"\\n'", "

[1] http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train/2141ac0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

so it is intermittent but it definitely promotion-blocker re-adding tag and alert too

adding note for now but as per irc earlier with ykarel he reckons this is more to do with image prepare and that the br-ex thing is a red herring you can find it in a good log see http://logs.rdoproject.org/56/690856/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/1f3f161/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

11:27 < ykarel> marios|ruck, see
        11:41 < marios|ruck> http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train/2141ac0/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

i updated the description just now... as per comment 5 the 'br-ex' thing is a red herring.

not clear what the root is here yet

17:16 < ykarel> was thinking if https://review.opendev.org/#/q/I3686531ab383951edef60ac62dc259d39155704a related
17:16 * ykarel fetch logs
17:17 < ykarel> mwhahaha,
https://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train/2141ac0/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz
17:18 <@mwhahaha> K I'll review the logs and see
17:18 < ykarel> ack
17:19 < ykarel> corresponding bug https://bugs.launchpad.net/tripleo/+bug/1849101
17:21 <@mwhahaha> if seen it take a long time if the shuffle ends up being bad and locks a bunch
17:21 <@mwhahaha> eg taking 30+ mins instead of the usual ~15
17:21 <@mwhahaha> so let me look at the logs
17:22 <@mwhahaha> ykarel: so actually that log doesn't have that patch
17:22 <@mwhahaha> ykarel: because you can see 2019-10-23 23:23:17,401 19304 WARNING tripleo_common.image.image_uploader [ ] No lock
                  information provided for layer sha256:2888d4f69e7794e69ec5928c02c8039a56f1760c22381c7ac928a940ac2d01a1
17:22 < ykarel> mwhahaha, nope it was from yesterday
17:23 < ykarel> that's why i asked if that patch can fix this issue
17:23 <@mwhahaha> it can help
17:23 <@mwhahaha> because it'll prevent the duplicate fetching

looks like the series from comment #7 is now merged
https://review.opendev.org/#/q/I3686531ab383951edef60ac62dc259d39155704a

and there are some green builds
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train

though not yet for the periodic... but i don't see any timeouts there

hoping this is resolved by alex patch https://review.opendev.org/#/c/690111/

leave it around for now especially until we see some green an no timeouts on the periodic

per comment #8 just checked again and don't see any timeout but many green at https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001

both check and periodic closing this out for now

we can re-file when we have something specific.