Job failed due to no devstack directory

A bit more triage shows http://logs.openstack.org/54/109754/13/check/check-tempest-dsvm-full/10d7fe8/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-09-03_08_50_25_414. Looks like git hit a TLS error fetching nova refs from the git mirrors. This made workspace setup short circuit and fail without cloning and updating devstack.

Failed in Juno gate twice in the same run:

http://logs.openstack.org/89/131189/1/gate/gate-tempest-dsvm-neutron/077f938
http://logs.openstack.org/89/131189/1/gate/gate-tempest-dsvm-neutron-full-2/454e40b

Copying error message into comments for completeness:

fatal: unable to access 'https://git.openstack.org/openstack/nova/': GnuTLS recv error (-9): A TLS packet with unexpected length was received.

Seeing it here too:

http://logs.openstack.org/47/133947/3/gate/gate-tempest-dsvm-neutron-heat-slow/d39bd4c/logs/devstack-gate-setup-workspace-new.txt.gz

I'm trying to get a decent e-r query for this.

Querying on this isn't good enough since there is a retry:

message:"error\: RPC failed; result=56, HTTP code = 0" AND tags:"console"

So it's not 100% fail. We could add a better error message in that log when the workspace setup fails and we've hit the total number of retries.

Until then we probably need to query on the failure in the console log:

http://logs.openstack.org/47/133947/3/gate/gate-tempest-dsvm-neutron-heat-slow/d39bd4c/console.html#_2014-11-18_12_44_21_376

message:"/opt/stack/new/devstack-gate/devstack-vm-gate.sh: line" AND message:"cd: /opt/stack/new/devstack: No such file or directory" AND tags:"console"

There are 161 hits for that:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiL29wdC9zdGFjay9uZXcvZGV2c3RhY2stZ2F0ZS9kZXZzdGFjay12bS1nYXRlLnNoOiBsaW5lXCIgQU5EIG1lc3NhZ2U6XCJjZDogL29wdC9zdGFjay9uZXcvZGV2c3RhY2s6IE5vIHN1Y2ggZmlsZSBvciBkaXJlY3RvcnlcIiBBTkQgdGFnczpcImNvbnNvbGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQxNjQxNTI4NjE5Nn0=

Related fix proposed to branch: master
Review: https://review.openstack.org/135654

Related fix proposed to branch: master
Review: https://review.openstack.org/135656

Reviewed: https://review.openstack.org/135654
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=5415de703547c0828bee6b89a2534c9f89a9370f
Submitter: Jenkins
Branch: master

commit 5415de703547c0828bee6b89a2534c9f89a9370f
Author: Matt Riedemann <email address hidden>
Date: Wed Nov 19 08:53:13 2014 -0800

    Add query for infra bug 1365046

    161 hits in 7 days, check and gate, all failures.

    There is a better error message to be had in
    devstack-gate so I'm going to work a separate change
    for that and then we can update this query, but for
    now this is the only 100% fail since there is a retry
    loop involved in devstack-gate when doing the git
    remote update.

    Change-Id: I2ac631ee42b7bdf6f4bcc3d61c4dadea321a23d9
    Related-Bug: #1365046

Reviewed: https://review.openstack.org/135656
Committed: https://git.openstack.org/cgit/openstack-infra/devstack-gate/commit/?id=aed2c8b87cafd43a53775622f19312b6dcb47fe5
Submitter: Jenkins
Branch: master

commit aed2c8b87cafd43a53775622f19312b6dcb47fe5
Author: Matt Riedemann <email address hidden>
Date: Wed Nov 19 08:58:45 2014 -0800

    Give a more specific error message when giving up in git_remote_update

    git_remote_update is intermittently failing in the gate but we don't
    have a good error message in the devstack logs for this specific case
    due to the retry loop (we don't get 100% failure on a query for the
    error messages in there today).

    This change adds a specific error message when we fail the retry
    attempts and exit so we can fingerprint on this with an elastic recheck
    query.

    Related-Bug: #1365046

    Change-Id: Ibb3269a534d5b28b2d7d8a531e30a77ba6011f2d

Related fix proposed to branch: master
Review: https://review.openstack.org/142169

Reviewed: https://review.openstack.org/142169
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=7783356a20ae1a151f86d60599ccc19304e5da10
Submitter: Jenkins
Branch: master

commit 7783356a20ae1a151f86d60599ccc19304e5da10
Author: Mauro S. M. Rodrigues <email address hidden>
Date: Tue Dec 16 17:43:17 2014 +0000

    Update query for Bug #1365046

    The Change Id Ibb3269a534d5b28b2d7d8a531e30a77ba6011f2d added a more
    specific message which help us to cover this bug more accurately.

    Change-Id: I942c2884d75eb138e9646198266fad61d080c8f7
    Related-Bug: #1365046

Might be inadvertently fixed.

I've seen this bug in review https://review.openstack.org/#/c/187571/6

Elastic Recheck recognized it as bug 1455123