Name resolution failure on gate job

logstash query: message:"git.openstack.org: Temporary failure in name resolution" AND filename:"console.html"

20 hits in logstash

Related fix proposed to branch: master
Review: https://review.openstack.org/68280

Reviewed: https://review.openstack.org/68280
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=d8e4e1ba0078b9b50575793bbc8344e97c5853e7
Submitter: Jenkins
Branch: master

commit d8e4e1ba0078b9b50575793bbc8344e97c5853e7
Author: Joe Gordon <email address hidden>
Date: Tue Jan 21 14:40:46 2014 -0800

    Add fingerprint for bug 1270382

    bug 1270382 is a infra bug

    Change-Id: I040b007634cff3d2ce21c6cf85c008167f385901
    Related-Bug: #1270382

The above logstash query does not cover failures like the following:

2014-01-16 19:23:47.779 | Downloading/unpacking dnspython>=1.9.4 (from -r /home/jenkins/workspace/gate-swift-docs/requirements.txt (line 1))
2014-01-16 19:23:47.779 | Error <urlopen error [Errno -3] Temporary failure in name resolution> while getting http://pypi.openstack.org/openstack/dnspython/dnspython-1.11.1.zip (from http://pypi.openstack.org/openstack/dnspython/)
2014-01-16 19:23:47.779 | Cleaning up...

See: http://logs.openstack.org/07/66407/2/gate/gate-swift-docs/6b1df29/console.html.gz

It would seem we should just search for message:"Temporary failure in name resolution".

Reviewed: https://review.openstack.org/68961
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=314486a36aa3e1ff46518b323bfee8cb52c8ea5b
Submitter: Jenkins
Branch: master

commit 314486a36aa3e1ff46518b323bfee8cb52c8ea5b
Author: Peter Portante <email address hidden>
Date: Fri Jan 24 11:41:38 2014 -0500

    Expand DNS failures to catch all "errno" string output

    Related bug 1270382

    Change-Id: I6a958d0575fb0b5ece0bdc3b3ce50990615b5557

This seems to be mainly effecting slaves in Rackspace (ironic since that's where the DNS record in question is hosted). My best guess is that their resolvers get overloaded from time to time.

One possible workaround would be to prime a local resolver cache on each slave with multiple retries to look up the names of each of the hosts we need to communicate with during the course of normal job runs.

Sounds like there isn't devstack work needed here

The underlying issue here was rackspace would blacklist IPs using their DNS resolvers if they queried too much. When these IPs were recycled and picked up by our slaves the DNS blacklist for the IPs was still in place preventing name resolution. To correct this we have installed unbound as local caching forwarding resolvers on all of our slaves. Instead of talking to rackspace and hpcloud DNS servers we now talk to google dns servers.

My job still failed on gate on 7/16/2014:

https://review.openstack.org/#/c/107181/

http://logs.openstack.org/81/107181/8/gate/gate-cookbook-monasca-thresh-chef-unit/b1b0322/console.html

2014-07-16 22:01:43.289 | Started by user anonymous
2014-07-16 22:01:43.292 | Building remotely on bare-precise-rax-dfw-928290 in workspace /home/jenkins/workspace/gate-cookbook-monasca-thresh-chef-unit
2014-07-16 22:01:50.040 | [gate-cookbook-monasca-thresh-chef-unit] $ /bin/bash /tmp/hudson4648257686961539642.sh
2014-07-16 22:01:52.009 | [gate-cookbook-monasca-thresh-chef-unit] $ /bin/bash -xe /tmp/hudson4842354825482043824.sh
2014-07-16 22:01:52.015 | + /usr/local/jenkins/slave_scripts/gerrit-git-prep.sh https://review.openstack.org git://git.openstack.org
2014-07-16 22:01:52.213 | Triggered by: https://review.openstack.org/107181
2014-07-16 22:01:52.213 | + [[ ! -e .git ]]
2014-07-16 22:01:52.213 | + ls -a
2014-07-16 22:01:52.216 | .
2014-07-16 22:01:52.216 | ..
2014-07-16 22:01:52.216 | + rm -fr '.[^.]*' '*'
2014-07-16 22:01:52.219 | + '[' -d /opt/git/stackforge/cookbook-monasca-thresh/.git ']'
2014-07-16 22:01:52.344 | + git clone file:///opt/git/stackforge/cookbook-monasca-thresh .
2014-07-16 22:01:53.462 | Cloning into '.'...
2014-07-16 22:02:03.988 | + git remote set-url origin git://git.openstack.org/stackforge/cookbook-monasca-thresh
2014-07-16 22:02:04.075 | + git remote update
2014-07-16 22:02:04.368 | Fetching origin
2014-07-16 22:02:04.380 | fatal: unable to connect to git.openstack.org:
2014-07-16 22:02:04.380 | git.openstack.org: Temporary failure in name resolution
2014-07-16 22:02:04.380 |
2014-07-16 22:02:04.381 | error: Could not fetch origin
2014-07-16 22:02:04.381 | + echo 'The remote update failed, so garbage collecting before trying again.'
2014-07-16 22:02:04.381 | The remote update failed, so garbage collecting before trying again.
2014-07-16 22:02:04.382 | + git gc
2014-07-16 22:02:20.297 | + git remote update
2014-07-16 22:02:20.303 | Fetching origin
2014-07-16 22:02:20.308 | fatal: unable to connect to git.openstack.org:
2014-07-16 22:02:20.308 | git.openstack.org: Temporary failure in name resolution
2014-07-16 22:02:20.308 |
2014-07-16 22:02:20.308 | error: Could not fetch origin
2014-07-16 22:02:20.317 | Build step 'Execute shell' marked build as failure
2014-07-16 22:02:20.426 | [SCP] Copying console log.
2014-07-16 22:02:20.972 | [SCP] Trying to create /srv/static/logs/81/107181/8/gate
2014-07-16 22:02:20.988 | [SCP] Trying to create /srv/static/logs/81/107181/8/gate/gate-cookbook-monasca-thresh-chef-unit
2014-07-16 22:02:20.990 | [SCP] Trying to create /srv/static/logs/81/107181/8/gate/gate-cookbook-monasca-thresh-chef-unit/b1b0322
2014-07-16 22:02:21.004 | Finished: FAILURE

Also running into the same error, but retriggering the job solved the problem for me (i know that this is not the solution ;) )