[regression] It is not possible to resolve slave nodes by host names

Ivan, what's in your /etc/resolv.conf? Did you repeat running fuelmenu after deployment?

[root@nailgun ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search test.domain.local
nameserver 8.8.8.8

I've configured DNS server 8.8.8.8 for OpenStack nodes deployment, not for master node.

It's because of NetworkManager. It reset /etc/resolv.conf. We've been pushing toward disabling it from the very beginning of deployment. I will investigate how to block it from updating /etc/resolv.conf and again about disabling it. To work around, just add the IP of your Fuel Master as the first nameserver

Matthew, JFI, we didn't use NetworkManager on the master node with VirtualBox since 8.0 without any problems:
https://github.com/openstack/fuel-main/blob/stable/8.0/virtualbox/functions/product.sh#L231

https://review.openstack.org/#/c/304785/0

Reviewed: https://review.openstack.org/304785
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0717a769afdd8fff0b120f570a0d03a91c8c2be9
Submitter: Jenkins
Branch: master

commit 0717a769afdd8fff0b120f570a0d03a91c8c2be9
Author: Matthew Mosesohn <email address hidden>
Date: Tue Apr 12 19:48:46 2016 +0300

    Remove NM on Fuel Master, fix dhclient-enter-hooks

    This patch blocks NetworkManager and dhclient from modifying
    /etc/resolv.conf.

    NetworkManager overwrites /etc/resolv.conf, occasionally
    breaking local node resolution. l23network module will remove
    it cleanly.

    After upgrade to CentOS7, the proper script for DHCP hook disabling
    is /etc/dhcp/dhclient-enter-hooks, which is now renamed.

    Change-Id: I9bf6a4ce2f59f320cc1681fe108da3dd4de70ed9
    Closes-Bug: #1567957

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/307449

Reviewed: https://review.openstack.org/307449
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=36b27f832137da346074b1651931517b5068c01d
Submitter: Jenkins
Branch: stable/mitaka

commit 36b27f832137da346074b1651931517b5068c01d
Author: Matthew Mosesohn <email address hidden>
Date: Tue Apr 12 19:48:46 2016 +0300

    Remove NM on Fuel Master, fix dhclient-enter-hooks

    This patch blocks NetworkManager and dhclient from modifying
    /etc/resolv.conf.

    NetworkManager overwrites /etc/resolv.conf, occasionally
    breaking local node resolution. l23network module will remove
    it cleanly.

    After upgrade to CentOS7, the proper script for DHCP hook disabling
    is /etc/dhcp/dhclient-enter-hooks, which is now renamed.

    Change-Id: I9bf6a4ce2f59f320cc1681fe108da3dd4de70ed9
    Closes-Bug: #1567957
    (cherry picked from commit 0717a769afdd8fff0b120f570a0d03a91c8c2be9)

[root@srvr418 etc]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|-------------|---------|------------|-------------------|-------------------|---------------|--------|---------
9 | ready | vse2100-19 | 1 | 10.20.0.9 | a0:42:3f:01:b0:d1 | compute | | True | 1
6 | ready | srvrorcl502 | 1 | 10.20.0.5 | 00:10:e0:3b:94:4a | compute | | True | 1
11 | ready | srvrhp604 | 1 | 10.20.0.7 | 2c:44:fd:96:24:88 | cinder | | True | 1
8 | ready | srvrhp601 | 1 | 10.20.0.6 | 2c:44:fd:96:49:c4 | controller, mongo | | True | 1
4 | ready | srvrorcl501 | 1 | 10.20.0.3 | 00:10:e0:3c:7e:48 | compute | | True | 1
10 | ready | srvr318 | 1 | 10.20.0.10 | 00:25:90:3b:ab:d8 | compute | | True | 1
12 | ready | srvrorcl500 | 1 | 10.20.0.4 | 00:10:e0:56:10:22 | compute | | True | 1
[root@srvr418 etc]# ping vse2100-19
PING vse2100-19.hitachi-cta.com (10.20.0.9) 56(84) bytes of data.
64 bytes from vse2100-19.hitachi-cta.com (10.20.0.9): icmp_seq=1 ttl=64 time=0.270 ms
64 bytes from vse2100-19.hitachi-cta.com (10.20.0.9): icmp_seq=2 ttl=64 time=0.311 ms
^C
--- vse2100-19.hitachi-cta.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.270/0.290/0.311/0.026 ms
[root@srvr418 etc]# ping node-9
ping: unknown host node-9

My resolv.conf file looks correct:

[root@srvr418 etc]# more /etc/resolv.conf
search hitachi-cta.com
domain hitachi-cta.com
nameserver 10.20.0.2
nameserver 192.168.35.101

[root@srvr418 etc]# cat /etc/fuel_build_id:
cat: /etc/fuel_build_id:: No such file or directory
[root@srvr418 etc]# cat /etc/fuel_release
8.0
[root@srvr418 etc]# cat /etc/fuel_build_id
cat: /etc/fuel_build_id: No such file or directory
[root@srvr418 etc]# cat /etc/fuel_
fuel_openstack_version fuel_release
[root@srvr418 etc]# cat /etc/fuel_
fuel_openstack_version fuel_release
[root@srvr418 etc]# cat /etc/fuel_openstack_version
liberty-8.0
[root@srvr418 etc]# rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun'
fuel-notify-8.0.0-1.mos7799.noarch
network-checker-8.0.0-1.mos58.x86_64
fuel-bootstrap-cli-8.0.0-1.mos237.noarch
fuel-migrate-8.0.0-1.mos7799.noarch
python-fuelclient-8.0.0-1.mos266.noarch
fuel-misc-8.0.0-1.mos7799.noarch
fuel-provisioning-scripts-8.0.0-1.mos8253.noarch
fuel-docker-images-8.0.0-1.x86_64
fuelmenu-8.0.0-1.mos232.noarch
fuel-agent-8.0.0-1.mos237.noarch
fuel-mirror-8.0.0-1.mos111.noarch
fuel-8.0.0-1.mos6270.noarch
fuel-openstack-metadata-8.0.0-1.mos8253.noarch
fuel-release-8.0.0-1.mos6270.noarch
fuel-dockerctl-8.0.0-1.mos7799.noarch
fuel-library8.0-8.0.0-1.mos7799.noarch
python-packetary-8.0.0-1.mos111.noarch
fuel-bootstrap-image-8.0.0-1.noarch

[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------------------+---------+-------------+-------------------+-------+---------------+--------+---------
 4 | discover | Untitled (05:c9) | | 10.109.0.4 | 64:83:22:e0:05:c9 | | | 1 |
 5 | discover | Untitled (f5:9a) | | 10.109.0.7 | 64:c6:ce:f6:f5:9a | | | 1 |
 1 | discover | node1 | | 10.109.0.5 | 64:4c:ef:df:c3:60 | | | 1 |
 7 | discover | Untitled (e1:88) | | 10.109.0.10 | 64:58:8c:9b:e1:88 | | | |
 9 | discover | Untitled (5a:89) | | 10.109.0.9 | 64:b5:6a:78:5a:89 | | | |
 8 | discover | Untitled (f2:46) | | 10.109.0.11 | 64:52:30:21:f2:46 | | | |
 2 | discover | node2 | | 10.109.0.6 | 64:3d:10:c5:e9:f9 | | | 1 |
 6 | discover | Untitled (2f:6e) | | 10.109.0.8 | 64:bf:5f:90:2f:6e | | | |
 3 | discover | Untitled (28:7d) | | 10.109.0.3 | 64:23:b2:d0:28:7d | | | 1 |

[root@nailgun ~]# ping node2
ping: unknown host node2
[root@nailgun ~]# ping node-2
ping: unknown host node-2

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id: 434
cat /etc/fuel_build_number: 434
cat /etc/fuel_release: 9.0
cat /etc/fuel_openstack_version: mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-misc-9.0.0-1.mos8421.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8421.noarch
 rubygem-astute-9.0.0-1.mos748.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8718.noarch
 network-checker-9.0.0-1.mos73.x86_64
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-openstack-metadata-9.0.0-1.mos8718.noarch
 fuel-notify-9.0.0-1.mos8421.noarch
 nailgun-mcagents-9.0.0-1.mos748.noarch
 python-fuelclient-9.0.0-1.mos319.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8421.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8421.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2713.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8718.noarch

[root@nailgun ~]# cat /etc/resolv.conf
search test.domain.local
domain test.domain.local
nameserver 8.8.8.8
nameserver 10.109.0.1

it looks like the issue wasn't fixed, Matthew, could you please take a look?

Please provide diagnostic snapshot with failed verification

Problem is not exists anymore:
[root@nailgun ~]# ssh node-1
Warning: Permanently added 'node-1' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-87-generic x86_64)

 * Documentation: https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@node-1:~# . openrc

Verified on:
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 443
cat /etc/fuel_build_number:
 443
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0

Reproduceed on 460 ISO 9.0
http://srv52-bud.infra.mirantis.net/fuelweb-iso/fuel-9.0-mos-460-2016-06-09_06-18-59.iso

Reproduced on 455 ISO 9.0.
http://srv52-bud.infra.mirantis.net/fuelweb-iso/fuel-9.0-mos-455-2016-06-08_06-18-59.iso

Actual result:
[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+--------+------------------+---------+-------------+-------------------+------------+---------------+--------+---------
 5 | ready | Untitled (3c:8e) | 1 | 10.109.25.7 | 64:a6:52:6a:3c:8e | compute | | 1 | 1
 2 | ready | Untitled (ac:4f) | 1 | 10.109.25.6 | 64:e6:3b:7f:ac:4f | controller | | 1 | 1
 4 | ready | Untitled (29:04) | 1 | 10.109.25.5 | 64:db:32:56:29:04 | compute | | 1 | 1
 3 | ready | Untitled (0d:f8) | 1 | 10.109.25.4 | 64:0f:3a:ef:0d:f8 | controller | | 1 | 1
 1 | ready | Untitled (dc:60) | 1 | 10.109.25.3 | 64:c7:38:30:dc:60 | controller | | 1 | 1
[root@nailgun ~]# ssh node-1
ssh: Could not resolve hostname node-1: Name or service not known

Reproduced on fuel-9.0-mos-467 iso.

[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------+---------+------------+-------------------+-----------------+---------------+--------+---------
 2 | ready | 2 | 1 | 10.109.0.4 | 64:fa:11:50:2f:3e | compute | | 1 | 1
 1 | ready | 1 | 1 | 10.109.0.3 | 64:bd:4a:b1:5d:6a | controller | | 1 | 1
 4 | ready | 4 | 2 | 10.109.0.6 | 64:fd:06:80:da:95 | controller | | 1 | 2
 5 | ready | 5 | 2 | 10.109.0.7 | 46:13:3d:ce:b2:32 | cinder, compute | | 1 | 2
 3 | ready | 3 | 2 | 10.109.0.5 | 0e:88:ae:eb:0f:5e | controller | | 1 | 2
 6 | discover | 6 | | 10.109.0.8 | 64:94:8d:81:fc:a4 | | | 1 |

[root@nailgun ~]# cat /etc/resolv.conf
search test.domain.local
domain test.domain.local
nameserver 8.8.8.8
nameserver 10.109.0.1

[root@nailgun ~]# ssh node-1
ssh: Could not resolve hostname node-1: Name or service not known
[root@nailgun ~]# ssh node-2
ssh: Could not resolve hostname node-2: Name or service not known
[root@nailgun ~]# ssh node-5
ssh: Could not resolve hostname node-5: Name or service not known
[root@nailgun ~]# ssh node-3
ssh: Could not resolve hostname node-3: Name or service not known
[root@nailgun ~]# ssh node-4
ssh: Could not resolve hostname node-4: Name or service not known

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id: 467
cat /etc/fuel_build_number: 467
cat /etc/fuel_release: 9.0
cat /etc/fuel_openstack_version: mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8454.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8454.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-openstack-metadata-9.0.0-1.mos8742.noarch
 fuel-notify-9.0.0-1.mos8454.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8454.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8742.noarch
 fuel-library9.0-9.0.0-1.mos8454.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos935.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8742.noarch

Move to confirmed according to previous comments

We should not be putting the external dns in resolv.conf and just updating dnsmasq to use it as the forwarder and leaving only the localhost as the nameserver. I'll check it out and see if this resolves it.

Fix proposed to branch: master
Review: https://review.openstack.org/328436

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/329360

Reviewed: https://review.openstack.org/328436
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7dc73f7cb8bf5f71b675ea571f6d4401017756e5
Submitter: Jenkins
Branch: master

commit 7dc73f7cb8bf5f71b675ea571f6d4401017756e5
Author: Alex Schultz <email address hidden>
Date: Fri Jun 10 11:54:29 2016 -0600

    Do not add upstream to resolv.conf on master

    In order to not break local name resolution for the fuel environment, we
    should rely on dnsmasq to query the upstream servers for us. Because of
    this, the resolv.conf on the master should only point to the cobbler
    dnsmasq instance which is configured with the upstream hosts to use if
    the local resolution fails.

    Change-Id: I757d89173d98dda419d21ef6132d59f186a97782
    Closes-Bug: #1567957

The priority changed to Critical for 9.0 because:
1. It is a regression issue
2. It affects all customers
3. we have fix on review which already merged to master
4. The user impact is the following:
a) Users can't work with fuel slave nodes using hostnames like 'ssh node-1', but we have a lot of documentation / screencasts where we use these hostnames
b) scripts for previous versions of fuel should be rewritten because of this regression issue
c) Some OSTF tests fail, example of related issue: https://bugs.launchpad.net/fuel/+bug/1592876

Reviewed: https://review.openstack.org/329360
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0254f124112340a972067985778241c40c52cbdd
Submitter: Jenkins
Branch: stable/mitaka

commit 0254f124112340a972067985778241c40c52cbdd
Author: Alex Schultz <email address hidden>
Date: Fri Jun 10 11:54:29 2016 -0600

    Do not add upstream to resolv.conf on master

    In order to not break local name resolution for the fuel environment, we
    should rely on dnsmasq to query the upstream servers for us. Because of
    this, the resolv.conf on the master should only point to the cobbler
    dnsmasq instance which is configured with the upstream hosts to use if
    the local resolution fails.

    Change-Id: I757d89173d98dda419d21ef6132d59f186a97782
    Closes-Bug: #1567957

Verified on 9.1 snapshot #56

[root@nailgun ~]# cat /etc/resolv.conf
search test.domain.local
domain test.domain.local
nameserver 10.109.21.2
[root@nailgun ~]# ping node-1
PING node-1.test.domain.local (10.109.21.3) 56(84) bytes of data.
64 bytes from node-1.test.domain.local (10.109.21.3): icmp_seq=1 ttl=64 time=0.647 ms
64 bytes from node-1.test.domain.local (10.109.21.3): icmp_seq=2 ttl=64 time=0.669 ms
64 bytes from node-1.test.domain.local (10.109.21.3): icmp_seq=3 ttl=64 time=0.530 ms
^C
--- node-1.test.domain.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.530/0.615/0.669/0.064 ms
[root@nailgun ~]# ping node-2
PING node-2.test.domain.local (10.109.21.4) 56(84) bytes of data.
64 bytes from node-2.test.domain.local (10.109.21.4): icmp_seq=1 ttl=64 time=0.735 ms
64 bytes from node-2.test.domain.local (10.109.21.4): icmp_seq=2 ttl=64 time=0.708 ms
64 bytes from node-2.test.domain.local (10.109.21.4): icmp_seq=3 ttl=64 time=0.960 ms
^C
--- node-2.test.domain.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.708/0.801/0.960/0.112 ms

[root@nailgun ~]# ssh node-1
Warning: Permanently added 'node-1' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-86-generic x86_64)

 * Documentation: https://help.ubuntu.com/
Last login: Wed Jul 27 11:52:31 2016 from 10.109.21.2
root@node-1:~#

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0