So today it got reproduced locally in an OVB deployment, some of the info i gathered:- Issue happened on:- controller-2 on interface:- eth2(ip: 172.17.0.25) stdout: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Error, some other host (A6:20:45:A7:BD:56) already uses address 172.17.0.25." (undercloud) [zuul@undercloud ~]$ openstack server list -c Name -c Networks +-------------------------+------------------------+ | Name | Networks | +-------------------------+------------------------+ | overcloud-controller-0 | ctlplane=192.168.24.10 | | overcloud-controller-2 | ctlplane=192.168.24.14 | | overcloud-novacompute-0 | ctlplane=192.168.24.9 | | overcloud-controller-1 | ctlplane=192.168.24.12 | +-------------------------+------------------------+ Running arping command on controller-2 manually returns the same mac address:- [heat-admin@overcloud-controller-2 ~]$ arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [A6:20:45:A7:BD:56] 0.708ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) Now the interesting part, when arping command is ran from other overcloud nodes or undercloud for same ip, returned mac address is different:- (undercloud) [zuul@undercloud ~]$ ssh heat-admin@192.168.24.14 sudo arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [A6:20:45:A7:BD:56] 0.681ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) (undercloud) [zuul@undercloud ~]$ ssh heat-admin@192.168.24.12 sudo arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [B6:29:F7:A1:64:97] 0.623ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) (undercloud) [zuul@undercloud ~]$ ssh heat-admin@192.168.24.10 sudo arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [0E:D0:C7:58:65:07] 0.721ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) (undercloud) [zuul@undercloud ~]$ ssh heat-admin@192.168.24.9 sudo arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [EE:79:BB:F1:55:37] 0.662ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) (undercloud) [zuul@undercloud ~]$ arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [3E:87:D5:0A:EE:22] 3.453ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) tcpdump output on undercloud:- (undercloud) [zuul@undercloud ~]$ arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [3E:87:D5:0A:EE:22] 4.002ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) (undercloud) [zuul@undercloud ~]$ ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── (undercloud) [zuul@undercloud ~]$ sudo tcpdump -lni any arp|grep 172.17.0.25 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 07:07:58.808008 ARP, Request who-has 172.17.0.25 (Broadcast) tell 0.0.0.0, length 28 07:07:58.811465 ARP, Reply 172.17.0.25 is-at 3e:87:d5:0a:ee:22, length 28 07:07:58.811486 ARP, Reply 172.17.0.25 is-at ea:cd:38:37:77:2b, length 28 07:07:58.811491 ARP, Reply 172.17.0.25 is-at 26:d8:e0:69:61:77, length 28 07:07:58.811495 ARP, Reply 172.17.0.25 is-at 8e:e5:25:84:04:1d, length 28 tcpdump on controller-2:- [heat-admin@overcloud-controller-2 ~]$ arping -c 2 -w 3 -D -I eth2 172.17.0.25 ARPING 172.17.0.25 from 0.0.0.0 eth2 Unicast reply from 172.17.0.25 [A6:20:45:A7:BD:56] 0.689ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── [heat-admin@overcloud-controller-2 ~]$ sudo tcpdump -lni any arp|grep 172.17.0.25 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 07:12:03.767705 ARP, Request who-has 172.17.0.25 (Broadcast) tell 0.0.0.0, length 28 07:12:03.767832 ARP, Reply 172.17.0.25 is-at a6:20:45:a7:bd:56, length 28 07:12:03.772584 ARP, Reply 172.17.0.25 is-at ee:79:bb:f1:55:37, length 28 07:12:03.772602 ARP, Reply 172.17.0.25 is-at b6:29:f7:a1:64:97, length 28 07:12:03.772608 ARP, Reply 172.17.0.25 is-at 0e:d0:c7:58:65:07, length 28 For correct ip's which are working on other overcloud nodes, tcpdump have single Reply, like:- 07:16:26.327020 ARP, Request who-has 172.17.0.11 (Broadcast) tell 0.0.0.0, length 28 07:16:26.331936 ARP, Reply 172.17.0.11 is-at fa:16:3e:29:d3:7f, length 28 So there is "something" which returns this bogus ARP response. what we can do to find that "something"? can it be some ovs/linux bridge on Host cloud or undercloud? i have a reproducer up currently, if someone from networking can jump on and find the culprit. Some environment info(issue happened on internal network, so adding those info only):- (undercloud) [zuul@undercloud ~]$ openstack network show internal_api +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | DOWN | | availability_zone_hints | | | availability_zones | | | created_at | 2019-09-24T20:50:51Z | | description | | | dns_domain | None | | id | 1c3116bf-5bdc-4321-8b48-fb2f30b1d380 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1350 | | name | internal_api | | port_security_enabled | True | | project_id | ef55a1f6d80b45398c94a1bfde8f0b79 | | provider:network_type | flat | | provider:physical_network | internal_api | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 3 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | d18c6065-52ad-47fb-90b3-02cab9600ee1 | | tags | | | updated_at | 2019-09-24T20:50:55Z | +---------------------------+--------------------------------------+ (undercloud) [zuul@undercloud ~]$ openstack subnet show internal_api_subnet +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 172.17.0.10-172.17.0.250 | | cidr | 172.17.0.0/24 | | created_at | 2019-09-24T20:50:55Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | None | | host_routes | | | id | d18c6065-52ad-47fb-90b3-02cab9600ee1 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | internal_api_subnet | | network_id | 1c3116bf-5bdc-4321-8b48-fb2f30b1d380 | | project_id | ef55a1f6d80b45398c94a1bfde8f0b79 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2019-09-24T20:50:55Z | +-------------------+--------------------------------------+ (undercloud) [zuul@undercloud ~]$ openstack port list|grep 172.17.0 | 24948f69-41a0-46a6-96f6-00105479cf7a | | fa:16:3e:8c:c2:3c | ip_address='172.17.0.19', subnet_id='d18c6065-52ad-47fb-90b3-02cab9600ee1' | DOWN | | 2b0b67d5-2d34-4bb0-bd6a-4250aea0ee10 | internal_api_virtual_ip | fa:16:3e:f1:c8:93 | ip_address='172.17.0.10', subnet_id='d18c6065-52ad-47fb-90b3-02cab9600ee1' | DOWN | | 7fd45d04-21f1-4fa5-8c38-dc5008117ec6 | | fa:16:3e:4b:83:f6 | ip_address='172.17.0.25', subnet_id='d18c6065-52ad-47fb-90b3-02cab9600ee1' | DOWN | | d76ce193-0b74-480f-b99f-0003a208ffca | redis_virtual_ip | fa:16:3e:ff:b1:9e | ip_address='172.17.0.17', subnet_id='d18c6065-52ad-47fb-90b3-02cab9600ee1' | DOWN | | edf20d56-1b0b-4630-b817-1abc32a52dad | | fa:16:3e:22:ce:d9 | ip_address='172.17.0.11', subnet_id='d18c6065-52ad-47fb-90b3-02cab9600ee1' | DOWN | | f1f67d86-72b9-43ba-b7c0-4ed8d64d611e | | fa:16:3e:99:9d:d9 | ip_address='172.17.0.22', subnet_id='d18c6065-52ad-47fb-90b3-02cab9600ee1' | DOWN | From host cloud:- (openstack_venv) [ykarel@yatinkarel tqwd]$ openstack network show overcloud_internal-1927 --max-width 60 +---------------------------+------------------------------+ | Field | Value | +---------------------------+------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2019-09-24T19:14:51Z | | description | | | dns_domain | None | | id | b9f4e5e8-f4d0-4234-98b8-3b8d | | | c660b3b1 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | location | Munch({'project': | | | Munch({'domain_id': None, | | | 'id': u'1510cd49292a424ba64c | | | 8bec08bf8e1a', 'name': | | | 'ykarel', 'domain_name': | | | None}), 'cloud': '', | | | 'region_name': 'regionOne', | | | 'zone': None}) | | mtu | 1450 | | name | overcloud_internal-1927 | | port_security_enabled | True | | project_id | 1510cd49292a424ba64c8bec08bf | | | 8e1a | | provider:network_type | None | | provider:physical_network | None | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 5 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | e7c9d546-1a25-4ebc-b49b- | | | cf51298e14f8 | | tags | | | updated_at | 2019-09-24T19:14:53Z | +---------------------------+------------------------------+ (openstack_venv) [ykarel@yatinkarel tqwd]$ openstack subnet show overcloud_internal-1927 --max-width 60 +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 172.17.0.1-172.17.0.254 | | cidr | 172.17.0.0/24 | | created_at | 2019-09-24T19:14:53Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | None | | host_routes | | | id | e7c9d546-1a25-4ebc-b49b-cf51298e14f8 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | location | Munch({'project': | | | Munch({'domain_id': None, 'id': | | | u'1510cd49292a424ba64c8bec08bf8e1a', | | | 'name': 'ykarel', 'domain_name': | | | None}), 'cloud': '', 'region_name': | | | 'regionOne', 'zone': None}) | | name | overcloud_internal-1927 | | network_id | b9f4e5e8-f4d0-4234-98b8-3b8dc660b3b1 | | prefix_length | None | | project_id | 1510cd49292a424ba64c8bec08bf8e1a | | revision_number | 2 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2019-09-24T19:14:53Z | +-------------------+--------------------------------------+ (openstack_venv) [ykarel@yatinkarel rdoinfo]$ openstack port list|grep 172.17.0 | 4ab10e9b-6600-4eff-a0b9-0bb5040aa330 | internal_baremetal-1927_2 | fa:16:3e:ad:b0:f1 | ip_address='172.17.0.7', subnet_id='e7c9d546-1a25-4ebc-b49b-cf51298e14f8' | ACTIVE | | 55045f17-7a45-436c-a0a8-0c0600a091bd | internal_baremetal-1927_1 | fa:16:3e:62:56:9b | ip_address='172.17.0.20', subnet_id='e7c9d546-1a25-4ebc-b49b-cf51298e14f8' | ACTIVE | | 77ba0a85-a056-42f5-8335-a63458505df9 | internal_baremetal-1927_3 | fa:16:3e:af:e2:1d | ip_address='172.17.0.10', subnet_id='e7c9d546-1a25-4ebc-b49b-cf51298e14f8' | ACTIVE | | 83ff3054-e119-479d-96e4-56b1d4d2dea9 | internal_baremetal-1927_0 | fa:16:3e:29:d3:7f | ip_address='172.17.0.1', subnet_id='e7c9d546-1a25-4ebc-b49b-cf51298e14f8' | ACTIVE | Let me know if i can gather some other info which would help in isolating the issue.