Some instances can't connect to metadata due to ARP failure
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
High
|
Vish Ishaya |
Bug Description
Instances that have a local route (due to installing Network Manager in ubuntu for example) cannot contact the metadata server. This is because they send out an ARP (who-has) for 169.254.169.254 and never get a response. This issue also affects windows vms. The issue can be solved by giving the ip address to the host that is running nova-network, something along the lines of:
ip addr add 169.254.169.254/32 scope link dev eth1
This causes the network host to respond to the arp and the metadata request succeeds. Nova should automatically add this address to avoid this failure.
Example Error messages are below:
Instance fails to access metadata server at launch resulting it fails to complete initial booting process including sshd startup.
Logs from tty linux.
=======
Lease of 10.0.0.5 obtained, lease time 120^M
starting DHCP forEthernet interface eth0 [ ^[[1;32mOK^[[0;39m ]^M
cloud-setup: checking http://
cloud-setup: failed 1/30: up 7.97. iid had 1.0^M
cloud-setup: failed 2/30: up 9.18. iid had 1.0^M
cloud-setup: failed 3/30: up 10.35. iid had 1.0^M
cloud-setup: failed 4/30: up 11.52. iid had 1.0^M
cloud-setup: failed 5/30: up 12.70. iid had 1.0^M
cloud-setup: failed 6/30: up 13.88. iid had 1.0^M
cloud-setup: failed 7/30: up 15.06. iid had 1.0^M
cloud-setup: failed 8/30: up 16.24. iid had 1.0^M
cloud-setup: failed 9/30: up 17.43. iid had 1.0^M
cloud-setup: failed 10/30: up 18.62. iid had 1.0^M
cloud-setup: failed 11/30: up 19.81. iid had 1.0^M
cloud-setup: failed 12/30: up 21.00. iid had 1.0^M
cloud-setup: failed 13/30: up 22.20. iid had 1.0^M
cloud-setup: failed 14/30: up 23.40. iid had 1.0^M
cloud-setup: failed 15/30: up 24.60. iid had 1.0^M
cloud-setup: failed 16/30: up 25.80. iid had 1.0^M
cloud-setup: failed 17/30: up 27.01. iid had 1.0^M
cloud-setup: failed 18/30: up 28.22. iid had 1.0^M
cloud-setup: failed 19/30: up 29.43. iid had 1.0^M
cloud-setup: failed 20/30: up 30.65. iid had 1.0^M
cloud-setup: failed 21/30: up 31.86. iid had 1.0^M
cloud-setup: failed 22/30: up 33.08. iid had 1.0^M
cloud-setup: failed 23/30: up 34.30. iid had 1.0^M
cloud-setup: failed 24/30: up 35.60. iid had 1.0^M
cloud-setup: failed 25/30: up 36.89. iid had 1.0^M
cloud-setup: failed 26/30: up 38.11. iid had 1.0^M
cloud-setup: failed 27/30: up 39.34. iid had 1.0^M
cloud-setup: failed 28/30: up 40.56. iid had 1.0^M
cloud-setup: failed 29/30: up 41.82. iid had 1.0^M
cloud-setup: failed 30/30: up 43.05. iid had 1.0^M
cloud-setup: after 30 fails, debugging^M
cloud-setup: running debug (30 tries reached)^M
############ debug start ##############^M
### /etc/rc.
stty: /dev/console^M
generating DSS host key [^[[1;33mWATING
generating RSA host key [^[[1;33mWATING
startup dropbear [ ^[[1;32mOK^[[0;39m ]^M
### ifconfig -a^M
eth0 Link encap:Ethernet HWaddr 02:16:3E:57:D3:B5 ^M
=======
Logs from UEC image.
=======
init: plymouth-splash main process (263) terminated with status 2
init: plymouth main process (48) killed by SEGV signal
cloud-init running: Tue, 15 Feb 2011 09:55:54 +0000. up 30.11 seconds
consuming user data failed!
Traceback (most recent call last):
File "/usr/bin/
main()
File "/usr/bin/
cloud.
File "/usr/lib/
if self.sem_
File "/usr/lib/
semfile = self.sem_
File "/usr/lib/
freqtok = self.datasource
File "/usr/lib/
return(
KeyError: 'instance-id'
init: cloud-init main process (334) terminated with status 1
mountall: Event failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Disconnected from Plymouth
init: plymouth-log main process (364) terminated with status 1
* Starting AppArmor profiles [ OK ]
Traceback (most recent call last):
File "/usr/bin/
Traceback (most recent call last):
File "/usr/bin/
main()
main()
File "/usr/bin/
cc = cloudinit.
File "/usr/lib/
File "/usr/bin/
cc = cloudinit.
File "/usr/lib/
self.cfg = self.get_
File "/usr/lib/
f=file(cfgfile)
Traceback (most recent call last):
File "/usr/bin/
self.cfg = self.get_
File "/usr/lib/
f=file(cfgfile)
IOError: IOError: [Errno 2] No such file or directory: '/var/lib/
main()
File "/usr/bin/
cc = cloudinit.
File "/usr/lib/
self.cfg = self.get_
File "/usr/lib/
[Errno 2] No such file or directory: '/var/lib/
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/
Traceback (most recent call last):
File "/usr/bin/
main()
File "/usr/bin/
cc = cloudinit.
File "/usr/lib/
self.cfg = self.get_
File "/usr/lib/
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/
Traceback (most recent call last):
File "/usr/bin/
main()
File "/usr/bin/
cc = cloudinit.
File "/usr/lib/
self.cfg = self.get_
File "/usr/lib/
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/
landscape-client is not configured, please run landscape-config.
Traceback (most recent call last):
File "/usr/bin/
main()
File "/usr/bin/
cc = cloudinit.
File "/usr/lib/
self.cfg = self.get_
File "/usr/lib/
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/
Related branches
- Devin Carlen (community): Approve
- Soren Hansen (community): Approve
-
Diff: 54 lines (+11/-1)2 files modifiednova/network/linux_net.py (+10/-1)
nova/network/manager.py (+1/-0)
Changed in nova: | |
status: | Incomplete → Triaged |
importance: | Undecided → High |
summary: |
- Instance fails to access metadata server + Some instances can't connect to metadata due to ARP failure |
description: | updated |
Changed in nova: | |
assignee: | nobody → Vish Ishaya (vishvananda) |
milestone: | none → cactus-gamma |
status: | Triaged → In Progress |
Changed in nova: | |
status: | In Progress → Fix Committed |
Changed in nova: | |
milestone: | cactus-gamma → none |
Changed in nova: | |
milestone: | none → 2011.2 |
status: | Fix Committed → Fix Released |
What network mode are you using ? Modes outside VlanNamager require specific routing for metadata server to work.