OpenStack Compute (nova)

FixedIpNotFoundForInstance: Instance 1 has zero fixed ips.

Bug #804317 reported by Dan Prince
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Wishlist
Dan Prince
OpenStack Compute (nova) 2011.3 "diablo"

Bug Description

Using nova trunk w/ FlatDHCP revision 1239:

I'm getting the following nova-network.log exceptions when trying to boot an instance:

(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 193, in allocate_for_instance
(nova): TRACE: ips = super(FloatingIP, self).allocate_for_instance(context, **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 372, in allocate_for_instance
(nova): TRACE: return self.get_instance_nw_info(context, instance_id, type_id)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 402, in get_instance_nw_info
(nova): TRACE: fixed_ips = self.db.fixed_ip_get_by_instance(context, instance_id)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/db/api.py", line 376, in fixed_ip_get_by_instance
(nova): TRACE: return IMPL.fixed_ip_get_by_instance(context, instance_id)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/db/sqlalchemy/api.py", line 117, in wrapper
(nova): TRACE: return f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/db/sqlalchemy/api.py", line 783, in fixed_ip_get_by_instance
(nova): TRACE: raise exception.FixedIpNotFoundForInstance(instance_id=instance_id)
(nova): TRACE: FixedIpNotFoundForInstance: Instance 1 has zero fixed ips.
(nova): TRACE:
2011-07-01 13:12:18,260 ERROR nova.rpc [-] Returning exception Instance 1 has zero fixed ips. to caller

---

I have created both fixed and floating IPs.

This exception is reproducible when installing nova from scratch and trying to boot an instance immediately after starting all the services.

If I wait a couple minutes after starting the various nova services (network, compute, etc.) instances seem to boot fine and I have no issues.

Is there a periodic task of some sort we need to run upon startup to obtain a set of fixed IPs for instances?

Related branches

lp://staging/~rackspace-titan/nova/network_create_with_host
Devin Carlen (community): Disapprove
Diff: 158 lines (+56/-9)
6 files modified
bin/nova-manage (+5/-3)
nova/db/sqlalchemy/api.py (+3/-1)
nova/db/sqlalchemy/migrate_repo/versions/035_network_add_configured.py (+40/-0)
nova/db/sqlalchemy/models.py (+1/-0)
nova/network/manager.py (+6/-5)
nova/tests/__init__.py (+1/-0)
Revision history for this message
Trey Morris (tr3buchet) wrote :

Yes. Originally the networks are not configured on the network hosts. You can solve this in two ways:
1) prior to booting the network hosts. Populate the host column in the networks table. Then the hosts will configure themselves as a part of their boot process.
2) wait for the network host period task to pick up the networks. They run about twice a minute. In order to scale linearly each host will only pick up and configure 1 network (so they take turns basically). So if you have 3 networks and 1 host. It'll take 1.5 minutes or so. If you have as many hosts and networks, you're all set once the first period task kicks.

Revision history for this message
Dan Prince (dan-prince) wrote :

I'd like to consider adding a nova-manage command to pre-set the network host for a network.

Something like:

nova-manage network preset-host network_id network_host_A

Changed in nova:
assignee: nobody → Dan Prince (dan-prince)
Thierry Carrez (ttx)
Changed in nova:
status: New → Incomplete
Revision history for this message
Thierry Carrez (ttx) wrote :

Following #2 comment

Changed in nova:
importance: Undecided → Wishlist
status: Incomplete → Confirmed
Revision history for this message
Trey Morris (tr3buchet) wrote :

Presetting the network host is a fine idea so long as it is only used prior to having nova running. If nova is already running we're basically setting up a race condition: someone running the preset-host command vs a network host picking up the network.

I can see solving the problem in a different way. First we can optionally pass in a host to the network create function. Then, if we were to add a column to network "configured" and have the network hosts refer to this column to determine if a network is configured. Network hosts could then:
1) get list of networks where "configured" == false
2) configure one of these networks where "host" == false. DB resulting in "host" being set to self.host and "configured" being set to true.
3) configure any of these networks in which "host" == self.host. DB resulting in "configured" being set to true.

This way any unconfigured networks are still picked up in a pseudo random fashion, but can optionally be preset to a specific host if desired. I think this option represents the best of both worlds.

Dan Prince (dan-prince)
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Dan Prince (dan-prince) wrote :

This was fixed when the 'multi_hosts' branch went in.

Dan Prince (dan-prince)
Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → diablo-3
Thierry Carrez (ttx)
Changed in nova:
milestone: diablo-3 → 2011.3
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.