Associate an already associated Floating IP to server is returning 202

Bug #1029911 reported by Rajalakshmi Ganesan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Eoghan Glynn

Bug Description

Association of a floating IP to server where floating IP is already associated to another server is returning 202 code and is being SUCCESSFUL, leading to associating same floating IP to two instances.

Steps to reproduce:

1. Create two instances 'a' and 'b' (wait till they are ACTIVE)
2. Create a floating IP
3. Associate floating IP to instance 'a'
4. Associate the same floating IP to instance 'b'

Expected result:
The error message should state that "This acction is not allowed as floating IP is already associted to 'a'." and should return 400 as it is a BadRequest.

Actual Result:
The association is being successful and returning 202 code.

LOG:
------

stack@ubuntu:~/devstack$ nova list
+--------------------------------------+--------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+------------------+
| 898c3ce0-5e06-4bbc-bb51-e165279aaa86 | test-2 | ACTIVE | private=10.0.0.3 |
| aba9b437-2302-4e01-9dba-0b1ae2aa7732 | test-1 | ACTIVE | private=10.0.0.2 |
+--------------------------------------+--------+--------+------------------+
stack@ubuntu:~/devstack$ nova floating-ip-list
+--------------+-------------+----------+------+
| Ip | Instance Id | Fixed Ip | Pool |
+--------------+-------------+----------+------+
| 172.24.4.225 | None | None | nova |
+--------------+-------------+----------+------+
stack@ubuntu:~/devstack$ nova add-floating-ip aba9b437-2302-4e01-9dba-0b1ae2aa7732 172.24.4.225
stack@ubuntu:~/devstack$ nova list
+--------------------------------------+--------+--------+--------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+--------------------------------+
| 898c3ce0-5e06-4bbc-bb51-e165279aaa86 | test-2 | ACTIVE | private=10.0.0.3 |
| aba9b437-2302-4e01-9dba-0b1ae2aa7732 | test-1 | ACTIVE | private=10.0.0.2, 172.24.4.225 |
+--------------------------------------+--------+--------+--------------------------------+

stack@ubuntu:~/devstack$ nova --debug add-floating-ip 898c3ce0-5e06-4bbc-bb51-e165279aaa86 172.24.4.225
connect: (10.233.52.237, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 10.233.52.237:5000\r\nContent-Length: 100\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "demo", "password": "muralik"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Vary: X-Auth-Token
header: Content-Type: application/json
header: Content-Length: 2158
header: Date: Fri, 27 Jul 2012 17:56:26 GMT

REQ: curl -i http://10.233.52.237:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient"

REQ BODY: {"auth": {"tenantName": "demo", "passwordCredentials": {"username": "demo", "password": "muralik"}}}

RESP:{'date': 'Fri, 27 Jul 2012 17:56:26 GMT', 'vary': 'X-Auth-Token', 'content-length': '2158', 'status': '200', 'content-type': 'application/json'} {"access": {"token": {"expires": "2012-07-28T17:56:26Z", "id": "058a0727ade643cc9c4480cf5e779cc3", "tenant": {"enabled": true, "id": "b84f99644c8b4fd494e21a57f71cc5b7", "name": "demo", "description": null}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.233.52.237:8774/v2/b84f99644c8b4fd494e21a57f71cc5b7", "region": "RegionOne", "publicURL": "http://10.233.52.237:8774/v2/b84f99644c8b4fd494e21a57f71cc5b7", "internalURL": "http://10.233.52.237:8774/v2/b84f99644c8b4fd494e21a57f71cc5b7"}], "endpoints_links": [], "type": "compute", "name": "Compute Service"}, {"endpoints": [{"adminURL": "http://10.233.52.237:3333", "region": "RegionOne", "publicURL": "http://10.233.52.237:3333", "internalURL": "http://10.233.52.237:3333"}], "endpoints_links": [], "type": "s3", "name": "S3 Service"}, {"endpoints": [{"adminURL": "http://10.233.52.237:9292", "region": "RegionOne", "publicURL": "http://10.233.52.237:9292", "internalURL": "http://10.233.52.237:9292"}], "endpoints_links": [], "type": "image", "name": "Image Service"}, {"endpoints": [{"adminURL": "http://10.233.52.237:8776/v1/b84f99644c8b4fd494e21a57f71cc5b7", "region": "RegionOne", "publicURL": "http://10.233.52.237:8776/v1/b84f99644c8b4fd494e21a57f71cc5b7", "internalURL": "http://10.233.52.237:8776/v1/b84f99644c8b4fd494e21a57f71cc5b7"}], "endpoints_links": [], "type": "volume", "name": "Volume Service"}, {"endpoints": [{"adminURL": "http://10.233.52.237:8773/services/Admin", "region": "RegionOne", "publicURL": "http://10.233.52.237:8773/services/Cloud", "internalURL": "http://10.233.52.237:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "EC2 Service"}, {"endpoints": [{"adminURL": "http://10.233.52.237:35357/v2.0", "region": "RegionOne", "publicURL": "http://10.233.52.237:5000/v2.0", "internalURL": "http://10.233.52.237:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "Identity Service"}], "user": {"username": "demo", "roles_links": [], "id": "43f9006aa4fd4068a2006631cb50b85a", "roles": [{"id": "4149bd78fa964f14bfb66896607efd6e", "name": "anotherrole"}, {"id": "e3ec12284694451c95c49a3a61864211", "name": "Member"}], "name": "demo"}}}

connect: (10.233.52.237, 8774)
send: u'GET /v2/b84f99644c8b4fd494e21a57f71cc5b7/servers/898c3ce0-5e06-4bbc-bb51-e165279aaa86 HTTP/1.1\r\nHost: 10.233.52.237:8774\r\nx-auth-project-id: demo\r\nx-auth-token: 058a0727ade643cc9c4480cf5e779cc3\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Compute-Request-Id: req-75f9c26c-429e-4766-a38e-47099773d778
header: Content-Type: application/json
header: Content-Length: 1245
header: Date: Fri, 27 Jul 2012 17:56:26 GMT

REQ: curl -i http://10.233.52.237:8774/v2/b84f99644c8b4fd494e21a57f71cc5b7/servers/898c3ce0-5e06-4bbc-bb51-e165279aaa86 -X GET -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: 058a0727ade643cc9c4480cf5e779cc3"

RESP:{'status': '200', 'content-length': '1245', 'content-location': u'http://10.233.52.237:8774/v2/b84f99644c8b4fd494e21a57f71cc5b7/servers/898c3ce0-5e06-4bbc-bb51-e165279aaa86', 'x-compute-request-id': 'req-75f9c26c-429e-4766-a38e-47099773d778', 'date': 'Fri, 27 Jul 2012 17:56:26 GMT', 'content-type': 'application/json'} {"server": {"OS-EXT-STS:task_state": null, "addresses": {"private": [{"version": 4, "addr": "10.0.0.3"}]}, "links": [{"href": "http://10.233.52.237:8774/v2/b84f99644c8b4fd494e21a57f71cc5b7/servers/898c3ce0-5e06-4bbc-bb51-e165279aaa86", "rel": "self"}, {"href": "http://10.233.52.237:8774/b84f99644c8b4fd494e21a57f71cc5b7/servers/898c3ce0-5e06-4bbc-bb51-e165279aaa86", "rel": "bookmark"}], "image": {"id": "d506424f-d702-4a9c-b2f4-47d82f490671", "links": [{"href": "http://10.233.52.237:8774/b84f99644c8b4fd494e21a57f71cc5b7/images/d506424f-d702-4a9c-b2f4-47d82f490671", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": "1", "links": [{"href": "http://10.233.52.237:8774/b84f99644c8b4fd494e21a57f71cc5b7/flavors/1", "rel": "bookmark"}]}, "id": "898c3ce0-5e06-4bbc-bb51-e165279aaa86", "user_id": "43f9006aa4fd4068a2006631cb50b85a", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "status": "ACTIVE", "updated": "2012-07-27T16:28:33Z", "hostId": "bbba5a83eb6349a3565861ed1731265bc9d992839c9754e40a585184", "key_name": null, "name": "test-2", "created": "2012-07-27T16:28:14Z", "tenant_id": "b84f99644c8b4fd494e21a57f71cc5b7", "metadata": {}}}

send: u'POST /v2/b84f99644c8b4fd494e21a57f71cc5b7/servers/898c3ce0-5e06-4bbc-bb51-e165279aaa86/action HTTP/1.1\r\nHost: 10.233.52.237:8774\r\nContent-Length: 46\r\nx-auth-project-id: demo\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nx-auth-token: 058a0727ade643cc9c4480cf5e779cc3\r\nuser-agent: python-novaclient\r\ncontent-type: application/json\r\n\r\n{"addFloatingIp": {"address": "172.24.4.225"}}'
reply: 'HTTP/1.1 202 Accepted\r\n'
header: Content-Type: text/html; charset=UTF-8
header: Content-Length: 0
header: Date: Fri, 27 Jul 2012 17:56:29 GMT

REQ: curl -i http://10.233.52.237:8774/v2/b84f99644c8b4fd494e21a57f71cc5b7/servers/898c3ce0-5e06-4bbc-bb51-e165279aaa86/action -X POST -H "X-Auth-Project-Id: demo" -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: 058a0727ade643cc9c4480cf5e779cc3"

REQ BODY: {"addFloatingIp": {"address": "172.24.4.225"}}

RESP:{'date': 'Fri, 27 Jul 2012 17:56:29 GMT', 'status': '202', 'content-length': '0', 'content-type': 'text/html; charset=UTF-8'}

stack@ubuntu:~/devstack$ nova list
+--------------------------------------+--------+--------+--------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+--------------------------------+
| 898c3ce0-5e06-4bbc-bb51-e165279aaa86 | test-2 | ACTIVE | private=10.0.0.3, 172.24.4.225 |
| aba9b437-2302-4e01-9dba-0b1ae2aa7732 | test-1 | ACTIVE | private=10.0.0.2, 172.24.4.225 |
+--------------------------------------+--------+--------+--------------------------------+
stack@ubuntu:~/devstack$

Eoghan Glynn (eglynn)
Changed in nova:
status: New → Confirmed
Eoghan Glynn (eglynn)
Changed in nova:
assignee: nobody → Eoghan Glynn (eglynn)
Revision history for this message
Eoghan Glynn (eglynn) wrote :

Looking at this in more detail, I think this may not be the bug that you think it is, but rather is partially reflective of the intended behavior.

Associating an already-associated floating IP with another instance is not treated as a failure. Rather it results in the floating IP being first disassociated from the original instance ('a' in your case) and then associated with the second instance ('b').

However what's confusing is that when this transfer of the floating IP to the second instance, the network info held in the DB instance info cache is only updated for the second instance. As a result the floating IP is reported as being associated with *both* instances, as in the example output of 'nova list' that you provide above.

Now the interesting thing is that this misreporting is temporary - eventually the periodic heal_instance_info_cache task will roll around and purge the inaccurante network info for the original instance, however depending on the frequency configured for this task and the size of the instance population, this may take a while.

I am going to propose a fix that ensures the obsolute network info is immediately purged when the reassignment occurs.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/10532

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/10532
Committed: http://github.com/openstack/nova/commit/6a5f095b0a9a73c57469b4e5f10694d1b738527f
Submitter: Jenkins
Branch: master

commit 6a5f095b0a9a73c57469b4e5f10694d1b738527f
Author: Eoghan Glynn <email address hidden>
Date: Fri Jul 27 20:34:05 2012 +0100

    Avoid associating floating IP with two instances.

    Addresses LP 1029911.

    Associating an already-associated floating IP with another instance
    results in the floating IP being first disassociated from the original
    instance and then associated with the second instance.

    However when this transfer of the floating IP to the second instance
    occurs, the network info held in the database instance info cache was
    previously only updated for the second instance (via the
    @nova.network.api.refresh_cache decorator).

    As a result, the floating IP was initially reported as being associated
    with *both* instances. This misreporting is temporary - eventually the
    periodic heal_instance_info_cache task will roll around and purge the
    inaccurante network info for the original instance. However depending
    on the frequency configured for this task and the size of the instance
    population, it could take a while for the refresh to occur.

    This issue is addressed by immediately purging the obsolute network info
    when the reassignment occurs.

    Change-Id: Ibfb04997ff4a983d144d81350857cb07475aff54

Changed in nova:
status: In Progress → Fix Committed
Eoghan Glynn (eglynn)
Changed in nova:
importance: Undecided → Medium
milestone: none → folsom-3
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
Joe Gordon (jogo) wrote :

This bug is not fixed:

From latest Devstack, running as admin user: http://paste.openstack.org/show/21071/

It looks like now the Floating IP will switch from one VM to the other

Revision history for this message
Joe Gordon (jogo) wrote :

This was changed Intentionally Changing the Tempest Test

Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-3 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.