Sometimes OSTF test "Launch instance with file injection" fails on step #3 "Assign floating ip to instance":
fuel_health.test: DEBUG: Adding <Server: ost1_test-server-smoke-file_inj-1204451443> to shared resources of TestNovaNetwork fuel_health.common.test_mixins: INFO: STEP:3, verify action: 'floating IP creation' fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 177, in verify result = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 733, in _create_floating_ip pool=floating_ips_pool[0].name) File "/usr/lib/python2.7/site-packages/novaclient/v2/floating_ips.py", line 41, in create return self._create("/os-floating-ips", {'pool': pool}, "floating_ip") File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create _resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 449, in post return self._cs_request(url, 'POST', **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 424, in _cs_request resp, body = self._time_request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 397, in _time_request resp, body = self.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 391, in request raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.FloatingIpNotFoundForAddress'> (HTTP 500) (Request-ID: req-6f1ed80b-d0c0-4ccc-b0c4-b1d29c20d3e5)
Here is appropriate Nova logs:
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions [req-6f1ed80b-d0c0-4ccc-b0c4-b1d29c20d3e5 a5858f2179534119aeef4f844b3e97db e026785f12d94783bd5a75d9c820a1e6 - - -] Unexpected exception in API method
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/floating_ips.py", line 131, in create
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions ip = self.network_api.get_floating_ip_by_address(context, address)
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1393, in get_floating_ip_by_address
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions fip = self._get_floating_ip_by_address(client, address)
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1467, in _get_floating_ip_by_address
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions raise exception.FloatingIpNotFoundForAddress(address=address)
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions FloatingIpNotFoundForAddress: Floating ip not found for address 10.109.8.131.
2015-12-29 01:55:03.940 26843 ERROR nova.api.openstack.extensions
And Neutron logs:
2015-12-29 01:55:03.844 22363 DEBUG neutron.db.ipam_non_pluggable_backend [req-98c00f16-8c46-4a73-ae37-d2d8d4defacf a5858f2179534119aeef4f844b3e97db e026785f12d94783bd5a75d9c820a1e6 - - -] Allocated IP - 10.109.8.131 from 10.109.8.132 to 10.109.8.254 _try_generate_ip /usr/lib/python2.7/dist-packages/neutron/db/ipam_non_pluggable_backend.py:79
2015-12-29 01:55:03.845 22363 DEBUG neutron.db.db_base_plugin_common [req-98c00f16-8c46-4a73-ae37-d2d8d4defacf a5858f2179534119aeef4f844b3e97db e026785f12d94783bd5a75d9c820a1e6 - - -] Allocated IP 10.109.8.131 (e5301901-4822-4650-8cc7-0fb5183fcbda/21c9a552-4786-431c-8427-77d2f85d4832/252e3cdb-d316-44c3-8737-2c1b56cf3e32) _store_ip_allocation /usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_common.py:102
2015-12-29 01:55:03.935 13739 INFO neutron.wsgi [req-54353934-50b4-4c1c-90a8-84d3efe7fdd6 a5858f2179534119aeef4f844b3e97db e026785f12d94783bd5a75d9c820a1e6 - - -] 10.109.6.3 - - [29/Dec/2015 01:55:03] "GET /v2.0/floatingips.json?floating_ip_address=10.109.8.131 HTTP/1.1" 200 227 0.007953
Steps to reproduce:
1. Create cluster with neutron vlan
2. Add 3 node with controller role
3. Add 1 node with compute
4. Deploy cluster
5. Remove one controller add new controller
6. Deploy changes
7. Run OSTF
Expected result: 1 OSTF test is failed, because some nova services are down (from removed controller)
Actual result: 2 health checks fail, floating IP can't be assigned to instance in the "Launch instance with file injection" test
I've checked Nova logs and see the following (http:// paste.openstack .org/show/ 482869/):
1) Nova allocates a floating IP in Neutron - receives 201 and info about the newly created resource
2) right after that Nova tries to get detailed info on the created floating IP by issuing a GET request and receives an *empty* list of floating IPs, just like if the DB entry hasn't been created yet or has been filtered out for some reason