Fuel for OpenStack

Nova-compute logs filled with neutron api connection errors

Bug #1361294 reported by Vladimir Kuklin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Fuel Library (Deprecated)
Fuel for OpenStack 6.0
5.0.x
Invalid
High
Tatyanka
Fuel for OpenStack 5.0.2

Bug Description

Fuel 5.1 (master-84)
Openstack deployment Centos, Multi-node with HA, Neutron with vlan, three nodes for Ceph OSD, ceph for ephemeral, images, block and object.

On Horizon UI Live migrate to any other hypervisor node fails with error:
Error: There was an error submitting the form. Please try again.

Instance is left in a migrating state and must be terminated. If try to shutdown get error:
Error: Unable to shut off instance

2014-08-22 00:51:44.225 3600 TRACE nova.network.neutronv2.api [instance: eb73e919-dcf1-4d13-b460-e2501a9d482f]
2014-08-22T01:51:44.230116+01:00 err: 2014-08-22 00:51:44.227 3600 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager Traceback (most recent call last):
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1504, in _allocate_network_async
Bug initially forked from https://bugs.launchpad.net/fuel/+bug/1360032

2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager dhcp_options=dhcp_options)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 360, in allocate_for_instance
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager LOG.exception(msg, port_id)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 194, in _create_port
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager network_id, instance=instance)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 183, in _create_port
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager port_id = port_client.create_port(port_req_body)['port']['id']
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager ret = self.function(instance, *args, **kwargs)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 316, in create_port
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager return self.post(self.ports_path, body=body)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1241, in post
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager headers=headers, params=params)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1150, in do_request
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager self.httpclient.authenticate_and_fetch_endpoint_url()
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 179, in authenticate_and_fetch_endpoint_url
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager self.authenticate()
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 239, in authenticate
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager content_type="application/json")
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 154, in _cs_request
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager raise exceptions.ConnectionFailed(reason=e)
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager ConnectionFailed: Connection to neutron failed: timed out
2014-08-22 00:51:44.227 3600 TRACE nova.compute.manager

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

reproduced each time on CI http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.centos.known_issues/51/consoleFull

Also it affected 5.0.1 too

Changed in fuel:
importance: Undecided → High
milestone: none → 5.1
status: New → Confirmed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

for 5-0-2 assigned for me to reproduce issue after update and add logs

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Andrew Woodward (xarses) wrote :

high to match 5.1

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Tatyana, live migration bug is assigned to MOS team as there is bug in Icehouse https://bugs.launchpad.net/mos/+bug/1361228.

This particular bug severity is still unknown as we do not know if it breaks anything now.

Changed in fuel:
importance: High → Undecided
Vladimir Kuklin (vkuklin)
Changed in fuel:
milestone: 5.1 → 6.0
status: Confirmed → Incomplete
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
importance: Undecided → High
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Have simmilar errors here
http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.centos.bonding_ha/4/console

Isnatnce creation fail with http://paste.openstack.org/show/122647/

http://paste.openstack.org/show/122651/

at the same time befor instance creation request was recieved, reschedule pahhens and errors here looks like:

http://paste.openstack.org/show/122652/

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Matthew Mosesohn (raytrac3r)
Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Tatyana, can you reproduce this again?

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

try to reproduce on fuel-5.1.1-9-2014-11-14, so came back with result later

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

move to the invalid for 5.1.1 according to I can not reproduce it on 5.1.1 iso (test on instance live migration passed and tests on sbalance bond is incorerct, so communication failures can take place here until qa fix the test)

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

not reproducible for 6.0 also

Changed in fuel:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.