Juniper Openstack

attaching an interface to a VM on icehouse fails

Series r1.1
Bug #1367220

Bug #1367220 reported by Vedamurthy Joshi on 2014-09-09

This bug report is a duplicate of: Bug #1377419: On Icehouse, nova interface-attach cmd fails but port does get bound to the VM. Edit Remove

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R1.1	New	High	Atul Moghe	Juniper Openstack r1.20
Trunk	New	High	Atul Moghe

Bug Description

R1.10 34

Interface-attach fails on an openstack node running icehouse.

CFGM : [u'nodea35', u'nodea34']
Control Nodes : [u'nodea35', u'nodec53']
Compute Nodes : [u'nodec54', u'nodec55', u'nodec56']
Collector : [u'nodea35', u'nodec53']
Database : [u'nodea35', u'nodea34', u'nodec53']
WebUI : nodec53
OpenstackUI !
: nodea34

root@nodea35:/var/log/contrail# nova interface-attach --port-id 35da1b9d-648c-4cfe-9d7a-18c3d6dc920a vm1
ERROR: Failed to attach interface (HTTP 500) (Request-ID: req-02d2fc6a-7a96-4152-a18d-3c5ad885a93e)
root@nodea35:/var/log/contrail#

neutron/server.log:
===================

2014-09-09 15:51:15.744 ERROR [neutron.notifiers.nova] Failed to notify nova on events: [{'name': 'network-changed', 'server_uuid': u'79b59245-2a06-42be-8f38-fcd65cd96a1b'}]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 187, in send_events
    batched_events)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/contrib/server_external_events.py", line 39, in create
    return_raw=True)
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 152, in _create
    _resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 286, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
BadRequest: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400)

---------------
nova/nova-api.log:
=================

2014-09-09 15:51:13.155 24221 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-09-09 15:51:13.289 24221 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-09-09 15:51:13.373 24221 INFO nova.osapi_compute.wsgi.server [req-20a80205-02a8-42c7-a78c-2238bd5b27e5 78cbc891f04d44bd858deb9f47492602 75b706c78d6a4834b145caa5c7ea2192] 10.204.216.31 "GET /v1.1/75b706c78d6a4834b145caa5c7ea2192/servers HTTP/1.1" status: 200 len: 533 time: 0.2297630
2014-09-09 15:51:13.564 24221 INFO nova.osapi_compute.wsgi.server [req-4658f5ce-9f18-46f9-907e-2caea8f0dd57 78cbc891f04d44bd858deb9f47492602 75b706c78d6a4834b145caa5c7ea2192] 10.204.216.31 "GET /v1.1/75b706c78d6a4834b145caa5c7ea2192/servers/79b59245-2a06-42be-8f38-fcd65cd96a1b HTTP/1.1
" status: 200 len: 1884 time: 0.18732602014-09-09 15:51:13.604 24221 AUDIT nova.api.openstack.compute.contrib.attach_interfaces [req-02d2fc6a-7a96-4152-a18d-3c5ad885a93e 78cbc891f04d
44bd858deb9f47492602 75b706c78d6a4834b145caa5c7ea2192] [instance: 79b59245-2a06-42be-8f38-fcd65cd96a1b] Attach interface2014-09-09 15:51:13.614 24221 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.204.216.31:5672
2014-09-09 15:51:14.640 24221 ERROR nova.api.openstack.compute.contrib.attach_interfaces [req-02d2fc6a-7a96-4152-a18d-3c5ad885a93e 78cbc891f04d44bd858deb9f47492602 75b706c78d6a4834b145caa5c7ea2192] <nova.objects.instance.Instance object at 0x4556610>
Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
incoming.message))

File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
result = getattr(endpoint, method)(ctxt, **new_args)

  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 396, in decorated_function
    return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4347, in attach_interface
    self.driver.attach_interface(instance, image_meta, network_info[0])

File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1422, in attach_interface
raise exception.InterfaceAttachFailed(instance)

InterfaceAttachFailed: <nova.objects.instance.Instance object at 0x4556610>
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces Traceback (most recent call last):
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/attach_interfaces.py", line 106, in create
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces instance, network_id, port_id, req_ip)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 201, in wrapped
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces return func(self, context, target, *args, **kwargs)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2864, in attach_interface
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces requested_ip=requested_ip)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 322, in attach_interface
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces port_id=port_id, requested_ip=requested_ip)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces wait_for_reply=True, timeout=timeout)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces timeout=timeout)2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces raise result
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces InterfaceAttachFailed_Remote: <nova.objects.instance.Instance object at 0x4556610>
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces Traceback (most recent call last):
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces incoming.message))
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 396, in decorated_function
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces return function(self, context, *args, **kwargs)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4347, in attach_interface
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces self.driver.attach_interface(instance, image_meta, network_info[0])
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1422, in attach_interface
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces raise exception.InterfaceAttachFailed(instance)
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces InterfaceAttachFailed: <nova.objects.instance.Instance object at 0x4556610>
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.640 24221 TRACE nova.api.openstack.compute.contrib.attach_interfaces
2014-09-09 15:51:14.644 24221 INFO nova.api.openstack.wsgi [req-02d2fc6a-7a96-4152-a18d-3c5ad885a93e 78cbc891f04d44bd858deb9f47492602 75b706c78d6a4834b145caa5c7ea2192] HTTP exception thrown: Failed to attach interface
2014-09-09 15:51:14.645 24221 INFO nova.osapi_compute.wsgi.server [req-02d2fc6a-7a96-4152-a18d-3c5ad885a93e 78cbc891f04d44bd858deb9f47492602 75b706c78d6a4834b145caa5c7ea2192] 10.204.216.31 "POST /v1.1/75b706c78d6a4834b145caa5c7ea2192/servers/79b59245-2a06-42be-8f38-fcd65cd96a1b/os-interface HTTP/1.1" status: 500 len: 278 time: 1.0785990
2014-09-09 15:51:15.741 24263 ERROR nova.api.openstack.wsgi [-] Exception handling resource: multi() got an unexpected keyword argument 'body'
2014-09-09 15:51:15.741 24263 TRACE nova.api.openstack.wsgi Traceback (most recent call last):
2014-09-09 15:51:15.741 24263 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 983, in _process_stack
2014-09-09 15:51:15.741 24263 TRACE nova.api.openstack.wsgi action_result = self.dispatch(meth, request, action_args)
2014-09-09 15:51:15.741 24263 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1070, in dispatch
2014-09-09 15:51:15.741 24263 TRACE nova.api.openstack.wsgi return method(req=request, **action_args)
2014-09-09 15:51:15.741 24263 TRACE nova.api.openstack.wsgi TypeError: multi() got an unexpected keyword argument 'body'
2014-09-09 15:51:15.741 24263 TRACE nova.api.openstack.wsgi
2014-09-09 15:51:15.743 24263 INFO nova.osapi_compute.wsgi.server [-] 127.0.0.1 "POST /None/os-server-external-events HTTP/1.1" status: 400 len: 274 time: 0.0026641

Tags:

Revision history for this message

Hampapur Ajay (hajay) wrote on 2014-10-20:

Observed the following in /var/log/libvirtd.log when this occurs:
root@nodec3:~# tail -f /var/log/libvirt/libvirtd.log &
root@nodec3:/usr/lib/python2.7/dist-packages/nova# nova interface-attach --port-id c9dc8241-a9e3-4f48-9829-6176e20e0a63 vm1
2014-10-20 22:46:19.357+0000: 1679: error : qemuMonitorJSONCheckError:354 : internal error: unable to execute QEMU command 'getfd': No file descriptor supplied via SCM_RIGHTS
ERROR: Failed to attach interface (HTTP 500) (Request-ID: req-ed274411-2e79-4eca-9ee6-0e43ba197eff)