Nova VM does not reach ACTIVE state in CentOS binary gate

Nova VM does not reach ACTIVE state in CentOS binary gate.

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/console.html#_2016-04-11_08_57_15_935

Looks like a nova / neutron issue.

nova-api.log:
==========

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/logs/kolla/nova/nova-api.log

Errors in the above nova-api.log:

2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions [req-ec671e79-2c9c-4d47-8544-45664916a4ba e1bc63258580423891d7141b17f4bf31 4a2971b47f5c446d83095126dfda5089 - - -] Unexpected exception in API method
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 630, in create
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions **create_kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1556, in create
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1139, in _create_instance
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions reservation_id, max_count)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions requested_networks, max_count)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 448, in _check_requested_networks
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions max_count)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapped
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return func(self, context, *args, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 399, in validate_networks
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions requested_networks)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 212, in validate_networks
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return self.client.call(ctxt, 'validate_networks', networks=networks)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 413, in call
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions return self.prepare().call(ctxt, method, **kwargs)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions retry=self.retry)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions retry=retry)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 0a95d829d05c492c9a11c7854e83c72c
2016-04-11 08:57:13.550 27 ERROR nova.api.openstack.extensions
2016-04-11 08:57:13.588 27 INFO nova.api.openstack.wsgi [req-ec671e79-2c9c-4d47-8544-45664916a4ba e1bc63258580423891d7141b17f4bf31 4a2971b47f5c446d83095126dfda5089 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2016-04-11 08:57:13.594 27 INFO nova.osapi_compute.wsgi.server [req-ec671e79-2c9c-4d47-8544-45664916a4ba e1bc63258580423891d7141b17f4bf31 4a2971b47f5c446d83095126dfda5089 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 483, in handle_one_response
    write(b''.join(towrite))
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write
    _writelines(towrite)
  File "/usr/lib64/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib64/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

nova-compute.log:
===============

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/logs/kolla/nova/nova-compute.log

Errors in the above nova-compute.log:

2016-04-11 08:56:00.824 1 ERROR nova.compute.manager [req-015ab636-e753-479d-a699-4809608beb8d - - - - -] No compute node record for host node1

neutron-dhcp-agent.log:
====================

http://logs.openstack.org/13/245013/6/check/gate-kolla-dsvm-deploy-centos-binary/497b18e/logs/kolla/neutron/

Errors in the above neutron-dhcp-agent.log:

2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent [req-df0efeb6-9915-4d73-b10e-a0b657e0f3c1 - - - - -] Failed reporting state!
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 581, in _report_state
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent ctx, self.agent_state, True)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 86, in report_state
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent return method(context, 'report_state', **kwargs)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent retry=self.retry)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent retry=retry)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id)
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID 7dfbc0addf3b4cbc8aa78ac4569422a9
2016-04-11 08:55:58.613 1 ERROR neutron.agent.dhcp.agent
2016-04-11 08:55:58.616 1 WARNING oslo.service.loopingcall [req-df0efeb6-9915-4d73-b10e-a0b657e0f3c1 - - - - -] Function 'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted interval by 30.09 sec
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent [-] Unable to sync network state.
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 154, in sync_state
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent active_networks = self.plugin_rpc.get_active_networks_info()
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 438, in get_active_networks_info
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent host=self.host)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent retry=self.retry)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent retry=retry)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent result = self._waiter.wait(msg_id, timeout)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent message = self.waiters.get(msg_id, timeout=timeout)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent 'to message ID %s' % msg_id)
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent MessagingTimeout: Timed out waiting for a reply to message ID 14a548c2b7104cac82922aef04acd141
2016-04-11 08:55:58.618 1 ERROR neutron.agent.dhcp.agent