Activity log for bug #1556549

Date Who What changed Old value New value Message
2016-03-13 06:10:08 Rahul Sharma bug added bug
2016-03-13 06:10:28 Rahul Sharma summary too many qbr or qvo entries on compute node even though I have 2-3 instances on that compute node too many qbr or qvo entries on compute node even though I have 7-8 instances on that compute node
2016-03-13 06:10:46 Rahul Sharma description I am seeing this wierd behavior in our production environment. Right now, we are seeing an issue where launching of an instance is failing since the compute node and neutron is not cleaning up the qbr or qvo it had created even after we try to terminate the failed instance. Here are the logs from nova-conductor:- 2016-03-08 01:35:49.478 14041 ERROR nova.scheduler.utils [req-6ec7ee4b-9663-4f1b-910a-a87d99ac941c c665814ae07a4f71b666d04fcb99c2e9 a0288bedbb884e07bc0c602e7a343de8 - - -] [instance: fa9c27b4-06dd-4c04-9647-44e1fb8c1a81] Error from last host: compute-42 (node compute-42): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2254, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2400, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance fa9c27b4-06dd-4c04-9647-44e1fb8c1a81 was re-scheduled: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if-exists', 'del-port', u'qvo3e44fa11-05', '--', 'add-port', 'br-int', u'qvo3e44fa11-05', '--', 'set', 'Interface', u'qvo3e44fa11-05', u'external-ids:iface-id=3e44fa11-05b5-44dc-8c0c-6b937fe7abe0', 'external-ids:iface-status=active', u'external-ids:attached-mac=fa:16:3e:60:aa:5e', 'external-ids:vm-uuid=fa9c27b4-06dd-4c04-9647-44e1fb8c1a81']\n"] This qvo still exists on the compute node:- [root@compute-42 rahul]# ifconfig | grep qvo3e44fa11-05 qvo3e44fa11-05: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 <----- this still exists [root@compute-42 rahul]# ifconfig | grep qvo | wc -l 392 <------------------------ there are about 350+ such entries [root@compute-42 rahul]# ifconfig | grep tap | wc -l 8 <----------------------- the compute is running only 8 instances, still more than 350+ entries for qvo-XX alone [root@compute-42 rahul]# I am running Kilo release and RHEL 7 Openstack rpms. Expected:- Shouldn't the qvo and qvb be deleted if creation of instance has failed? I am seeing this weird behavior in our production environment. Right now, we are seeing an issue where launching of an instance is failing since the compute node and neutron is not cleaning up the qbr or qvo it had created even after we try to terminate the failed instance. Here are the logs from nova-conductor:- 2016-03-08 01:35:49.478 14041 ERROR nova.scheduler.utils [req-6ec7ee4b-9663-4f1b-910a-a87d99ac941c c665814ae07a4f71b666d04fcb99c2e9 a0288bedbb884e07bc0c602e7a343de8 - - -] [instance: fa9c27b4-06dd-4c04-9647-44e1fb8c1a81] Error from last host: compute-42 (node compute-42): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2254, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2400, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance fa9c27b4-06dd-4c04-9647-44e1fb8c1a81 was re-scheduled: Error during following call to agent: ['ovs-vsctl', '--timeout=120', '--', '--if-exists', 'del-port', u'qvo3e44fa11-05', '--', 'add-port', 'br-int', u'qvo3e44fa11-05', '--', 'set', 'Interface', u'qvo3e44fa11-05', u'external-ids:iface-id=3e44fa11-05b5-44dc-8c0c-6b937fe7abe0', 'external-ids:iface-status=active', u'external-ids:attached-mac=fa:16:3e:60:aa:5e', 'external-ids:vm-uuid=fa9c27b4-06dd-4c04-9647-44e1fb8c1a81']\n"] This qvo still exists on the compute node:- [root@compute-42 rahul]# ifconfig | grep qvo3e44fa11-05 qvo3e44fa11-05: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 <----- this still exists [root@compute-42 rahul]# ifconfig | grep qvo | wc -l 392 <------------------------ there are about 350+ such entries [root@compute-42 rahul]# ifconfig | grep tap | wc -l 8 <----------------------- the compute is running only 8 instances, still more than 350+ entries for qvo-XX alone [root@compute-42 rahul]# I am running Kilo release and RHEL 7 Openstack rpms. Expected:- Shouldn't the qvo and qvb be deleted if creation of instance has failed?
2016-03-13 13:39:19 Davanum Srinivas (DIMS) bug task added neutron
2016-03-13 20:40:31 Ihar Hrachyshka tags ovs
2016-03-13 20:40:41 Ihar Hrachyshka neutron: importance Undecided Medium
2016-03-13 20:41:41 Ihar Hrachyshka tags ovs needs-attention ovs
2016-03-13 20:42:14 Ihar Hrachyshka neutron: status New Incomplete
2016-03-16 16:52:47 Matt Riedemann tags needs-attention ovs libvirt needs-attention ovs
2016-03-16 16:56:05 Matt Riedemann nova: status New Incomplete
2016-03-16 17:52:51 Rahul Sharma attachment added compute-42-logs.txt https://bugs.launchpad.net/neutron/+bug/1556549/+attachment/4601303/+files/compute-42-logs.txt
2016-03-16 18:12:35 Rahul Sharma attachment added compute-openvswitch-agent-logs.txt https://bugs.launchpad.net/neutron/+bug/1556549/+attachment/4601323/+files/compute-openvswitch-agent-logs.txt
2016-03-17 17:42:41 Rahul Sharma attachment added neutron-server-logs.txt https://bugs.launchpad.net/neutron/+bug/1556549/+attachment/4602415/+files/neutron-server-logs.txt
2016-03-21 09:20:59 Markus Zoeller (markus_z) nova: status Incomplete New
2016-03-21 19:48:38 Matt Riedemann neutron: status Incomplete New
2016-03-21 19:53:22 Matt Riedemann bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=1158701
2016-04-18 18:03:24 Sean Dague nova: status New Confirmed
2016-04-18 18:03:29 Sean Dague neutron: status New Confirmed
2016-04-18 18:24:19 Ilya Chukhnakov neutron: status Confirmed Invalid
2016-04-24 23:13:55 Markus Zoeller (markus_z) nova: importance Undecided Medium
2016-05-06 05:55:16 Rahul Sharma nova: assignee Rahul Sharma (rahulsharmaait)
2016-06-07 08:31:32 Takashi Natsume nova: status Confirmed In Progress
2017-06-27 19:23:18 Sean Dague nova: status In Progress Confirmed
2017-06-27 19:23:24 Sean Dague nova: assignee Rahul Sharma (rahulsharmaait)