CI: rh1 compute nodes not spawning vms correctly
Bug #1649252 reported by
Sagi (Sergey) Shnaidman
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Unassigned |
Bug Description
Currently all OVB jobs fail in CI, mostly because of slow creating instances by nova.
Changed in tripleo: | |
status: | New → In Progress |
importance: | Undecided → Critical |
milestone: | none → ocata-2 |
To post a comment you must log in.
I've found a few concerning things in the logs on the compute nodes (which appear to be the cause of these failures). I'm looking at compute 15.
nova-compute shows this error output for a failed instance:
2016-12-12 16:49:12.119 3417 ERROR nova.network. linux_net [req-8ce28da4- 1b90-4dfa- 970b-f4b52b7ffc 5c ba119eef29ce49f 5b8697f4d63948e 3c b79291658f384b7 ebbc9019b6349e5 c9 - - -] Unable to execute ['ovs-vsctl', '--timeout=120', '--', '--if-exists', 'del-port', u'qvo439addb1-14', '--', 'add-port', 'br-int', u'qvo439addb1-14', '--', 'set', 'Interface', u'qvo439addb1-14', u'external- ids:iface- id=439addb1- 14a6-47dc- 89ce-332ede23cd fa', 'external- ids:iface- status= active' , u'external- ids:attached- mac=fa: 16:3e:44: ea:df', 'external- ids:vm- uuid=44acb3af- ecad-448a- ad91-eb54302b7e 8e']. Exception: Unexpected error while running command. rootwrap. conf ovs-vsctl --timeout=120 -- --if-exists del-port qvo439addb1-14 -- add-port br-int qvo439addb1-14 -- set Interface qvo439addb1-14 external- ids:iface- id=439addb1- 14a6-47dc- 89ce-332ede23cd fa external- ids:iface- status= active external- ids:attached- mac=fa: 16:3e:44: ea:df external- ids:vm- uuid=44acb3af- ecad-448a- ad91-eb54302b7e 8e 12-12T16: 49:12Z| 00002|fatal_ signal| WARN|terminatin g with signal 14 (Alarm clock)\n' manager [req-8ce28da4- 1b90-4dfa- 970b-f4b52b7ffc 5c ba119eef29ce49f 5b8697f4d63948e 3c b79291658f384b7 ebbc9019b6349e5 c9 - - -] [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] Instance failed to spawn manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] Traceback (most recent call last): manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] File "/usr/lib/ python2. 7/site- packages/ nova/compute/ manager. py", line 2218, in _build_resources manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] yield resources manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] File "/usr/lib/ python2. 7/site- packages/ nova/compute/ manager. py", line 2064, in _build_ and_run_ instance manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] block_device_ info=block_ device_ info) manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] File "/usr/lib/ python2. 7/site- packages/ nova/virt/ libvirt/ driver. py", line 2780, in spawn manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] LOG.debug("Instance is running", instance=instance) manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] File "/usr/lib/ python2. 7/site- packages/ nova/virt/ libvirt/ driver. py", line 4920, in _create_ domain_ and_network manager [instance: 44acb3af- ecad-448a- ad91-eb54302b7e 8e] with self._lxc_ disk_handler( instance, instance. image_meta, manage. ..
Command: sudo nova-rootwrap /etc/nova/
Exit code: 142
Stdout: u''
Stderr: u'2016-
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.
2016-12-12 16:49:12.120 3417 ERROR nova.compute.