Was doing some more investigating and found this in the nova-all.log. This looks to like an issue like the device (0f:00.0) is busy, however it shouldn't be as the only one in use *should* be 10:00.0.
All devices seem to be claimed by pci-stub which from my understanding indicates that they can't be claimed by the current running OS.
<179>Sep 27 18:53:48 node-13 nova-conductor: 2016-09-27 18:53:48.631 24595 ERROR nova.scheduler.utils [req-dfd5dfe7-ea36-4ce0-8fe7-2412df59db20 11a8bdff50d34c64b2a9fc2b477af74b 81d1532551c2436793417cd7ef0abf35 - - -] [instance: e5fadc3b-6fab-4524-9a35-c8ac954014bd] Error from last host: cirrascale1 (node cirrascale1): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1926, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2116, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance e5fadc3b-6fab-4524-9a35-c8ac954014bd was re-scheduled: internal error: process exited while connecting to monitor: 2016-09-27T18:53:46.506916Z qemu-system-x86_64: -device vfio-pci,host=0f:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: Error: Failed to setup INTx fd: Device or resource busy\n2016-09-27T18:53:46.507929Z qemu-system-x86_64: -device vfio-pci,host=0f:00.0,id=hostdev0,bus=pci.0,addr=0x5: Device initialization failed\n2016-09-27T18:53:46.507952Z qemu-system-x86_64: -device vfio-pci,host=0f:00.0,id=hostdev0,bus=pci.0,addr=0x5: Device 'vfio-pci' could not be initialized\n\n"]
Was doing some more investigating and found this in the nova-all.log. This looks to like an issue like the device (0f:00.0) is busy, however it shouldn't be as the only one in use *should* be 10:00.0.
All devices seem to be claimed by pci-stub which from my understanding indicates that they can't be claimed by the current running OS.
<179>Sep 27 18:53:48 node-13 nova-conductor: 2016-09-27 18:53:48.631 24595 ERROR nova.scheduler. utils [req-dfd5dfe7- ea36-4ce0- 8fe7-2412df59db 20 11a8bdff50d34c6 4b2a9fc2b477af7 4b 81d1532551c2436 793417cd7ef0abf 35 - - -] [instance: e5fadc3b- 6fab-4524- 9a35-c8ac954014 bd] Error from last host: cirrascale1 (node cirrascale1): [u'Traceback (most recent call last):\n', u' File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 1926, in _do_build_ and_run_ instance\ n filter_ properties) \n', u' File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 2116, in _build_ and_run_ instance\ n instance_ uuid=instance. uuid, reason= six.text_ type(e) )\n', u"RescheduledEx ception: Build of instance e5fadc3b- 6fab-4524- 9a35-c8ac954014 bd was re-scheduled: internal error: process exited while connecting to monitor: 2016-09- 27T18:53: 46.506916Z qemu-system-x86_64: -device vfio-pci, host=0f: 00.0,id= hostdev0, bus=pci. 0,addr= 0x5: vfio: Error: Failed to setup INTx fd: Device or resource busy\n2016- 09-27T18: 53:46.507929Z qemu-system-x86_64: -device vfio-pci, host=0f: 00.0,id= hostdev0, bus=pci. 0,addr= 0x5: Device initialization failed\ n2016-09- 27T18:53: 46.507952Z qemu-system-x86_64: -device vfio-pci, host=0f: 00.0,id= hostdev0, bus=pci. 0,addr= 0x5: Device 'vfio-pci' could not be initialized\n\n"]