| 2016-04-19 08:41:54 |
Ivan Kolodyazhny |
description |
Problem discovered on MOS 8.0 ISO #361
When I start a large amount of instances (in my case, 250), a big part of them failed with error like "Build of instance 070995e4-66f4-4acd-87c7-92815a489edd aborted: Block Device Mapping is Invalid."
Message: Build of instance 070995e4-66f4-4acd-87c7-92815a489edd aborted: Block Device Mapping is Invalid.
Code: 500
Details: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance filter_properties) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2025, in _build_and_run_instance 'create.error', fault=e) File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1996, in _build_and_run_instance block_device_mapping) as resources: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2143, in _build_resources reason=e.format_message())
Environment:
* 50 baremetal nodes
* Ceph *is not* used
- Cinder LVM over iSCSI for volumes is enabled, rest of checkboxes are disabled
* Ceilometer, Murano and Sahara *are* installed
* Each node has 32142MB RAM, 12 CPU
Steps to reproduce:
* Upload an image with Ubuntu Trusty
* Proceed to Horizon
* Proceed to 'Instances', click on 'Launch instance'
* Select m1.small flavour (1 vCPU, root disk - 20GB, RAM - 2048 MB)
* Set 'Boot from image (creates a new volume)'
* Choose uploaded image
* Set device size to 20 GB
* Ensure that 'Delete on Terminate' checkbox is active
* Set 250 instances
* Choose network
* Click on 'Launch'
Expected results:
* All instances are up and running
Real result:
* Only 81 instance started. 169 failed with error above
* Cinder has 235 created volumes available or in-use
Due to bug https://bugs.launchpad.net/bugs/1529182 I cannot create diagnostic snapshot, so I've gathered all logs after error happened:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkWk05Ykl2V1NZWE0/view?usp=sharing
Please find cinder and nova output attached. |
Problem discovered on MOS 8.0 ISO #361
When I start a large amount of instances (in my case, 250), a big part of them failed with error like "Build of instance 070995e4-66f4-4acd-87c7-92815a489edd aborted: Block Device Mapping is Invalid."
Message: Build of instance 070995e4-66f4-4acd-87c7-92815a489edd aborted: Block Device Mapping is Invalid.
Code: 500
Details: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance filter_properties) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2025, in _build_and_run_instance 'create.error', fault=e) File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1996, in _build_and_run_instance block_device_mapping) as resources: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2143, in _build_resources reason=e.format_message())
Environment:
* 50 baremetal nodes
* Ceph *is not* used
- Cinder LVM over iSCSI for volumes is enabled, rest of checkboxes are disabled
* Ceilometer, Murano and Sahara *are* installed
* Each node has 32142MB RAM, 12 CPU
Steps to reproduce:
* Upload an image with Ubuntu Trusty
* Proceed to Horizon
* Proceed to 'Instances', click on 'Launch instance'
* Select m1.small flavour (1 vCPU, root disk - 20GB, RAM - 2048 MB)
* Set 'Boot from image (creates a new volume)'
* Choose uploaded image
* Set device size to 20 GB
* Ensure that 'Delete on Terminate' checkbox is active
* Set 250 instances
* Choose network
* Click on 'Launch'
Expected results:
* All instances are up and running
Real result:
* Only 81 instance started. 169 failed with error above
* Cinder has 235 created volumes available or in-use
Due to bug https://bugs.launchpad.net/bugs/1529182 I cannot create diagnostic snapshot, so I've gathered all logs after error happened:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkWk05Ykl2V1NZWE0/view?usp=sharing
Please find cinder and nova output attached.
Related upstream bug: https://bugs.launchpad.net/cinder/+bug/1556902 |
|
