Comment 0 for bug 1533197

Problem discovered on MOS 8.0 ISO #361

When I start a large amount of instances (in my case, 250), a big part of them failed with error like "Build of instance 070995e4-66f4-4acd-87c7-92815a489edd aborted: Block Device Mapping is Invalid."

Message: Build of instance 070995e4-66f4-4acd-87c7-92815a489edd aborted: Block Device Mapping is Invalid.
Code: 500
Details: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance filter_properties) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2025, in _build_and_run_instance 'create.error', fault=e) File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1996, in _build_and_run_instance block_device_mapping) as resources: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2143, in _build_resources reason=e.format_message())

Environment:
    * 50 baremetal nodes
    * Ceph *is not* used
        - Cinder LVM over iSCSI for volumes is enabled, rest of checkboxes are disabled
    * Ceilometer, Murano and Sahara *are* installed
    * Each node has 32142MB RAM, 12 CPU

Steps to reproduce:
    * Upload an image with Ubuntu Trusty
    * Proceed to Horizon
    * Proceed to 'Instances', click on 'Launch instance'
    * Select m1.small flavour (1 vCPU, root disk - 20GB, RAM - 2048 MB)
    * Set 'Boot from image (creates a new volume)'
    * Choose uploaded image
    * Set device size to 20 GB
    * Ensure that 'Delete on Terminate' checkbox is active
    * Set 250 instances
    * Choose network
    * Click on 'Launch'

Expected results:
    * All instances are up and running

Real result:
    * Only 81 instance started. 169 failed with error above
    * Cinder has 235 created volumes available or in-use

Due to bug https://bugs.launchpad.net/bugs/1529182 I cannot create diagnostic snapshot, so I've gathered all logs after error happened:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkWk05Ykl2V1NZWE0/view?usp=sharing

Please find cinder and nova output attached.