nova-scheduler doesn't account for create-new-volume disk space when using ceph
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
New
|
Undecided
|
ABHAY |
Bug Description
Description
===========
It seems that nova-scheduler may not account for disk space appropriately when creating a new instance using a new cinder volume. We have ceph backing cinder and glance, so in theory if we spin up a new instance (boot from image create new volume) that is backed by ceph, the scheduler should only take ceph disk space into account. However, it seems like it may take local disk space on compute nodes (we have this being used for ephemeral disks if not using cinder) when scheduling.
This causes an issue if we have limited local disk space but plenty of storage space in ceph, and we try to spin up a new instance fully backed by ceph but based on a flavor with root disk specification too large for local nodes (even though this gets overwritten when spinning up on new volume since you manually specify volume size). The instance fails to boot.
Steps to reproduce
==================
1. Environment uses ephemeral storage local to compute nodes, ceph backs cinder/glance.
2. Create a flavor that has root disk size > available ephemeral storage on compute nodes.
3. Launch instance from image (create new volume) so it's fully backed by ceph and it should not need the ephemeral storage on compute nodes, using the previously created flavor. Specify a disk size for new volume that is smaller than available ceph space but larger than ephemeral disk
4. Instance will fail to launch and drop errors pasted below.
Now,
1. Create another flavor with root disk size < available ephemeral storage on compute nodes
2. Launch instance again using same settings, so still create new volume and ensure volume is greater in size than ephemeral space.
3. Note instance launches and works no issue.
This shows that the ephemeral disk space specified on flavor has no real affect on ability to spin up the instance outside of initial scheduling, because that space isn't actually used when spinning up an instance where cinder/glance is backed by ceph. The only thing is it is taken into consideration during scheduling and it will fail to try and create the instance if there isn't enough ephemeral space.
Logs
=====
<180>Aug 9 15:55:37 controller01 nova-scheduler: 2016-08-09 15:55:37.261 155487 WARNING nova.scheduler.
<182>Aug 9 15:55:37 controller01 nova-scheduler: 2016-08-09 15:55:37.262 155487 INFO nova.filters [req-c10f09a5-
<182>Aug 9 15:55:37 controller01 nova-scheduler: 2016-08-09 15:55:37.263 155487 INFO nova.filters [req-c10f09a5-
<180>Aug 9 15:55:37 controller01 nova-conductor: 2016-08-09 15:55:37.266 155590 WARNING nova.scheduler.
Traceback (most recent call last):
File "/usr/lib/
return func(*args, **kwargs)
File "/usr/lib/
filter_
File "/usr/lib/
raise exception.
NoValidHost: No valid host was found. There are not enough hosts available.
<180>Aug 9 15:55:37 controller01 nova-conductor: 2016-08-09 15:55:37.267 155590 WARNING nova.scheduler.
Expected Result
===============
If ephemeral disk space will not be utilized or touched as part of instance launching, it should not be used as part of diskfilter / scheduling as it may lead to unnecessary errors.
Actual Result
=============
Cannot launch instance because scheduler fails find a valid host (even if enough disk space is available)
Environment
===========
Liberty based, MOS 8.0. It looks like Mirantis has some of their own packages for nova-scheduler, not sure if this issue exists upstream in liberty but imagine it does.
ii nova-api 2:12.0.
ii nova-cert 2:12.0.
ii nova-common 2:12.0.
ii nova-conductor 2:12.0.
ii nova-consoleauth 2:12.0.
ii nova-consoleproxy 2:12.0.
ii nova-objectstore 2:12.0.
ii nova-scheduler 2:12.0.
ii python-nova 2:12.0.
ii python-novaclient 2:2.30.
Libvert+KVM
Ceph 0.94.5 for cinder/glance. nova/ephemeral disk is using lvm I believe.
Neutron+openvswitch
Changed in nova: | |
assignee: | nobody → ABHAY (ankatare) |
Hi, /bugs.launchpad .net/nova/ +bug/1428176, see also /bugs.launchpad .net/nova/ +bug/1469179 and thread: lists.openstack .org/pipermail/ openstack- dev/2016- August/ 102308. html
it sounds like a duplicate of
https:/
https:/
http://
BR,
Konstantin