Not able to launch new volume-based instance in different availability_zone

@krogon-intel, could you please provide output of 'cinder-manage service list' ?

# cinder-manage service list 2>/dev/null
Binary Host Zone Status State Updated At
cinder-backup rbd:volumes nova enabled :-) 2015-08-06 15:16:07
cinder-volume rbd:volumes nova enabled :-) 2015-08-06 15:16:07
cinder-scheduler rbd:volumes nova enabled :-) 2015-08-06 15:16:07
# cinder availability-zone-list
+------+-----------+
| Name | Status |
+------+-----------+
| nova | available |
+------+-----------+

I not sure cinder is wrong here, because nova is sending that POST. Do you thing cinder improperly handles that?

@krogon-intel, please, provide output of 'cinder availability-zone-list' and 'nova availability-zone-list' too

krogon-intel, you tries to create a volume in 'aaa' AZ. Cinder doesn't have such one. It can't create a volume with such AZ. Are you sure that it worked in a previous release?

Just like mentioned in description
1) this is something I do in Horizon
2) this was working in Fuel 6.0
3) I know there is no AZ like "aaa" in cinder, but it tries to use AZ from nova
4) I didn't found easy way to add new AZ in cinder: each controller have the same service name (rbd:volumes) so I cannot change storage_availability_zone in /etc/cinder/cinder.conf (race condition of last updated controller)

To confirm my word I think this is the commit that changed the behavior: https://github.com/openstack/nova/commit/d23ddc475c9e51ac3f8d0d1574aeb0707c9fd82c

Now let's see it in action:
[node from MOS 6.0]# grep "volume_api.create(" /usr/lib/python2.6/site-packages/nova/virt/block_device.py -A1
            vol = volume_api.create(context, self.volume_size,
                                    '', '', snapshot)
--
            vol = volume_api.create(context, self.volume_size,
                                    '', '', image_id=self.image_id)
--
            vol = volume_api.create(context, self.volume_size, vol_name, '')
            if wait_func:
[node from MOS 6.1]# grep "volume_api.create(" /usr/lib/python2.6/site-packages/nova/virt/block_device.py -A1
            vol = volume_api.create(context, self.volume_size, '', '',
                                    snapshot, availability_zone=av_zone)
--
            vol = volume_api.create(context, self.volume_size,
                                    '', '', image_id=self.image_id,
--
            vol = volume_api.create(context, self.volume_size, vol_name, '',
                                    availability_zone=av_zone)

There was a bug in nova which is fixed by the commit: https://github.com/openstack/nova/commit/6060888b58db42eea826939852838f9e1c204d2c

@Ivan: I do not think you get it correctly. In previous comment I linked to the same commit (to be clearly merge of that commit). They call it fix, because it was impossible to spawn VM in different cinder AZ. However this introduces the problems in MOS architecture as there is only single cinder AZ and you cannot that change that easily.

Discussion started on mailing irc about proper behaviour: http://lists.openstack.org/pipermail/openstack-dev/2015-August/071732.html

@krogon-intel, according to the documentation [1] you should select available AZ in a new host aggregate. If you specify a new AZ, it won't work in nova too [2].

Before [3] was merged, nova didn't pass AZ to Cinder so it was able to use volume in any available AZ.

MOS architecture and Cinder extension is out of scope of this issue, so I would like to close this issue for MOS 7.0 and implement better solutions both in MOS and Cinder in the following releases.

[1] http://docs.openstack.org/havana/config-reference/content/host-aggregates.html
[2] http://paste.openstack.org/show/429718/
[3] https://github.com/openstack/nova/commit/6060888b58db42eea826939852838f9e1c204d2c

You still do not get the point.
You can have unlimited nova AZ, but single cinder AZ. How this may work? Whole AZ feature and host-aggregate is useless.

You are right that mentioned commit in nova introduce the problem. But you are the integrator! You have backported that commit from Kilo 2015.1.1 to Juno. It stopped working after upgrading from MOS 6.0 to 6.1.

Guys from cinder are working on blueprint to solve that (allow multiple AZ assignment to single cinder service or additional flag to ignore AZ from scheduler), but this will be ready for M-release. Till that time you should address this issue.

BTW. You are wrong with the example you showed. After adding some host to aggregate new AZ is created: http://paste.openstack.org/show/434630/

The upstream bug in Horizon:
https://bugs.launchpad.net/horizon/+bug/1485578

krogon-intel, I will backport patch from Liberty to MOS7. Thanks for helping me to understand why it is needed.

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Ivan Kolodyazhny <email address hidden>
Review: https://review.fuel-infra.org/11056

Reviewed: https://review.fuel-infra.org/11056
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: c5a863eaa47ad097361f01fa2cfbdf4540194a04
Author: Ivan Kolodyazhny <email address hidden>
Date: Wed Sep 2 10:57:58 2015

Adds allow_availability_zone_fallback option to Cinder

This change adds allow_availability_zone_fallback, which allows Cinder
to fall back to an already configured availability zone if the
requested availability zone is unavailable.

Change-Id: Id6b43fc3243b4de82241cfab34ff10953b537874
DocImpact: cinder
Closes-Bug: #1481818

on verification

verified on MOS 7.0 ISO 265
{"build_id": "265", "build_number": "265", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "265", "build_number": "265", "api": "1.0", "fuel-library_sha": "4fdf3d6b070204366593012428395d173698678a", "nailgun_sha": "0dfcf73deb8ae99654f3da2ea95b7b68b9ee7273", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "4fdf3d6b070204366593012428395d173698678a", "nailgun_sha": "0dfcf73deb8ae99654f3da2ea95b7b68b9ee7273", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"}

Any chance for 6.1 bugfix?

That was fixed in 7.0, I nominated that bug to 6.1-updates. The fix looks pretty simple, it's likely to be included into the next 6.1 maintenance update.

krogon-intel, please stay tuned.

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Ivan Kolodyazhny <email address hidden>
Review: https://review.fuel-infra.org/12786

Fix was merged with the latest stable/liberty code

Fix for this introduces new config option. But bug is customer-found. I'm not sure that we could merge it

Closing as Won't Fix for 6.1-updates as we don't introduce new features in maintenance updates. allow_availability_zone_fallback option is available in 7.0 and 8.0.

Change abandoned by Vitaly Sedelnik <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/12786
Reason: Abandoned as we don't accept new features to stable branch.

I've verified on MOS 8.0 (build 264) - bug is reproduced.
Env: HA, 3 controllers, 2 compute, neutron vlan, ceph for all
Details here http://paste.openstack.org/show/481890/

Alexander,

Fix is merged for to 8.0 branch [1]. Did you set allow_availability_zone_fallback=True in cinder.conf to make this patch work?

[1] https://review.fuel-infra.org/gitweb?p=openstack/cinder.git;a=commit;h=b85d2812a8256ff82934d150dbc4909e041d8b31

I've verified it again on MOS 8.0 (build 368) with allow_availability_zone_fallback=True in cinder.conf - all works fine!
ENV: Neutron VLAN, 3 controller with ceph for all, 2 compute
Details: http://pastebin.com/nDRsX3fq