Juju won't add nova-compute to the correct AZ, even with customize-failure-domain: true
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Nova Compute Charm |
Triaged
|
High
|
Unassigned |
Bug Description
As in the title I have 3 zone setup in MAAS, Juju is deploying all units correctly, CEPH has correct CRASH map spread across the zones, but nova-compute hosts always end up in NOVA default AZ.
I even tried with pre-created AZ in OpenStack so names do match JUJU but still, the new host always ends up in NOVA AZ. I've confirmed Juju has JUJU_AVAILABILI
juju run --unit nova-compute/0 'env | grep JUJU_AVAILABILI
Model Controller Cloud/Region Version SLA Timestamp
openstack-ha maas-controller lab-maas/default 2.8.0 unsupported 10:10:10Z
App Version Status Scale Charm Store Rev OS Notes
ceph-mon 15.2.3 active 3 ceph-mon jujucharms 48 ubuntu
ceph-osd 15.2.3 active 3 ceph-osd jujucharms 303 ubuntu
ceph-radosgw 15.2.3 active 3 ceph-radosgw jujucharms 289 ubuntu
cinder 16.1.0 active 3 cinder jujucharms 303 ubuntu
cinder-ceph 16.1.0 active 3 cinder-ceph jujucharms 256 ubuntu
cinder-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
dashboard-
glance 20.0.0 active 3 glance jujucharms 297 ubuntu
glance-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
hacluster-
hacluster-cinder active 3 hacluster jujucharms 68 ubuntu
hacluster-glance active 3 hacluster jujucharms 68 ubuntu
hacluster-horizon active 3 hacluster jujucharms 68 ubuntu
hacluster-keystone active 3 hacluster jujucharms 68 ubuntu
hacluster-neutron active 3 hacluster jujucharms 68 ubuntu
hacluster-nova active 3 hacluster jujucharms 68 ubuntu
hacluster-placement active 3 hacluster jujucharms 68 ubuntu
keystone 17.0.0 active 3 keystone jujucharms 316 ubuntu
keystone-
memcached active 3 memcached jujucharms 29 ubuntu
mysql-innodb-
neutron-api 16.0.0 active 3 neutron-api jujucharms 286 ubuntu
neutron-
neutron-
nova-cloud-
nova-compute 21.0.0 active 2 nova-compute jujucharms 318 ubuntu
nova-mysql-router 8.0.20 active 3 mysql-router jujucharms 2 ubuntu
openstack-dashboard 18.3.2 active 3 openstack-dashboard jujucharms 304 ubuntu
ovn-central 20.03.0 active 3 ovn-central jujucharms 0 ubuntu
ovn-chassis 20.03.0 active 2 ovn-chassis jujucharms 2 ubuntu
placement 3.0.0 active 3 placement jujucharms 11 ubuntu
placement-
rabbitmq-server 3.8.2 active 3 rabbitmq-server jujucharms 103 ubuntu
vault 1.1.1 active 1 vault jujucharms 39 ubuntu
vault-mysql-router 8.0.20 active 1 mysql-router jujucharms 2 ubuntu
Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 0/lxd/0 10.10.70.115 Unit is ready and clustered
ceph-mon/1 active idle 1/lxd/0 10.10.70.113 Unit is ready and clustered
ceph-mon/2* active idle 2/lxd/0 10.10.70.112 Unit is ready and clustered
ceph-osd/0 active idle 0 10.10.70.101 Unit is ready (1 OSD)
ceph-osd/1 active idle 1 10.10.70.102 Unit is ready (1 OSD)
ceph-osd/2* active idle 2 10.10.70.103 Unit is ready (1 OSD)
ceph-radosgw/0 active idle 3/lxd/0 10.10.20.129 80/tcp Unit is ready
hacluster-
ceph-radosgw/1 active idle 4/lxd/0 10.10.20.131 80/tcp Unit is ready
hacluster-
ceph-radosgw/2* active idle 5/lxd/0 10.10.20.112 80/tcp Unit is ready
hacluster-
cinder/0 active idle 3/lxd/1 10.10.20.125 8776/tcp Unit is ready
cinder-ceph/1 active idle 10.10.20.125 Unit is ready
cinder-
hacluster-
cinder/1 active idle 4/lxd/1 10.10.20.128 8776/tcp Unit is ready
cinder-ceph/2 active idle 10.10.20.128 Unit is ready
cinder-
hacluster-
cinder/2* active idle 5/lxd/1 10.10.20.107 8776/tcp Unit is ready
cinder-ceph/0* active idle 10.10.20.107 Unit is ready
cinder-
hacluster-
glance/0 active idle 3/lxd/2 10.10.20.127 9292/tcp Unit is ready
glance-
hacluster-
glance/1 active idle 4/lxd/2 10.10.20.138 9292/tcp Unit is ready
glance-
hacluster-
glance/2* active idle 5/lxd/2 10.10.20.110 9292/tcp Unit is ready
glance-
hacluster-
keystone/0 active idle 3/lxd/3 10.10.20.136 5000/tcp Unit is ready
hacluster-
keystone-
keystone/1 active idle 4/lxd/3 10.10.20.124 5000/tcp Unit is ready
hacluster-
keystone-
keystone/2* active idle 5/lxd/3 10.10.20.116 5000/tcp Unit is ready
hacluster-
keystone-
memcached/0* active idle 3/lxd/4 10.10.20.117 11211/tcp Unit is ready and clustered
memcached/1 active idle 4/lxd/4 10.10.20.133 11211/tcp Unit is ready and clustered
memcached/2 active idle 5/lxd/4 10.10.20.114 11211/tcp Unit is ready and clustered
mysql-innodb-
mysql-innodb-
mysql-innodb-
neutron-api/3 active idle 3/lxd/15 10.10.20.120 9696/tcp Unit is ready
hacluster-
neutron-
neutron-
neutron-api/4 active idle 4/lxd/14 10.10.20.113 9696/tcp Unit is ready
hacluster-
neutron-
neutron-
neutron-api/5* active idle 5/lxd/14 10.10.20.132 9696/tcp Unit is ready
hacluster-
neutron-
neutron-
nova-cloud-
hacluster-nova/1 active idle 10.10.20.123 Unit is ready and clustered
nova-
nova-cloud-
hacluster-nova/2 active idle 10.10.20.118 Unit is ready and clustered
nova-
nova-cloud-
hacluster-nova/0* active idle 10.10.20.108 Unit is ready and clustered
nova-
nova-compute/0* active idle 6 10.10.20.103 Unit is ready
ovn-chassis/3* active idle 10.10.20.103 Unit is ready
nova-compute/2 active idle 8 10.10.20.101 Unit is ready
ovn-chassis/5 active idle 10.10.20.101 Unit is ready
openstack-
dashboard-
hacluster-
openstack-
dashboard-
hacluster-
openstack-
dashboard-
hacluster-
ovn-central/3 active idle 3/lxd/13 10.10.20.141 6641/tcp,6642/tcp Unit is ready (northd: active)
ovn-central/4 active idle 4/lxd/12 10.10.20.142 6641/tcp,6642/tcp Unit is ready
ovn-central/5* active idle 5/lxd/12 10.10.20.140 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db)
placement/0 active idle 3/lxd/10 10.10.20.139 8778/tcp Unit is ready
hacluster-
placement-
placement/1 active idle 4/lxd/10 10.10.20.126 8778/tcp Unit is ready
hacluster-
placement-
placement/2* active idle 5/lxd/10 10.10.20.109 8778/tcp Unit is ready
hacluster-
placement-
rabbitmq-server/0* active idle 3/lxd/11 10.10.20.122 5672/tcp,15672/tcp Unit is ready and clustered
rabbitmq-server/1 active idle 4/lxd/11 10.10.20.119 5672/tcp,15672/tcp Unit is ready and clustered
rabbitmq-server/2 active idle 5/lxd/11 10.10.20.106 5672/tcp,15672/tcp Unit is ready and clustered
vault/0* active idle 3/lxd/12 10.10.20.130 8200/tcp Unit is ready (active: true, mlock: disabled)
vault-
Machine State DNS Inst id Series AZ Message
0 started 10.10.70.101 lab-02 focal az-01 Deployed
0/lxd/0 started 10.10.70.115 juju-b7ff43-0-lxd-0 focal az-01 Container started
1 started 10.10.70.102 lab-05 focal az-02 Deployed
1/lxd/0 started 10.10.70.113 juju-b7ff43-1-lxd-0 focal az-02 Container started
2 started 10.10.70.103 lab-06 focal az-03 Deployed
2/lxd/0 started 10.10.70.112 juju-b7ff43-2-lxd-0 focal az-03 Container started
3 started 10.10.20.105 lab-03 focal az-01 Deployed
3/lxd/0 started 10.10.20.129 juju-b7ff43-3-lxd-0 focal az-01 Container started
3/lxd/1 started 10.10.20.125 juju-b7ff43-3-lxd-1 focal az-01 Container started
3/lxd/2 started 10.10.20.127 juju-b7ff43-3-lxd-2 focal az-01 Container started
3/lxd/3 started 10.10.20.136 juju-b7ff43-3-lxd-3 focal az-01 Container started
3/lxd/4 started 10.10.20.117 juju-b7ff43-3-lxd-4 focal az-01 Container started
3/lxd/5 started 10.10.20.137 juju-b7ff43-3-lxd-5 focal az-01 Container started
3/lxd/7 started 10.10.20.123 juju-b7ff43-3-lxd-7 focal az-01 Container started
3/lxd/10 started 10.10.20.139 juju-b7ff43-
3/lxd/11 started 10.10.20.122 juju-b7ff43-
3/lxd/12 started 10.10.20.130 juju-b7ff43-
3/lxd/13 started 10.10.20.141 juju-b7ff43-
3/lxd/14 started 10.10.20.134 juju-b7ff43-
3/lxd/15 started 10.10.20.120 juju-b7ff43-
4 started 10.10.20.102 lab-04 focal az-02 Deployed
4/lxd/0 started 10.10.20.131 juju-b7ff43-4-lxd-0 focal az-02 Container started
4/lxd/1 started 10.10.20.128 juju-b7ff43-4-lxd-1 focal az-02 Container started
4/lxd/2 started 10.10.20.138 juju-b7ff43-4-lxd-2 focal az-02 Container started
4/lxd/3 started 10.10.20.124 juju-b7ff43-4-lxd-3 focal az-02 Container started
4/lxd/4 started 10.10.20.133 juju-b7ff43-4-lxd-4 focal az-02 Container started
4/lxd/5 started 10.10.20.121 juju-b7ff43-4-lxd-5 focal az-02 Container started
4/lxd/7 started 10.10.20.118 juju-b7ff43-4-lxd-7 focal az-02 Container started
4/lxd/10 started 10.10.20.126 juju-b7ff43-
4/lxd/11 started 10.10.20.119 juju-b7ff43-
4/lxd/12 started 10.10.20.142 juju-b7ff43-
4/lxd/13 started 10.10.20.135 juju-b7ff43-
4/lxd/14 started 10.10.20.113 juju-b7ff43-
5 started 10.10.20.104 lab-07 focal az-03 Deployed
5/lxd/0 started 10.10.20.112 juju-b7ff43-5-lxd-0 focal az-03 Container started
5/lxd/1 started 10.10.20.107 juju-b7ff43-5-lxd-1 focal az-03 Container started
5/lxd/2 started 10.10.20.110 juju-b7ff43-5-lxd-2 focal az-03 Container started
5/lxd/3 started 10.10.20.116 juju-b7ff43-5-lxd-3 focal az-03 Container started
5/lxd/4 started 10.10.20.114 juju-b7ff43-5-lxd-4 focal az-03 Container started
5/lxd/5 started 10.10.20.111 juju-b7ff43-5-lxd-5 focal az-03 Container started
5/lxd/7 started 10.10.20.108 juju-b7ff43-5-lxd-7 focal az-03 Container started
5/lxd/10 started 10.10.20.109 juju-b7ff43-
5/lxd/11 started 10.10.20.106 juju-b7ff43-
5/lxd/12 started 10.10.20.140 juju-b7ff43-
5/lxd/13 started 10.10.20.115 juju-b7ff43-
5/lxd/14 started 10.10.20.132 juju-b7ff43-
6 started 10.10.20.103 lab-01 focal az-01 Deployed
8 started 10.10.20.101 lab-08 focal az-02 Deployed
I was able to reproduce it although:
* JUJU_AVAILABILI TY_ZONE is set correctly; availability_ zone config in the DEFAULT section of nova.conf.
* the charm correctly renders the default_
So it does what it is supposed to do but the intended result is not achieved. We only check this functionality via a unit test that's why we haven't caught it in functional testing.
Based on the Nova code, unless the host aggregate overrides the AZ value, it is taken from nova.conf's default_ availability_ zone: /opendev. org/openstack/ nova/src/ commit/ e91540f96538c7b a568617db7c03cf 3a195b87ff/ nova/availabili ty_zones. py#L93- L100 /opendev. org/openstack/ nova/src/ commit/ e91540f96538c7b a568617db7c03cf 3a195b87ff/ nova/availabili ty_zones. py#L71- L109 /opendev. org/openstack/ nova/src/ commit/ e91540f96538c7b a568617db7c03cf 3a195b87ff/ nova/conf/ availability_ zone.py# L32-L34
(stein)
https:/
https:/
https:/
juju config keystone openstack-origin
cloud:bionic-stein
openstack compute service list ------- ------- ----+-- ------- ------- ------+ ------- ---+--- ------+ ------- +------ ------- ------- ------- -+ ------- ------- ----+-- ------- ------- ------+ ------- ---+--- ------+ ------- +------ ------- ------- ------- -+ 27-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 55.000000 | 27-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 55.000000 | 27-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 57.000000 | 24T10:50: 58.000000 | 24T10:50: 59.000000 | 24T10:50: 55.000000 | 24T10:50: 53.000000 | 24T10:50: 56.000000 | 24T10:50: 54.000000 | 24T10:50: 59.000000 | 24-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 56.000000 | 24-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 56.000000 | 24-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 57.000000 | 31-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 57.000000 | 31-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 56.000000 | 31-lxd- 6 | internal | enabled | up | 2020-07- 24T10:50: 57.000000 | ------- ------- ----+-- ------- ------- ------+ ------- ---+--- ------+ ------- +------ ------- ------- ------- -+
+-----+
| ID | Binary | Host | Zone | Status | State | Updated At |
+-----+
| 1 | nova-scheduler | juju-554d51-
| 7 | nova-conductor | juju-554d51-
| 19 | nova-consoleauth | juju-554d51-
| 22 | nova-compute | server-maas-6.maas | nova | enabled | up | 2020-07-
| 25 | nova-compute | server-maas-11.maas | nova | enabled | up | 2020-07-
| 28 | nova-compute | server-maas-2.maas | nova | enabled | up | 2020-07-
| 31 | nova-compute | server-maas-8.maas | nova | enabled | up | 2020-07-
| 64 | nova-compute | server-maas-3.maas | nova | enabled | up | 2020-07-
| 67 | nova-compute | server-maas-10.maas | nova | enabled | up | 2020-07-
| 70 | nova-compute | server-maas-4.maas | nova | enabled | up | 2020-07-
| 73 | nova-consoleauth | juju-554d51-
| 76 | nova-conductor | juju-554d51-
| 82 | nova-scheduler | juju-554d51-
| 91 | nova-scheduler | juju-554d51-
| 100 | nova-consoleauth | juju-554d51-
| 103 | nova-conductor | juju-554d51-
+-----+
openstack aggregate list ; echo $?
0
juju config nova-compute-kvm ...