Juju Charms Collection
nova-cloud-controller package

Adding a unit to an existing service overwrites the vip in the catalogue

Bug #1544959 reported by Leonardo Borda
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Ceilometer Charm
Fix Released
Critical
Liam Young
OpenStack Ceilometer Charm 17.11
OpenStack Cinder Charm
Fix Released
Critical
Liam Young
OpenStack Cinder Charm 17.11
OpenStack Glance Charm
Fix Released
Critical
Liam Young
OpenStack Glance Charm 17.11
OpenStack Neutron API Charm
Fix Released
Critical
Liam Young
OpenStack Neutron API Charm 17.11
OpenStack Nova Cloud Controller Charm
Fix Released
Critical
Liam Young
OpenStack Nova Cloud Controller Charm 17.11
nova-cloud-controller (Juju Charms Collection)
Invalid
Undecided
Unassigned

Bug Description

When adding and removing a new nova-cloud-controller we notice that juju registers that new controller as the new nova-api endpoint. When we remove that same unit it does not update the VIP.

$ juju-deployer -f keystone
2016-02-10 13:59:20 Service: keystone address: 10.5.0.219
10.5.0.219

$ keystone endpoint-list
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| 14eb9c3acb124c998056952e0a8a46d2 | RegionOne | http://10.5.0.224:8774/v2/$(tenant_id)s | http://10.5.0.224:8774/v2/$(tenant_id)s | http://10.5.0.224:8774/v2/$(tenant_id)s | 3f5fc02efa084a41b8bceb7f59599547 |
| 15e648ddbd1742738c92346a9d2c57f5 | RegionOne | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | 98a4e475fe114978aad641730199a95c |
| 33e17dad386e4dc2b9acc87057929bc4 | RegionOne | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:35357/v2.0 | 11e189355f734b5f9d7a755dfc4ebaa0 |
| 556ce143e6564fff8aef61fd6d51fe0b | RegionOne | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | 6a609c0d87d44f2b96dbb1413014d018 |
| 8c23368a7694465eb00e4c1042aa6144 | RegionOne | http://10.5.0.212:8777 | http://10.5.0.212:8777 | http://10.5.0.212:8777 | a896e1fde4cf4b7b9b08d74688743d39 |
| 9da5dd218ec546ae908b56bb6060c8b6 | RegionOne | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | 30b8c40ec3954b8f874cadb65a4153a5 |
| a76c5fbb2f384b58b4f45dabe8c3a744 | RegionOne | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080 | 06965a3aa2664622997c289e3890babe |
| b1f50dd93b5043c488cdc5bbba5d2863 | RegionOne | http://10.5.0.217:9292 | http://10.5.0.217:9292 | http://10.5.0.217:9292 | 77030f9ab1834ef284f25a8d3528d782 |
| b2ec91885f204c57ba1359366a7be34f | RegionOne | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | f6cc6ea9f8fc44acb6083a5541339f0a |
| fda0adfd56074a5a855612fc1c230d3b | RegionOne | http://10.5.0.222:9696 | http://10.5.0.222:9696 | http://10.5.0.222:9696 | 43436fb4474f4e6fb54da09c6c8531b0 |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+

$ juju add-unit nova-cloud-controller

$ !806
keystone endpoint-list
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| 15e648ddbd1742738c92346a9d2c57f5 | RegionOne | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | http://10.5.0.218:8000/v1 | 98a4e475fe114978aad641730199a95c |
| 33e17dad386e4dc2b9acc87057929bc4 | RegionOne | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:5000/v2.0 | http://10.5.0.219:35357/v2.0 | 11e189355f734b5f9d7a755dfc4ebaa0 |
| 556ce143e6564fff8aef61fd6d51fe0b | RegionOne | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | http://10.5.0.216:8776/v2/$(tenant_id)s | 6a609c0d87d44f2b96dbb1413014d018 |
| 8c23368a7694465eb00e4c1042aa6144 | RegionOne | http://10.5.0.212:8777 | http://10.5.0.212:8777 | http://10.5.0.212:8777 | a896e1fde4cf4b7b9b08d74688743d39 |
| 9da5dd218ec546ae908b56bb6060c8b6 | RegionOne | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | http://10.5.0.218:8004/v1/$(tenant_id)s | 30b8c40ec3954b8f874cadb65a4153a5 |
| a76c5fbb2f384b58b4f45dabe8c3a744 | RegionOne | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080/v1/AUTH_$(tenant_id)s | http://10.5.0.230:8080 | 06965a3aa2664622997c289e3890babe |
| b1f50dd93b5043c488cdc5bbba5d2863 | RegionOne | http://10.5.0.217:9292 | http://10.5.0.217:9292 | http://10.5.0.217:9292 | 77030f9ab1834ef284f25a8d3528d782 |
| b2ec91885f204c57ba1359366a7be34f | RegionOne | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | http://10.5.0.216:8776/v1/$(tenant_id)s | f6cc6ea9f8fc44acb6083a5541339f0a |
| d9516d98cf8447fea2ff8e3a4c673fdd | RegionOne | http://10.5.0.235:8774/v2/$(tenant_id)s | http://10.5.0.235:8774/v2/$(tenant_id)s | http://10.5.0.235:8774/v2/$(tenant_id)s | 3f5fc02efa084a41b8bceb7f59599547 |
| fda0adfd56074a5a855612fc1c230d3b | RegionOne | http://10.5.0.222:9696 | http://10.5.0.222:9696 | http://10.5.0.222:9696 | 43436fb4474f4e6fb54da09c6c8531b0 |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+

It's changed from http://10.5.0.224:8774 to http://10.5.0.235:8774

$juju remove-unit nova-cloud-controller/2

then we lose connection to the nova-api

$ nova list
ERROR: HTTPConnectionPool(host='10.5.0.235', port=8774): Max retries exceeded with url: /v2/705c2be789d847689909184d29615784/servers/detail (Caused by <class 'socket.error'>: [Errno 113] No route to host)

Revision history for this message
Leonardo Borda (lborda) wrote :

same thing with next-ha.yaml
http://bazaar.launchpad.net/~ost-maintainers/openstack-charm-testing/trunk/view/head:/bundles/dev/next-ha.yaml

Revision history for this message
Leonardo Borda (lborda) wrote :

workaround:
find out legacy_endpoint_id x keystone endpoint-list and change the database directly.

use keystone
select * from endpoint where legacy_endpoint_id='<id>'
update endpoint set url='http://<ip_nova-cloud-controller-vip>:8774/v2/$(tenant_id)s' where legacy_endpoint_id='<id>';

James Page (james-page)
Changed in nova-cloud-controller (Juju Charms Collection):
status: New → Invalid
Liam Young (gnuoy)
Changed in charm-nova-cloud-controller:
assignee: nobody → Liam Young (gnuoy)
importance: Undecided → Critical
status: New → Confirmed
summary: - juju remove-unit nova-cloud-controller/x does not update nova-api
- endpoint
+ Adding a unit to an existing nova-cloud-controller overwrites the vip in
+ the catalogue
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.openstack.org/508845

Changed in charm-nova-cloud-controller:
status: Confirmed → In Progress
Liam Young (gnuoy)
Changed in charm-nova-cloud-controller:
milestone: none → 17.11
Liam Young (gnuoy)
Changed in charm-neutron-api:
importance: Undecided → Critical
assignee: nobody → Liam Young (gnuoy)
milestone: none → 17.11
status: New → Confirmed
Liam Young (gnuoy)
Changed in charm-cinder:
status: New → In Progress
Changed in charm-neutron-api:
status: Confirmed → In Progress
Changed in charm-cinder:
importance: Undecided → Critical
assignee: nobody → Liam Young (gnuoy)
milestone: none → 17.11
Liam Young (gnuoy)
Changed in charm-glance:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Liam Young (gnuoy)
milestone: none → 17.11
Liam Young (gnuoy)
Changed in charm-ceilometer:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Liam Young (gnuoy)
milestone: none → 17.11
Liam Young (gnuoy)
summary: - Adding a unit to an existing nova-cloud-controller overwrites the vip in
- the catalogue
+ Adding a unit to an existing service overwrites the vip in the catalogue
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-api (master)

Fix proposed to branch: master
Review: https://review.openstack.org/509121

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/509133

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/509164

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/509172

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.openstack.org/508845
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=36e95189217d1cd68de34985e90382cb39020006
Submitter: Jenkins
Branch: master

commit 36e95189217d1cd68de34985e90382cb39020006
Author: Liam Young <email address hidden>
Date: Mon Oct 2 10:43:13 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: Ibe7d2286de130f045257ad6552f7a5b1279958b6
    Partial-Bug: #1544959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.openstack.org/510034

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.openstack.org/510034
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=1fcbc696490e4b130837ea574c7afc4f0350b788
Submitter: Jenkins
Branch: master

commit 1fcbc696490e4b130837ea574c7afc4f0350b788
Author: Liam Young <email address hidden>
Date: Fri Oct 6 09:34:06 2017 +0000

    Return early from identity-service-relation-joined

    After the previous commit landed it was suggested on that it would
    be cleaner to return early from identity-service-relation-joined
    if the cluster was partially formed rather than use a long if/else
    . This change makes that style update.

    Change-Id: I9dab5d930e13dda6cfe2c6e11648587c87bdb955
    Partial-Bug: #1544959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-glance (master)

Reviewed: https://review.openstack.org/509164
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=be3fcbcad318ff654538772dcca31722b5b4e5ff
Submitter: Jenkins
Branch: master

commit be3fcbcad318ff654538772dcca31722b5b4e5ff
Author: Liam Young <email address hidden>
Date: Tue Oct 3 10:50:02 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: I85e7304b1480a95c784f46490dbcc0c4cc1c3129
    Partial-Bug: #1544959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceilometer (master)

Reviewed: https://review.openstack.org/509172
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=24e359957214893326aa5940bccc8b7399d35350
Submitter: Jenkins
Branch: master

commit 24e359957214893326aa5940bccc8b7399d35350
Author: Liam Young <email address hidden>
Date: Tue Oct 3 13:21:39 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: Icf2753b6b21347af8106ebad03d81309177d09f2
    Partial-Bug: #1544959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-api (master)

Reviewed: https://review.openstack.org/509121
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=6cc620ab270bc0d9f12a2ff3e3b439c7f4e10a90
Submitter: Jenkins
Branch: master

commit 6cc620ab270bc0d9f12a2ff3e3b439c7f4e10a90
Author: Liam Young <email address hidden>
Date: Tue Oct 3 09:38:21 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: I233e0cccb8ccd732080fd239df6d1e7db174eba5
    Partial-Bug: #1544959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (master)

Reviewed: https://review.openstack.org/509133
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=1975c4127121bd026bcbc128c21aeb8585e242a2
Submitter: Jenkins
Branch: master

commit 1975c4127121bd026bcbc128c21aeb8585e242a2
Author: Liam Young <email address hidden>
Date: Tue Oct 3 09:54:35 2017 +0000

    Block endpoint reg if cluster partially formed

    When an existing cluster of the service is scaled out the new unit
    will join with keystone before it is fully clustered. In identity
    joined hook the charmhelpers function canonical_url is called which
    in turn uses another charmhelpers function, resolve_address.
    resolve_address will only return the vip if the vip is set in config
    AND the unit is clustered. This means that the units local address
    is returned and that is then registered with keystone.

    This change gates registering an endpoint if the cluster is
    partially formed.

    Change-Id: If483147e17dab8de2883058ee0f2718a3b7f8ca6
    Partial-Bug: #1544959

Liam Young (gnuoy)
Changed in charm-cinder:
status: In Progress → Fix Committed
Changed in charm-neutron-api:
status: In Progress → Fix Committed
Changed in charm-glance:
status: In Progress → Fix Committed
Changed in charm-ceilometer:
status: In Progress → Fix Committed
Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed
James Page (james-page)
tags: added: stable-backport
James Page (james-page)
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
Changed in charm-neutron-api:
status: Fix Committed → Fix Released
Changed in charm-cinder:
status: Fix Committed → Fix Released
Changed in charm-glance:
status: Fix Committed → Fix Released
Changed in charm-ceilometer:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.