[Tesora DBaaS] Unable to get list of or create new instance after controller deletion

Bug #1587010 reported by Andrii Konovka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel Plugins
Won't Fix
High
Brian Hunter

Bug Description

Steps to reproduce:
1. Create new environment with 3 controllers, 1 compute + cinder role, 1 node with Tesora DBaaS role.
2. Deploy envrironment.
3. Delete one controller and deploy environment.

Expected result:
* All plugins resources are migrated to another Controller node
* The environment is redeployed successfully when adding/removing Controller node.

Actual result:
* Unable to create new instance or get list of existing.

Diagnostic snapshot, trove config and logs attached.

Debug log:
root@node-10:/opt/tesora/dbaas/bin# trove --debug list
DEBUG (session:198) REQ: curl -g -i -X GET http://10.109.1.3:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:203) Starting new HTTP connection (1): 10.109.1.3
DEBUG (connectionpool:383) "GET /v2.0 HTTP/1.1" 200 336
DEBUG (session:215) RESP: [200] content-length: 336 vary: X-Auth-Token server: Apache connection: close date: Thu, 26 May 2016 14:25:00 GMT content-type: application/json x-openstack-request-id: req-1d40af7d-d35b-4882-bb73-2047931bfd1b
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.109.1.3:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2:86) Making authentication request to http://10.109.1.3:5000/v2.0/tokens
INFO (connectionpool:238) Resetting dropped connection: 10.109.1.3
DEBUG (connectionpool:383) "POST /v2.0/tokens HTTP/1.1" 200 4959
DEBUG (iso8601:184) Parsed 2016-05-26T15:25:00.986488Z into {'tz_sign': None, 'second_fraction': u'986488', 'hour': u'15', 'daydash': u'26', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'00', 'tz_minute': None, 'year': u'2016', 'separator': u'T', 'monthdash': u'05', 'day': None, 'minute': u'25'} with default timezone <iso8601.iso8601.Utc object at 0x7fd9f7ed6590>
DEBUG (iso8601:140) Got u'2016' for 'year' with default None
DEBUG (iso8601:140) Got u'05' for 'monthdash' with default 1
DEBUG (iso8601:140) Got 5 for 'month' with default 5
DEBUG (iso8601:140) Got u'26' for 'daydash' with default 1
DEBUG (iso8601:140) Got 26 for 'day' with default 26
DEBUG (iso8601:140) Got u'15' for 'hour' with default None
DEBUG (iso8601:140) Got u'25' for 'minute' with default None
DEBUG (iso8601:140) Got u'00' for 'second' with default None
DEBUG (iso8601:184) Parsed 2016-05-26T15:25:00.986488Z into {'tz_sign': None, 'second_fraction': u'986488', 'hour': u'15', 'daydash': u'26', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'00', 'tz_minute': None, 'year': u'2016', 'separator': u'T', 'monthdash': u'05', 'day': None, 'minute': u'25'} with default timezone <iso8601.iso8601.Utc object at 0x7fd9f7ed6590>
DEBUG (iso8601:140) Got u'2016' for 'year' with default None
DEBUG (iso8601:140) Got u'05' for 'monthdash' with default 1
DEBUG (iso8601:140) Got 5 for 'month' with default 5
DEBUG (iso8601:140) Got u'26' for 'daydash' with default 1
DEBUG (iso8601:140) Got 26 for 'day' with default 26
DEBUG (iso8601:140) Got u'15' for 'hour' with default None
DEBUG (iso8601:140) Got u'25' for 'minute' with default None
DEBUG (iso8601:140) Got u'00' for 'second' with default None
DEBUG (session:198) REQ: curl -g -i -X GET http://10.109.1.3:8779/v1.0/c723adf2c5504a86811a75f92bb45920/instances -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}260d8a416d936eed364c459b6e0079a7226c3c7d"
INFO (connectionpool:203) Starting new HTTP connection (1): 10.109.1.3
DEBUG (connectionpool:383) "GET /v1.0/c723adf2c5504a86811a75f92bb45920/instances HTTP/1.1" 503 100
DEBUG (session:215) RESP: [503] date: Thu, 26 May 2016 14:25:01 GMT content-length: 100 content-type: text/plain; charset=UTF-8
RESP BODY: 503 Service Unavailable

The server is currently unavailable. Please try again at a later time.

DEBUG (shell:736) Service Unavailable (HTTP 503)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/troveclient/shell.py", line 731, in main
    sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/troveclient/shell.py", line 546, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/troveclient/v1/shell.py", line 232, in do_list
    include_clustered=args.include_clustered)
  File "/usr/lib/python2.7/dist-packages/troveclient/v1/instances.py", line 171, in list
    {"include_clustered": include_clustered})
  File "/usr/lib/python2.7/dist-packages/troveclient/base.py", line 70, in _paginated
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/troveclient/client.py", line 446, in request
    raise exceptions.from_response(resp, body, url)
ServiceUnavailable: Service Unavailable (HTTP 503)
ERROR: Service Unavailable (HTTP 503)

Tags: tesora
Revision history for this message
Andrii Konovka (akonovka) wrote :
Revision history for this message
Andrii Konovka (akonovka) wrote :
Revision history for this message
Andrii Konovka (akonovka) wrote :
Changed in fuel-plugins:
importance: Undecided → High
assignee: nobody → Brian Hunter (bhunter)
milestone: none → 8.0
tags: added: tesora
Revision history for this message
Irina Povolotskaya (ipovolotskaya) wrote :

per Brian's question:
What *seems* to be happening is that our mysql password for the trove user is getting changed as we perform this controller delete operation.

We have a "mysql password" we're auto-generating and storing in our plugin attributes/metadata.
Here's a bit of environment_config.yaml:
attributes:
  metadata:
    ..
    mysql_password:
      generator: 'password'

When the original 3 controllers have completed provisioning, a particular value will exist for the password, and this appear the same on all 3 controllers. for example:
root@node-9:~# hiera fuel-plugin-tesora-dbaas
{"tesora_accept_terms"=>"I AGREE",
 "tesora_guest_download_username"=>"xxxxxxxx",
 "tesora_guest_download_password"=>"xxxxxxxx",
 "metadata"=>
  {"mysql_password"=>"bnL1TIMlk9lQbG4ZTClyF1YN",

When the primary controller has been deleted and the environment redeployed, the value for 'mysql_password' will have changed.
(We see a problem because the plugin's mysql code has run, but it appears the trove config files have not been regenerated.)

Is there a way to prevent this regeneration of the password?

Does this password regeneration happen on all changes to controllers, or just ones affecting primary_controller?
Does it happen on any other events/conditions?

Changed in fuel-plugins:
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.