| 2016-01-20 15:53:17 |
Ryan Beisner |
bug |
|
|
added bug |
| 2016-01-20 16:28:56 |
Ryan Beisner |
description |
glance (next) identity-service-relation-changed hook fails (object-store not found)
I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.
`juju resolved --retry glance/0` results in all-units ready and OK.
hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
# juju stat
http://pastebin.ubuntu.com/14582852/
# glance unit
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership |
glance (next) identity-service-relation-changed hook fails (object-store not found)
I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.
`juju resolved --retry glance/0` results in all-units ready and OK.
hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
# juju stat
http://pastebin.ubuntu.com/14582852/
# glance unit
http://pastebin.ubuntu.com/14583061/
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership |
|
| 2016-01-20 18:15:28 |
Ryan Beisner |
description |
glance (next) identity-service-relation-changed hook fails (object-store not found)
I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.
`juju resolved --retry glance/0` results in all-units ready and OK.
hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
# juju stat
http://pastebin.ubuntu.com/14582852/
# glance unit
http://pastebin.ubuntu.com/14583061/
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership |
glance (next) identity-service-relation-changed hook fails (object-store not found)
I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.
`juju resolved --retry glance/0` results in all-units ready and OK.
hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
# juju stat
http://pastebin.ubuntu.com/14582852/
# glance/0 status history
http://pastebin.ubuntu.com/14583722/
# glance unit
http://pastebin.ubuntu.com/14583061/
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership |
|
| 2016-01-20 18:18:02 |
Ryan Beisner |
description |
glance (next) identity-service-relation-changed hook fails (object-store not found)
I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.
`juju resolved --retry glance/0` results in all-units ready and OK.
hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
# juju stat
http://pastebin.ubuntu.com/14582852/
# glance/0 status history
http://pastebin.ubuntu.com/14583722/
# glance unit
http://pastebin.ubuntu.com/14583061/
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership |
glance (next) identity-service-relation-changed hook fails (object-store not found)
I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.
`juju resolved --retry glance/0` results in all-units ready and OK.
hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
# juju stat
http://pastebin.ubuntu.com/14582852/
# glance/0 status history
http://pastebin.ubuntu.com/14583722/
# keystone/0 status history
http://paste.ubuntu.com/14583733/
# glance unit
http://pastebin.ubuntu.com/14583061/
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership |
|
| 2016-01-21 15:49:36 |
James Page |
glance (Juju Charms Collection): status |
New |
In Progress |
|
| 2016-01-21 15:49:39 |
James Page |
glance (Juju Charms Collection): importance |
Undecided |
High |
|
| 2016-01-21 15:49:42 |
James Page |
glance (Juju Charms Collection): assignee |
|
James Page (james-page) |
|
| 2016-01-21 15:49:44 |
James Page |
glance (Juju Charms Collection): milestone |
|
16.01 |
|
| 2016-01-21 16:23:35 |
James Page |
branch linked |
|
lp:~james-page/charms/trusty/glance/lp1536241 |
|
| 2016-01-21 19:26:24 |
Ryan Beisner |
branch linked |
|
lp:~1chb1n/charms/trusty/glance/next-lp1536241-addl |
|
| 2016-01-22 09:52:54 |
James Page |
glance (Juju Charms Collection): status |
In Progress |
Fix Committed |
|
| 2016-01-22 09:53:02 |
James Page |
tags |
uosci |
openstack uosci |
|
| 2016-01-28 14:06:22 |
James Page |
glance (Juju Charms Collection): status |
Fix Committed |
Fix Released |
|
