authorize_charm action context deadline exceeded

Bug #1865547 reported by Ashley Lai
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
vault-charm
Triaged
Medium
Unassigned

Bug Description

juju run-action -m kubernetes --format=yaml vault/leader authorize-charm token=*

vault/0 active idle 6 10.244.48.217 8200/tcp Unit is ready (active: true, mlock: enabled)
vault/1 active idle 7 10.244.48.207 8200/tcp Unit is ready (active: false, mlock: enabled)
vault/2* error idle 8 10.244.48.209 8200/tcp hook failed: "update-status"

The authorize-charm action left one of the vault unit in error state. The other two vault units are active.

2020-02-27 16:34:28 DEBUG juju-log Opening vault port
2020-02-27 16:34:28 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:248:mysql_setup
2020-02-27 16:34:28 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:277:database_not_ready
2020-02-27 16:34:29 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:360:cluster_connected
2020-02-27 16:34:29 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:504:send_vault_url_and_ca
2020-02-27 16:34:29 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:534:prime_assess_status
2020-02-27 16:34:29 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:742:publish_ca_info
2020-02-27 16:34:34 DEBUG update-status active
2020-02-27 16:34:34 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:764:publish_global_client_cert
2020-02-27 16:34:35 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:848:tune_pki_backend_config_changed
2020-02-27 16:34:35 DEBUG update-status active
2020-02-27 16:34:40 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 866, in tune_pki_backend_config_changed
    vault_pki.tune_pki_backend(ttl=ttl, max_ttl=max_ttl)
  File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault_pki.py", line 53, in tune_pki_backend
    client = vault.get_local_client()
  File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault.py", line 249, in get_local_client
    client.auth_approle(app_role_id)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
    return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 1729, in auth
    **kwargs
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 159, in auth
    response = self.post(url, **kwargs).json()
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post
    return self.request('post', url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 233, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/utils.py", line 39, in raise_for_error
    raise exceptions.InternalServerError(message, errors=errors)
hvac.exceptions.InternalServerError: context deadline exceeded

2020-02-27 16:34:40 DEBUG update-status Traceback (most recent call last):
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/charm/hooks/update-status", line 22, in <module>
2020-02-27 16:34:40 DEBUG update-status main()
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
2020-02-27 16:34:40 DEBUG update-status bus.dispatch(restricted=restricted_mode)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
2020-02-27 16:34:40 DEBUG update-status _invoke(other_handlers)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
2020-02-27 16:34:40 DEBUG update-status handler.invoke()
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
2020-02-27 16:34:40 DEBUG update-status self._action(*args)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 866, in tune_pki_backend_config_changed
2020-02-27 16:34:40 DEBUG update-status vault_pki.tune_pki_backend(ttl=ttl, max_ttl=max_ttl)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault_pki.py", line 53, in tune_pki_backend
2020-02-27 16:34:40 DEBUG update-status client = vault.get_local_client()
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault.py", line 249, in get_local_client
2020-02-27 16:34:40 DEBUG update-status client.auth_approle(app_role_id)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
2020-02-27 16:34:40 DEBUG update-status return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 1729, in auth
2020-02-27 16:34:40 DEBUG update-status **kwargs
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 159, in auth
2020-02-27 16:34:40 DEBUG update-status response = self.post(url, **kwargs).json()
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post
2020-02-27 16:34:40 DEBUG update-status return self.request('post', url, **kwargs)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 233, in request
2020-02-27 16:34:40 DEBUG update-status utils.raise_for_error(response.status_code, text, errors=errors)
2020-02-27 16:34:40 DEBUG update-status File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.6/site-packages/hvac/utils.py", line 39, in raise_for_error
2020-02-27 16:34:40 DEBUG update-status raise exceptions.InternalServerError(message, errors=errors)
2020-02-27 16:34:40 DEBUG update-status hvac.exceptions.InternalServerError: context deadline exceeded
2020-02-27 16:34:40 ERROR juju.worker.uniter.operation runhook.go:132 hook "update-status" failed: exit status 1

Revision history for this message
Ashley Lai (alai) wrote :
Revision history for this message
Andrew McLeod (admcleod) wrote :

Hi,

What distribution / release combination was this?
Is this intermittent or every attempt?

Changed in vault-charm:
status: New → Incomplete
Revision history for this message
Ashley Lai (alai) wrote :

It is intermittent on bionic stein. Bundle is attached.

Ashley Lai (alai)
Changed in vault-charm:
status: Incomplete → New
Changed in vault-charm:
importance: Undecided → Medium
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.