OpenStack Nova Compute Charm

Secrets storage relation changed fails with invalide secret id

Bug #1886540 reported by Alexander Balderson
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
New
Undecided
Unassigned

Bug Description

2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed update-alternatives: using /var/lib/charm/nova-compute-kvm/vaultlocker.conf to provide /etc/vaultlocker/vaultlocker.conf (vaultlocker.conf) in auto mode
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.33.15
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed DEBUG:urllib3.connectionpool:http://192.168.33.15:8200 "POST /v1/auth/approle/login HTTP/1.1" 400 33
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed vaultlocker: invalid secret id
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed Traceback (most recent call last):
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/hooks/secrets-storage-relation-changed", line 774, in <module>
During a fresh deploy of Openstack with vault for disk encrytion
one nova compute unit (3) failed to connect to vault with the error below

It's possible that the ttl for the token was too short for this deployment to come up fully

See the link below for artifacts
https://solutions.qa.canonical.com/#/qa/testRun/3a29d80f-6e83-4014-bdd3-73671e34c41f

2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed main()
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/hooks/secrets-storage-relation-changed", line 767, in main
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed hooks.execute(sys.argv)
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/hooks/charmhelpers/core/hookenv.py", line 943, in execute
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed self._hooks[hook_name]()
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/hooks/secrets-storage-relation-changed", line 704, in secrets_storage_changed
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed configure_local_ephemeral_storage()
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-kvm-3/charm/hooks/nova_compute_utils.py", line 1053, in configure_local_ephemeral_storage
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed dev])
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed raise CalledProcessError(retcode, cmd)
2020-07-04 09:54:57 DEBUG secrets-storage-relation-changed subprocess.CalledProcessError: Command '['vaultlocker', 'encrypt', '--uuid', '93073c15-ed1c-44d1-8b9e-8fea4321bbc0', '/dev/disk/by-dname/bcache-nova-ephemeral']' returned non-zero exit status 1.
2020-07-04 09:54:58 ERROR juju.worker.uniter.operation runhook.go:132 hook "secrets-storage-relation-changed" failed: exit status 1
2020-07-04 09:54:58 INFO juju.worker.uniter resolver.go:126 awaiting error resolution for "relation-changed" hook
2020-07-04 09:56:19 INFO juju.worker.uniter resolver.go:126 awaiting error resolution for "relation-changed" hook

Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

I'm seeing the same in this testrun: https://solutions.qa.canonical.com/testruns/testRun/9f614214-d21f-4485-a120-fa0294f98207 but with a different cause:

```
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 176, in retrieve_secret_id
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed response = client._post('/v1/sys/wrapping/unwrap')
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1239, in _post
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed return self.__request('post', url, **kwargs)
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1263, in __request
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed response = self.session.request(method, url, headers=headers,
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed resp = self.send(prep, **send_kwargs)
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed r = adapter.send(request, **kwargs)
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed raise ConnectionError(e, request=request)
unit-nova-compute-0: 19:48:51 WARNING unit.nova-compute/0.secrets-storage-relation-changed requests.exceptions.ConnectionError: HTTPConnectionPool(host='vault-internal.silo2.lab0.solutionsqa', port=8200): Max retries exceeded with url: /v1/sys/wrapping/unwrap (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2e931c8340>: Failed to establish a new connection: [Errno -2] Name or service not known'))
unit-nova-compute-0: 19:48:51 ERROR juju.worker.uniter.operation hook "secrets-storage-relation-changed" (via explicit, bespoke hook script) failed: exit status 1
```

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.