vault 1.7 token create failed, local node not active but active cluster node not found

Bug #1987677 reported by Alexander Balderson
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
vault-charm
New
Undecided
Unassigned

Bug Description

On a Jammy deployment on AWS with a single vault node running 1.7/stable charm. vault successfully ran operator init and unsealed, but creating the token failed with an error making the API request:

juju exec -m foundations-aws:kubernetes-aws -u vault/leader 'export VAULT_TOKEN=s.6wi8c8wGt50Jc1w0hNXDoudM && export VAULT_ADDR=http://127.0.0.1:8200 && /snap/bin/vault token create --ttl=5m'
2022-08-24-23:16:22 root ERROR [localhost] STDOUT follows:
b''
2022-08-24-23:16:22 root ERROR [localhost] STDERR follows:
Error creating token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/create
Code: 500. Errors:

* local node not active but active cluster node not found

There doesnt seem to be any helpful logging in the juju output or in the syslog about the failure.

Revision history for this message
Nobuto Murata (nobuto) wrote :

I can confirm this behavior. From what I saw, it looks like some moments are necessary to settle down the HA status of the vault cluster after unsealing it even with a single node cluster.

Here is an output from the unseal command and as we can see "n/a" is recorded for HA Cluster status and there is no cluster address either.

+ vault operator unseal Lp0QKU6ts/9Eydb/A3s3DUs5qcyzmxfpDI5M4jfmhY8=
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.8.8
Storage Type raft
Cluster Name vault-cluster-777ad2a5
Cluster ID 0afa6371-5941-493f-d258-d76389cdbecc
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
Raft Committed Index 24
Raft Applied Index 24

With the status above, `token create` can fail by missing active cluster node.

++ vault token create -ttl=10m -format json
Error creating token: Error making API request.

URL: POST http://192.168.151.108:8200/v1/auth/token/create
Code: 500. Errors:

* local node not active but active cluster node not found

Here is the expected status after some time for the record.

$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.8.8
Storage Type raft
Cluster Name vault-cluster-777ad2a5
Cluster ID 0afa6371-5941-493f-d258-d76389cdbecc
HA Enabled true
HA Cluster https://192.168.151.108:8201
HA Mode active
Active Since 2023-10-18T15:22:40.840977987Z
Raft Committed Index 1478
Raft Applied Index 1478

Revision history for this message
Nobuto Murata (nobuto) wrote :

A quick trick would be something like the following before executing `token create`.

until vault status -format json | jq -r --exit-status .leader_cluster_address; do
    sleep 1
done

[just after unsealing]
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.8.8",
  "migration": false,
  "cluster_name": "vault-cluster-eb767fbb",
  "cluster_id": "3011c079-df2d-bd0d-dd3b-1e8c32a4ecf8",
  "recovery_seal": false,
  "storage_type": "raft",
  "ha_enabled": true,
  "active_time": "0001-01-01T00:00:00Z",
  "raft_committed_index": 24,
  "raft_applied_index": 24
}

[after settling down]
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.8.8",
  "migration": false,
  "cluster_name": "vault-cluster-eb767fbb",
  "cluster_id": "3011c079-df2d-bd0d-dd3b-1e8c32a4ecf8",
  "recovery_seal": false,
  "storage_type": "raft",
  "ha_enabled": true,
  "is_self": true,
  "active_time": "2023-10-19T02:24:16.7102222Z",
  "leader_address": "http://192.168.151.110:8200",
  "leader_cluster_address": "https://192.168.151.110:8201",
  "raft_committed_index": 43,
  "raft_applied_index": 43
}

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.