| 2022-04-29 07:58:18 |
Liam Young |
description |
If the certificates-relation-joined hook runs before vault has been configured
then the hook fails. This is because the charm tries to access the running
vault service but at this point vault is not configured or running. This
regression appears to have been introduced by 1159e547
( https://review.opendev.org/c/openstack/charm-vault/+/828885 ). This patch
seems to incorrectly gate on the `certificates.available` flag. Despite the
name, `certificates.available` only indicates that certificate have
been requested i.e. it means "a certificate is available to be processed" it
does not mean that vault is ready.
The issue can be reproduced with this bundle:
series: focal
applications:
keystone-mysql-router:
charm: ch:mysql-router
channel: latest/edge
vault-mysql-router:
charm: ch:mysql-router
channel: latest/edge
mysql-innodb-cluster:
charm: ch:mysql-innodb-cluster
constraints: mem=3072M
num_units: 3
channel: latest/edge
vault:
num_units: 3
charm: ch:vault
channel: latest/edge
keystone:
charm: ch:keystone
num_units: 1
options:
admin-password: openstack
channel: latest/edge
relations:
- - 'vault:shared-db'
- 'vault-mysql-router:shared-db'
- - 'keystone:shared-db'
- 'keystone-mysql-router:shared-db'
- - 'keystone-mysql-router:db-router'
- 'mysql-innodb-cluster:db-router'
- - 'vault:certificates'
- 'keystone:certificates'
Note that in the bundle the relation between vault-mysql-router and
mysql-innodb-cluster is missing. This simulates the situation where
a `certificates-relation-joined` fires before vault has been setup
because the initial configuration of vault is gated on
`shared-db.available` flag being set.
This bug can present itself in subtly different ways that may initially
appear like the db-router/shared-db relations are at fault. In the
output below vault/0 and vault/2 are both hitting this bug and in the
case of vault/0 the bug was hit before the unit sent its db access request
to vault-mysql-router/2 which is why vault-mysql-router/2 is reporting it
has missing data.
Unit Workload Message
vault/0 error hook failed: "certificates-relation-joined"
vault-mysql-router/2 waiting shared-db' incomplete, Waiting for proxied
DB creation from cluster
vault/1* blocked Vault needs to be initialized
vault-mysql-router/1 active Unit is ready
vault/2 error hook failed: "certificates-relation-joined"
vault-mysql-router/0* active Unit is ready |
If the certificates-relation-joined hook runs before vault has been
configured then the hook fails. This is because the charm tries to access the running vault service but at this point vault is not configured or running. This regression appears to have been introduced by 1159e547
( https://review.opendev.org/c/openstack/charm-vault/+/828885 ). This patch seems to incorrectly gate on the `certificates.available` flag. Despite the name, `certificates.available` only indicates that certificate have been requested i.e. it means "a certificate is available to be processed" it does not mean that vault is ready.
The issue can be reproduced with this bundle:
series: focal
applications:
keystone-mysql-router:
charm: ch:mysql-router
channel: latest/edge
vault-mysql-router:
charm: ch:mysql-router
channel: latest/edge
mysql-innodb-cluster:
charm: ch:mysql-innodb-cluster
constraints: mem=3072M
num_units: 3
channel: latest/edge
vault:
num_units: 3
charm: ch:vault
channel: latest/edge
keystone:
charm: ch:keystone
num_units: 1
options:
admin-password: openstack
channel: latest/edge
relations:
- - 'vault:shared-db'
- 'vault-mysql-router:shared-db'
- - 'keystone:shared-db'
- 'keystone-mysql-router:shared-db'
- - 'keystone-mysql-router:db-router'
- 'mysql-innodb-cluster:db-router'
- - 'vault:certificates'
- 'keystone:certificates'
Note that in the bundle the relation between vault-mysql-router and
mysql-innodb-cluster is missing. This simulates the situation where
a `certificates-relation-joined` fires before vault has been setup
because the initial configuration of vault is gated on
`shared-db.available` flag being set.
This bug can present itself in subtly different ways that may initially
appear like the db-router/shared-db relations are at fault. In the
output below vault/0 and vault/2 are both hitting this bug and in the
case of vault/0 the bug was hit before the unit sent its db access request
to vault-mysql-router/2 which is why vault-mysql-router/2 is reporting it
has missing data.
Unit Workload Message
vault/0 error hook failed: "certificates-relation-joined"
vault-mysql-router/2 waiting shared-db' incomplete, Waiting for proxied
DB creation from cluster
vault/1* blocked Vault needs to be initialized
vault-mysql-router/1 active Unit is ready
vault/2 error hook failed: "certificates-relation-joined"
vault-mysql-router/0* active Unit is ready |
|
