kolla

Keystone key repository missing for credentials

  1. Series train
  2. Bug #1863643
Bug #1863643 reported by Pierre Riteau
50
This bug affects 10 people
Affects Status Importance Assigned to Milestone
kolla
Rocky
New
Medium
Unassigned
Stein
New
Medium
Unassigned
Train
New
Medium
Unassigned
Ussuri
Triaged
Medium
Dincer Celik
kolla-ansible
Won't Fix
Medium
Unassigned
Rocky
Won't Fix
Medium
Unassigned
Stein
Won't Fix
Medium
Unassigned
Train
Won't Fix
Medium
Unassigned
Ussuri
Won't Fix
Medium
Unassigned

Bug Description

Keystone uses the [credential]/key_repository configuration option to point to a directory containing Fernet keys used to encrypt and decrypt credentials stored in the credential backend. The default value is /etc/keystone/credential-keys/

This is not configured with Kolla-Ansible. On a Stein deployment, we see periodic ERROR logs associated with it:

2020-02-17 17:00:25.263 30 ERROR keystone.common.fernet_utils [req-386e223e-5f08-4959-bd0b-3dbbfd3b534f 73c7a9a7175e46e9971181963267225b efa5d1ce80324476b4ce189686dda2d1 - default default] Either [None] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/

Note from the docs: Fernet keys used to encrypt credentials have no relationship to Fernet keys used to encrypt Fernet tokens. Both sets of keys should be managed separately and require different rotation policies. Do not share this repository with the repository used to manage keys for Fernet tokens.

Revision history for this message
Mark Goddard (mgoddard) wrote :

Pierre, do you know what would be affected by this? Seems like it must be a less frequently used feature otherwise we'd hear about it.

Revision history for this message
Mark Goddard (mgoddard) wrote :

Perhaps it implies our credentials are not encrpyted in the DB? https://docs.openstack.org/keystone/pike/admin/identity-credential-encryption.html

Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → Medium
Dincer Celik (dincercelik)
Changed in kolla-ansible:
assignee: nobody → Dincer Celik (osmanlicilegi)
Dincer Celik (dincercelik)
affects: kolla-ansible → kolla
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/708955

Changed in kolla:
status: New → In Progress
Revision history for this message
Christopher Wellie (well100) wrote :

I have the same problem under train

Mark Goddard (mgoddard)
Changed in kolla:
milestone: 10.0.0 → none
Revision history for this message
Christopher Wellie (well100) wrote :

When will the bug be processed?

Revision history for this message
Mark Goddard (mgoddard) wrote :

Hi Christopher. This bug has been open for some time, and although it is marked as in progress, I don't think Dincer is working on it (he may correct me). In that case, it will be processed when someone picks it up and works on it. Perhaps you are able to help?

Changed in kolla:
status: In Progress → Triaged
Revision history for this message
Till Plüer (tplueer) wrote :

Same problem still exists in victoria

Revision history for this message
Jacolex (jacolex) wrote :

Hi, the issue still persists in Xena. I have to deploy manual scripts based on fernet-rotate.sh with oneliner

keystone-manage --config-file /etc/keystone/keystone.conf credential_rotate --keystone-user keystone --keystone-group keystone

Additionally I've added rsync just like fernet-node-sync.sh and I've modified crontab.

Revision history for this message
yule sun (syle87) wrote :

I have a issue like this when i upgrade my cluster from Queens to Rocky, I got an error like this :

2022-09-15 14:47:36.707 23 WARNING stevedore.named [-] Could not load uuid
2022-09-15 14:48:28.868 133 ERROR keystone.common.fernet_utils [-] Either [credential] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/
2022-09-15 14:48:28.869 133 ERROR keystone.common.fernet_utils [-] Either [credential] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/

And i can't access the keystone endpoint , is this a same issue?

[root@zlmanager01 keystone]# curl http://192.168.2.254:35357/v3
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 root@localhost to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

Revision history for this message
Pierre Riteau (priteau) wrote :

Hello yule sun (syle87),

The error you are seeing is more likely related to the first WARNING log:

2022-09-15 14:47:36.707 23 WARNING stevedore.named [-] Could not load uuid

UUID tokens were removed in Queens. You probably need to update keystone.conf to use

[token]
provider = fernet

instead of:

[token]
provider = uuid

The ERROR logs about credential keys are not fatal.

Revision history for this message
Jacolex (jacolex) wrote :

Hello
My files and script templates changes:

additional lines:

/usr/local/share/kolla-ansible/ansible/roles/keystone/defaults/main.yml
[...]
  keystone-ssh:
    volumes:
      - "keystone_credential_tokens:/etc/keystone/credential-keys"
  keystone-fernet:
    volumes:
      - "keystone_credential_tokens:/etc/keystone/credential-keys"
keystone_default_volumes:
  - "{% if keystone_token_provider == 'fernet' %}keystone_credential_tokens:/etc/keystone/credential-keys{% endif %}"

additional lines in two scripts regarding to credential-keys. Also commenting: set -o errexit, set -o pipefail.

/usr/local/share/kolla-ansible/ansible/roles/keystone/templates/fernet-push.sh.j2
#set -o errexit
#set -o pipefail
[...]
/usr/bin/rsync --dry-run -az -e 'ssh -i /var/lib/keystone/.ssh/id_rsa -p {{ hostvars[host]['keystone_ssh_port'] }} -F /var/lib/keystone/.ssh/config' --delete /etc/keystone/credential-keys/ keystone@{{ 'api' | kolla_address(host) | put_address_in_context('url') }}:/etc/keystone/credential-keys
[...]
/usr/bin/rsync -az -e 'ssh -i /var/lib/keystone/.ssh/id_rsa -p {{ hostvars[host]['keystone_ssh_port'] }} -F /var/lib/keystone/.ssh/config' --delete /etc/keystone/credential-keys/ keystone@{{ 'api' | kolla_address(host) | put_address_in_context('url') }}:/etc/keystone/credential-keys
[...]

/usr/local/share/kolla-ansible/ansible/roles/keystone/templates/fernet-rotate.sh.j2
#set -o errexit
#set -o pipefail
[...]
keystone-manage --config-file /etc/keystone/keystone.conf fernet_rotate --keystone-user {{ keystone_username }} --keystone-group {{ keystone_groupname }}
keystone-manage --config-file /etc/keystone/keystone.conf credential_migrate --keystone-user {{ keystone_username }} --keystone-group {{ keystone_groupname }}
keystone-manage --config-file /etc/keystone/keystone.conf credential_rotate --keystone-user {{ keystone_username }} --keystone-group {{ keystone_groupname }}
[...]

Revision history for this message
yule sun (syle87) wrote :

    Thank you Pierre Riteau and Jacolex
    Fixed this issuse when i upgraded the /etc/kolla/keystone/keystone.conf, change the provider=uuid to provider=fernet, and then use he command reconfigure to update the keystone roles.
    Thank you both very much for your replies and answers.
    Best wishes to you.

Revision history for this message
Michal Nasiadka (mnasiadka) wrote :

All versions from Rocky to Ussuri are EOL, marking as won't fix.

no longer affects: kolla
Changed in kolla-ansible:
status: New → Won't Fix
Revision history for this message
Jacolex (jacolex) wrote :

Hello
What about new versions of kolla? Stil no support for credential keys.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.