Unable to create trust errors in glance-api

Could you try this with the log level set at debug? I can't find that error message in keystoneauth1 or glance_store, so I'm not sure where it's coming from. glance_store doesn't try to use collect_timing directly, so it must be somewhere else.

I'm seeing the same messages on OpenStack Victoria 22.1.0. I set log level to debug, here's my output:

Apr 06 15:04:54 infra1-glance-container-99614ac2 glance-wsgi-api[1824]: 2021-04-06 15:04:54.274 1824 INFO glance.api.v2.image_data [req-c572bb10-58e1-4006-8f0e-9bd721b2986b 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.
Apr 06 15:07:14 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:10.195 1837 ERROR keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Bad response code while validating token: 500 An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672): keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672)
Apr 06 15:07:40 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:38.197 1837 WARNING keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Identity response: {"error": {"code": 500, "title": "Internal Server Error", "message": "An unexpected error prevented the server from fulfilling your request."}}: keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-55767c13-a450-4f81-90ab-5a7af6b3f672)
Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1822]: DAMN ! worker 13 (pid: 1836) died, killed by signal 9 :( trying respawn ...
Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1822]: Respawned uWSGI worker 13 (new pid: 5799)
Apr 06 15:07:44 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:40.427 1837 CRITICAL keystonemiddleware.auth_token [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to validate token: Failed to fetch token data from identity server: keystonemiddleware.auth_token._exceptions.ServiceError: Failed to fetch token data from identity server
Apr 06 15:07:46 infra1-glance-container-99614ac2 uwsgi[1837]: Tue Apr 6 15:07:45 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during GET /v2/images?limit=1000&sort_key=created_at&sort_dir=desc (192.168.110.250)
Apr 06 15:07:45 infra1-glance-container-99614ac2 glance-wsgi-api[1837]: 2021-04-06 15:07:45.629 1837 CRITICAL glance [req-ad8d0db9-b630-4237-9fff-d7ff282155d2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error
                                                                        2021-04-06 15:07:45.629 1837 ERROR glance OSError: write error
                                                                        2021-04-06 15:07:45.629 1...

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance/+/795772

Reviewed: https://review.opendev.org/c/openstack/glance/+/795772
Committed: https://opendev.org/openstack/glance/commit/9b683678b2a19bb1069aace58e054150a78b5f6f
Submitter: "Zuul (22348)"
Branch: master

commit 9b683678b2a19bb1069aace58e054150a78b5f6f
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/glance/+/800101

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/glance/+/800102

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/glance/+/800103

This issue was fixed in the openstack/glance 23.0.0.0b2 development milestone.

Reviewed: https://review.opendev.org/c/openstack/glance/+/800101
Committed: https://opendev.org/openstack/glance/commit/7ce475ce11ebb5813ce487bfa9cf64458ac08ac5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7ce475ce11ebb5813ce487bfa9cf64458ac08ac5
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052
    (cherry picked from commit 9b683678b2a19bb1069aace58e054150a78b5f6f)

Reviewed: https://review.opendev.org/c/openstack/glance/+/800102
Committed: https://opendev.org/openstack/glance/commit/3139ef39cb34c411d340572b098a790c92cfdf7e
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 3139ef39cb34c411d340572b098a790c92cfdf7e
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052
    (cherry picked from commit 9b683678b2a19bb1069aace58e054150a78b5f6f)

Reviewed: https://review.opendev.org/c/openstack/glance/+/800103
Committed: https://opendev.org/openstack/glance/commit/f213c1bf44388fde5802bd707cf2850f88a26d5e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f213c1bf44388fde5802bd707cf2850f88a26d5e
Author: Erno Kuvaja <email address hidden>
Date: Thu Jun 10 13:51:49 2021 +0100

    Revert "Remove all usage of keystoneclient"

    This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

    The Swift driver not being able to use Trusts had nothing to do
    about the store driver itself nor that keystoneauth1 would have
    broken the feature, but rather it not having the functionality
    in the first place and us not catching that on reviews.

    We should figure out how to test this before we try to replace
    this code again.

    merge-conflict on requirements jsonschema.

    Change-Id: If12a013404296486dc387b099477d1608b24ba63
    Closes-Bug: #1916052
    (cherry picked from commit 9b683678b2a19bb1069aace58e054150a78b5f6f)

This issue was fixed in the openstack/glance 20.2.0 release.

This issue was fixed in the openstack/glance 21.1.0 release.

This issue was fixed in the openstack/glance 22.1.0 release.

Hello,

I upgraded from Victoria to Yoga.

DISTRIB_ID="OSA"
DISTRIB_RELEASE="25.2.0"
DISTRIB_CODENAME="Yoga"
DISTRIB_DESCRIPTION="OpenStack-Ansible"

I have the same error infra1-glance-container-dc13a04b glance-wsgi-api[282311]: 2023-01-23 13:25:06.745 282311 INFO glance.api.v2.image_data [req-10359154-0be4-4da1-9e85-2d94079c17b4 b2dca74976034d5b9925bdcb03470603 021ce436ab004cde851055bac66370bc - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.