component glance master scenario1 standalone fails tempestconf

seeing errors in the glance container logs which seems to line up with the error seen in tempestconf timewise

https://logserver.rdoproject.org/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-glance-master/f778547/logs/undercloud/var/log/containers/glance/api.log.txt.gz

2021-07-30 10:02:10.477 180 DEBUG oslo_concurrency.processutils [-] 'blkid overlay -s UUID -o value' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:474
2021-07-30 10:02:10.477 180 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7003306d-4ba2-48e9-a117-88a327268641]: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '' _process_cmd /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:482
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 477, in _process_cmd
    ret = func(*f_args, **f_kwargs)
  File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 274, in _wrap
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
    return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
  File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 146, in custom_execute
    on_completion=on_completion, *cmd, **kwargs)
  File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute
    cmd=sanitized_cmd)
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: ''

(Thanks to Rajat for looking into this)

2 patches from cinder side are missing from the run
[1] python-cinderclient: https://review.opendev.org/c/openstack/python-cinderclient/+/783628
[2] cinder: https://review.opendev.org/c/openstack/cinder/+/783389

The actual error is:- TypeError: create() missing 1 required positional argument: 'instance_id'.
blkid overlay -s UUID -o value error is seen in successful jobs too, would be good to get that fixed too to avoid those errors.

<< 2 patches from cinder side are missing from the run
<< [1] python-cinderclient: https://review.opendev.org/c/openstack/python-cinderclient/+/783628
cinderclient-8.0.0 update in RDO waiting for https://review.opendev.org/q/topic:%22volumev2-removal%22+(status:open%20OR%20status:merged), some context can be found in https://review.rdoproject.org/r/c/rdoinfo/+/34695

<< [2] cinder: https://review.opendev.org/c/openstack/cinder/+/783389
cinder patch is there in the deployment, latest run have https://github.com/openstack/cinder/commit/b06a30.

The issue is caused with glance_store patch https://review.opendev.org/c/openstack/glance_store/+/782200. Which needs cinderclient-8.0.0 to get fixed and that's blocked as mentioned above. So we can pin glance_store to 2.6.0 in rdoinfo same as upper-constraints. Ideally it should have been pinned like other libraries, but need to check on history why it's not so or intentionally kept unpinned.
And when cinderclient-8.0.0 get's unblocked glance_store can also be unpinned back if needed.

@ykarel you mean pin glance_store here? https://github.com/redhat-openstack/rdoinfo/blob/master/buildsys-tags/cloud8s-openstack-xena-testing.yml

@marios at https://github.com/redhat-openstack/rdoinfo/blob/master/tags/xena-uc.yml#L159, add source-branch: 2.6.0 as per u-c https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L552

<< Ideally it should have been pinned like other libraries, but need to check on history why it's not so or intentionally kept unpinned.
I checked this and our automation didn't picked it as there is difference in project name in rdoinfo glance_store vs in upper-constraints glance-store. If storage team thinks need to follow u-c like other libraries, will work on fixing automation as this is the only exception i see. I think it should follow like other libraries and client but since it's not happening for long looking for confirmation. The only short time side effect will be RDO will build package for pinned tag, so going back for cases where there are new commits than last tag.

thanks @ykarel - I was a bit confused because I checked and 2.6.0 is the latest tag at https://github.com/openstack/glance_store/tags

but it looks like the offending patch at https://review.opendev.org/c/openstack/glance_store/+/782200 merged after that was made (14th July tag, 23rd july patch).

posted there https://review.rdoproject.org/r/c/rdoinfo/+/34869

thanks @ykarel

looks like the pin has helped. it merged Aug 06 4:36 PM [1] and then we had 2 green runs at [2]

        * openstack-component-glance 1 hr 9 mins 29 secs 2021-08-08 09:05:36 SUCCESS
        * openstack-component-glance 1 hr 19 mins 44 secs 2021-08-07 09:06:54 SUCCESS

[1] https://review.rdoproject.org/r/c/rdoinfo/+/34869/1#message-f7d33cddc3ee81b462f5b40043b88f26051fc710
[2] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario001-standalone-glance-master

Just to close this out, after the pin at [1] to unblock us for this bug, the patch at [2] will ensure the pin is synchronized with upper constraints.

Moving the bug fix released please move back if you disagree and reach out to |ruck or |rover on oftc #tripleo or #oooq

[1] https://review.rdoproject.org/r/c/rdoinfo/+/34869
[2] https://review.rdoproject.org/r/c/rdo-infra/releng/+/34884