Buld-containers-quay-push: libselinux-python aren't installed

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/864852

Looks like that started at some point on the 16th - from build history at [1]

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 24 mins 55 secs 2022-11-17 00:22:11 FAILURE

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 16 mins 22 secs 2022-11-16 16:14:36 FAILURE

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 50 mins 10 secs 2022-11-16 00:16:37 SUCCESS

  * periodic-tripleo-ci-build-containers-ubi-9-quay-push-master openstack/tripleo-ci master openstack-periodic-integration-main 38 mins 41 secs 2022-11-15 17:25:12 SUCCESS

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-9-quay-push-master&pipeline=openstack-periodic-integration-main&skip=0

haven't found workaround yet - here's what i tried:

* adding dnf install python3-libselinux package [1] (Takashi posted the same but earlier in pre.yaml in comment #1 avove but I wanted to explicitly add it to main) and that didn't work ... see [1] it was already installed on the host

* tried adding into tcib/tripleo-common with [2] but didn't work -> same result [3]

* tried adding pip install (venv for container builds) @ [4] -> same :/ [5]

[1] https://review.opendev.org/c/openstack/tripleo-ci/+/864889/1/roles/build-containers/tasks/main.yaml#230
[2] https://review.opendev.org/c/openstack/tripleo-common/+/864892
[3] https://logserver.rdoproject.org/32/46132/2/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/54de10f/logs/build.log
[4] https://review.opendev.org/c/openstack/tripleo-ci/+/864889/2/roles/build-containers/vars/centos-9.yml
[5] https://logserver.rdoproject.org/32/46132/3/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/29e7d45/job-output.txt

https://github.com/openstack/python-tripleoclient/commit/d18b9273859dc2c5909104f1aa7dab0c2c4f982e looks to have added "ANSIBLE_PYTHON_INTERPRETER": "{{ansible_playbook_python}}"

Considering the issue and solution in: https://bugs.launchpad.net/tripleo/+bug/1862330, tried the DNM patch: https://review.opendev.org/c/openstack/python-tripleoclient/+/864952 to remove that line.

Testing in https://review.rdoproject.org/r/c/testproject/+/25325 shows the build to be progressing - with no selinux errors.

Looks like these jobs are using venv to build containers. Should we change those jobs to not use venv as there is libselinux-python required? Proposed https://review.rdoproject.org/r/c/config/+/46139

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/864852
Reason: This does not solve the problem. We are now looking into https://review.rdoproject.org/r/c/config/+/46139

Thanks to Rabi and Takashi for the fix.

Testproject: https://review.rdoproject.org/r/c/testproject/+/25325/125/.zuul.yaml#7 is green var "buildcontainers_venv: false"

https://logserver.rdoproject.org/25/25325/125/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/e519ee1/logs/build_containers.sh confirms no virtual env used.

Adding a summary of the discussion with Rabi:

Patch[0] brought the issue to notice as we changed that playbook to use the ansible controller interpreter rather than discovering it.

For now, we can merge patches [1] and [2] and look at if there is a case to use venv later. If we plan to use venv for some reason ( we should install all python libraries using pip). But we don't use virtual env for content provider jobs[3].

Rabi plans to cherry-pick [1] at least till Zed.

[0] https://review.opendev.org/c/openstack/python-tripleoclient/+/864647/3/tripleoclient/v2/tripleo_container_image.py
[1] https://review.rdoproject.org/r/c/config/+/46139
[2] https://review.rdoproject.org/r/c/rdo-jobs/+/46141
[3] https://c18ded86635bab17724f-51fb74e788bc28a37082204769947c3c.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/master/tripleo-ci-centos-9-content-provider/3352f61/logs/undercloud/home/zuul/tripleo_container_image_build.sh

We are going with https://review.rdoproject.org/r/c/config/+/46139

One unknown still from me: if this was caused by [1] and that has not yet been backported to stable/zed at [2] (top commit "Pull flake8 from Github" 8b9c3b6bd96fdc452cfe62ad8235abd656796308) then how did we see this issue on zed builds?

From the bug description we have [3]

[1] https://review.opendev.org/c/openstack/python-tripleoclient/+/864647
[2] https://github.com/openstack/python-tripleoclient/commits/stable/zed
[3] https://logserver.rdoproject.org/69/41469/36/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-zed/48b2722/logs/build.log

Thanks ysandeep - as just discussed in irc - to answer comment #8 it is because the zed job is checking out master branch for required-projects (i.e. using master tripleoclient).

Patch at https://review.rdoproject.org/r/c/rdo-jobs/+/46142 will fix it.

Below patches fixed the issue.

https://review.rdoproject.org/r/c/config/+/46139
https://review.rdoproject.org/r/c/rdo-jobs/+/46141