c9 scen001 and scen004 standalone test are failing - "Non-zero exit code 1 from systemctl start ceph" - selinux enforcing

Looking at the log [1] I see that the mon process starts, but at some point it fails because of:

"""
Dec 06 11:37:59 standalone.localdomain bash[68635]: Error: open /etc/containers/networks/netavark.lock: permission denied
"""

Still investigating the issue, but it's not coming from the Ceph version (which has not been changed).

[1] https://logserver.rdoproject.org/b6/b63515f8d458cd4599a36d0f60a050c09b8fbfce/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/b62e851/logs/undercloud/var/log/extra/journal.txt.gz

<fpantano> rlandy|rover: and it's basically selinux that prevents the ceph container to access to "/etc/containers/networks/netavark.lock"-> https://logserver.rdoproject.org/b6/b63515f8d458cd4599a36d0f60a050c09b8fbfce/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/b62e851/logs/undercloud/var/log/extra/selinux_denials.txt.gz

The root cause of the issue is related to selinux which deny access to /etc/containers/networks/netavark.lock [1].

This means that w/ "setenforce 0" we're able to get the Ceph cluster deployed.

Investigating to see what changed on that front.

[1] https://logserver.rdoproject.org/b6/b63515f8d458cd4599a36d0f60a050c09b8fbfce/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/b62e851/logs/undercloud/var/log/extra/selinux_denials.txt.gz

So two things here:

1 - Jobs are running enforcing on centos 9. iirc, we don't usually do that.
2 - Probably related to https://github.com/containers/container-selinux/issues/198 and https://bugzilla.redhat.com/show_bug.cgi?id=2150283 as owalsh points out

Failing job has:

https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-zed/98c7f12/logs/undercloud/etc/selinux/config.gz

SELINUX=enforcing

selinux-policy.noarch 38.1.2-1.el9 @anaconda

Passing job has:

https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-zed/14991ca/logs/undercloud/etc/selinux/config.gz

SELINUX=permissive

selinux-policy.noarch 38.1.1-1.el9 @anaconda

The c9 node seems to start:

[zuul@node-0003339614 ~]$ getenforce
Enforcing

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/866806

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d87/866020/1/gate/tripleo-ci-centos-9-undercloud-containers/d87a9f8/logs/undercloud/etc/selinux/config - seems that nodes are still permissive in opendev zuul - so this is an RDO zuul node issue.

Wondering why it's trying to access the netavark thing in the first place. Aren't the container running with "net=host" or something like that? Or is the service running on the host now?

I'd be against passing the CI to permissive. It allows to actually find issues before they bite us back in enforcing environments.

@Cedric fine goal to have enforcing but not OK to block the production chain while we do it.

So I think we should go with https://review.opendev.org/c/openstack/tripleo-ci/+/866806 if there are no better suggestions in order to unblock us.

Then if we want to use enforcing then we need to make a plan to work towards it.

noting the test result for /tripleo-ci/+/866806 at [1]

we are sending the workaround to gates

[1] https://review.rdoproject.org/r/c/testproject/+/36254/204#message-1cc1e9ebcb562014a45cd2c9456c1e5e4bcdb0fa

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/866806
Committed: https://opendev.org/openstack/tripleo-ci/commit/e5ea2013f7349b212507ea454b0969b49f9cfc1a
Submitter: "Zuul (22348)"
Branch: master

commit e5ea2013f7349b212507ea454b0969b49f9cfc1a
Author: Ronelle Landy <email address hidden>
Date: Tue Dec 6 16:07:28 2022 -0500

    Ensure CentOS nodes are selinux Permissive

    CentOS 9 nodes are startng with selinux in
    Enforcing mode. This is not the expected
    configuration to run RDO/CentOS tests.

    This patch sets selinux to permissive in pre
    when running on CentOS.

    Change-Id: I14dff0c0bb2d793ef4cd52ddcc2ff5ca4f870b97
    Related-Bug: #1998954

Unclear where/what was setting selinux permissive before, CS9 qcows always had selinux enforcing since ever, and nodepool image history is gone now, only last two are kept in Glance.
In any case, explicit env setup in prep is the best approach.