[FIPS] Standalone deploy failing with: "Error in GnuTLS initialization: Error while performing self checks"

Bug #1984237 reported by Douglas Viroel
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

Standalone FIPS jobs are failing since 08-10:
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-standalone-fips

with the following error:

FATAL | Run init bundle puppet on the host for mysql | standalone | error={"changed": false, "cmd": "puppet apply --detailed-exitcodes --summarize --color=false --modulepath '/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules' --tags 'pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation' -e '[\"Mysql_datadir\", \"Mysql_user\", \"Mysql_database\", \"Mysql_grant\", \"Mysql_plugin\"].each |String $val| { noop_resource($val) }; include ::tripleo::profile::base::pacemaker; include ::tripleo::profile::pacemaker::database::mysql_bundle'\n", "delta": "0:08:17.557003", "end": "2022-08-10 10:09:41.167441", "failed_when_result": true, "msg": "non-zero return code", "rc": 4, "start": "2022-08-10 10:01:23.610438", "stderr": "Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5\n (file: /etc/puppet/hiera.yaml)\nWarning: Undefined variable '::deploy_config_name'; \n (file & line not available)\nError in GnuTLS initialization: Error while performing self checks.\nWarning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.16/deprecated_language.html\n (file & line not available)\nError in GnuTLS initialization: Error while performing self checks.\nDeprecation Warning: This command is deprecated and will be removed. Please use 'pcs property config' instead.\nError: unable to get crm_config\nError in GnuTLS initialization: Error while performing self checks.\nCould not connect to the CIB: Update does not conform to the configured schema\nInit failed, could not perform requested operations\n\nError: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20220810-63608-vrr9bv property set stonith-enabled=false failed: Error: unable to get cib. Too many tries\n
...

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_652/852239/2/check/tripleo-ci-centos-9-standalone-fips/6528969/logs/undercloud/home/zuul/standalone_deploy.log

Comparing last success and first failed jobs we have:
Passing job[1]:
 - gnutls.x86_64 3.7.3-10.el9 @baseos
Failing job[2]:
 - gnutls.x86_64 3.7.6-4.el9 @baseos

[1] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9bc/850458/14/check/tripleo-ci-centos-9-standalone-fips/9bc1b7f/logs/undercloud/var/log/extra/package-list-installed.txt
[2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_652/852239/2/check/tripleo-ci-centos-9-standalone-fips/6528969/logs/undercloud/var/log/extra/package-list-installed.txt

Douglas Viroel (dviroel)
tags: added: promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/852914

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/852914
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/2d2fbe3ba6ca1edb6b2b31d587ef4cf6054fb0fb
Submitter: "Zuul (22348)"
Branch: master

commit 2d2fbe3ba6ca1edb6b2b31d587ef4cf6054fb0fb
Author: Douglas Viroel <email address hidden>
Date: Thu Aug 11 15:42:07 2022 -0300

    [FIPS] Exclude gnutls 3.7.6 which is failing on fips

    Excludes gnutls 3.7.6 package from baseos repo, until
    fips related issue is solved.

    Related-Bug: #1984237
    Change-Id: Ie5eb9d5fde53916f11d371f1014b1aff966cc32c

Revision history for this message
Damien Ciabrini (dciabrin) wrote :

As to why GnuTLS in involved:

The failure happening during ansible task "Run init bundle puppet on the host for mysql" is due to several factors.

1. This tasks essentially runs puppet on the host like the following:

   # puppet apply --debug --verbose --detailed-exitcodes --summarize --tags 'pacemaker::property' -e 'noop_resource("Mysql_datadir"); include ::tripleo::profile::base::pacemaker'

The resource we're noop'ing tells puppet to load puppet-mysql, which - as a side effects - configures LD_LIBRARY_PATH during the puppet run.

2. When puppet-pacemaker runs a subshell to inspect the state of the pacemaker cluster, the environment of the subshell is basically:

   # LD_LIBRARY_PATH=:/usr/lib:/usr/lib64:/opt/rh/rh-mysql56/root/usr/lib:/opt/rh/rh-mysql56/root/usr/lib64:/opt/rh/rh-mysql57/root/usr/lib:/opt/rh/rh-mysql57/root/usr/lib64:/opt/rh/rh-mysql80/root/usr/lib:/opt/rh/rh-mysql80/root/usr/lib64:/opt/rh/rh-mariadb100/root/usr/lib:/opt/rh/rh-mariadb100/root/usr/lib64:/opt/rh/rh-mariadb101/root/usr/lib:/opt/rh/rh-mariadb101/root/usr/lib64:/opt/rh/rh-mariadb102/root/usr/lib:/opt/rh/rh-mariadb102/root/usr/lib64:/opt/rh/rh-mariadb103/root/usr/lib:/opt/rh/rh-mariadb103/root/usr/lib64:/opt/rh/mysql55/root/usr/lib:/opt/rh/mysql55/root/usr/lib64:/opt/rh/mariadb55/root/usr/lib:/opt/rh/mariadb55/root/usr/lib64:/usr/mysql/5.5/lib:/usr/mysql/5.5/lib64:/usr/mysql/5.6/lib:/usr/mysql/5.6/lib64:/usr/mysql/5.7/lib:/usr/mysql/5.7/lib64 crm_node -l
   Error in GnuTLS initialization: Error while performing self checks.
   1 standalone member

3. While the call to crm_node succeeds, there's additional output on stdout, and a later subshell ran by puppet-pacemaker cannot parse the output it expects:

   # LD_LIBRARY_PATH=/usr/lib64:<...puppet-mysql-libs...> cibadmin -Q --scope crm_config
   Error in GnuTLS initialization: Error while performing self checks.
   <crm_config>
     <cluster_property_set id="cib-bootstrap-options">
       <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/>
       <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.4-2.el9-dc6eb4362e"/>
       <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
       <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="tripleo_cluster"/>
       <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>
     </cluster_property_set>
   </crm_config>

4. This bad parsing ultimately makes the puppet run fail.

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Marios Andreou (marios-b) wrote :

chkumar posted https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582/ with this as related-bug

Revert "[FIPS] Exclude gnutls 3.7.6 which is failing on fips"

Revision history for this message
chandan kumar (chkumar246) wrote :

gnutls 3.7.6 package is no longer exists and is breaking the content provider
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider
with following error
```
No package gnutls-3.7.3-10.el9 available.", "2022-10-21 03:06:25 | Packages for argument containers-common-1-40.el9 available, but not installed.", "2022-10-21 03:06:25 | Error: No packages marked for downgrade."]}
```

As a proposed fix, revert the pinning of gnutls
https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582 will unblock the gate.

Revision history for this message
Marios Andreou (marios-b) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/ac6d97ab256ce37157b1cd649be962398936966a
Submitter: "Zuul (22348)"
Branch: master

commit ac6d97ab256ce37157b1cd649be962398936966a
Author: chandan kumar <email address hidden>
Date: Fri Sep 9 09:37:55 2022 +0000

    Revert "[FIPS] Exclude gnutls 3.7.6 which is failing on fips"

    This reverts commit 2d2fbe3ba6ca1edb6b2b31d587ef4cf6054fb0fb.

    Reason for revert: this version is no longer available
    Fixed version gnutls-3.7.6-11.el9 is now published:
    https://bugzilla.redhat.com/show_bug.cgi?id=2119822#c7

    Note: It is not a clean revert
    The content provider jobs are failing with following error[1]:
    ```
    Packages for argument containers-common-1-40.el9 available, but not installed.
    ```
    It modifies the downgrade command to downgrade a containers-common
    package only if it is installed.

    [1]. https://zuul.opendev.org/t/openstack/build/75f028ea950349c9a2653f6d2aa2a157/log/logs/undercloud/home/zuul/repo_setup.log#220

    Related-Bug: #1984237

    Change-Id: I888b446518f30b3f63f9f566c4341a0ad289b51d

Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote :

The fix has been merged.

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
Douglas Viroel (dviroel) wrote :
Changed in tripleo:
status: Fix Released → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878524

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/878654

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/884544
Committed: https://opendev.org/openstack/tripleo-common/commit/a27e9268aaba5e9301f1653c0cb45eca0d8598ef
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit a27e9268aaba5e9301f1653c0cb45eca0d8598ef
Author: Douglas Viroel <email address hidden>
Date: Tue May 30 14:33:07 2023 +0000

    Always install nettle and gnutls in CentOS-Stream-9

    This patch update previous workaround to always update
    both "nettle" and "gnutls" packages in base container.
    This solution guarantee that we don't have any of these
    packages from ubi repositories, avoiding GNUTLS FIPS
    validation issues. The fix should always install the latest
    package version from CentOS repos and should not fail even
    if some container build needs to downgrade gnutls version.

    This reverts commit 5fc8ba5c8ad49b08c5147fc3ba59563c9f7d27ed:
      "[FIPS] Install nettle-3.8-3.el9 in tcib base container"
    Reason for revert: new ubi release

    Closes-Bug: #1984237

    Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart/+/884774

    Change-Id: Icb3519879bfb44536e5a8812e39a50ea61d61595

tags: added: in-stable-wallaby
Douglas Viroel (dviroel)
tags: added: fips
Revision history for this message
Douglas Viroel (dviroel) wrote :
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.