User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0

Steps to reproduce:

When compiling nss with LTO enabled (gcc -flto) on s390x the resulting libnss3 is not fully functional. I noticed this as the build causes a regression in the dogtag-pki tests which are part of the dogtag-pki Ubuntu package.

Newer releases of Ubuntu enable LTO by default when building packages. This specific issue will be worked around by disabling the optimizations specifically for this package and on s390x, however the problem is worth investigating upstream.

The error printout doesn't immediately point to optimization issues, however this is always reproducible, and reliably goes away by turning LTO off.

Steps to reproduce:
 - Build nss on s390x with LTO enabled.
 - Install dogtag-pki and ensure it uses the just built libnss3.
 - Exercise the following tests: https://salsa.debian.org/freeipa-team/dogtag-pki/-/blob/master/debian/tests/pkispawn.


Actual results:

The tests fail:

autopkgtest [09:34:17]: test pkispawn: [-----------------------
>>>> IP address is 10.226.183.135
>>>> Hostname was: 
>>>> /etc/hosts now has:
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.226.183.135 autopkgtest.debci autopkgtest
Starting installation...
Completed installation for pki-tomcat
Notice: Trust flag u is set automatically if the private key is present.
/usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for autopkgtest.debci has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(
ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
    cert = deployer.setup_cert(client, tag)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
    return client.setupCert(request)
  File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
    response = self.connection.post(
  File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)

Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-ca-spawn.20210723093512.log
Installing CA into /var/lib/pki/pki-tomcat.

Installation failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

>>>> CA spawn failed:
2021-07-23 09:35:38 ERROR: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  File "/usr/lib/python3/dist-packages/pki/server/pkispawn.py", line 575, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/scriptlets/configuration.py", line 995, in spawn
    cert = deployer.setup_cert(client, tag)
  File "/usr/lib/python3/dist-packages/pki/server/deployment/__init__.py", line 355, in setup_cert
    return client.setupCert(request)
  File "/usr/lib/python3/dist-packages/pki/system.py", line 389, in setupCert
    response = self.connection.post(
  File "/usr/lib/python3/dist-packages/pki/client.py", line 55, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pki/client.py", line 293, in post
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)

autopkgtest [09:35:38]: test pkispawn: -----------------------]
autopkgtest [09:35:39]: test pkispawn:  - - - - - - - - - - results - - - - - - - - - -
pkispawn             FAIL non-zero exit status 1
autopkgtest [09:35:39]: @@@@@@@@@@@@@@@@@@@@ summary
pkispawn             FAIL non-zero exit status 1



Expected results:

The tests all pass:

autopkgtest [22:28:02]: test pkispawn: [-----------------------
>>>> IP address is 10.226.183.148
>>>> Hostname was: 
>>>> /etc/hosts now has:
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.226.183.148 autopkgtest.debci autopkgtest
Starting installation...
Completed installation for pki-tomcat
Notice: Trust flag u is set automatically if the private key is present.
/usr/lib/python3/dist-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for autopkgtest.debci has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-ca-spawn.20210722222833.log
Installing CA into /var/lib/pki/pki-tomcat.

    ==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

      Administrator's username:             caadmin
      Administrator's PKCS #12 file:
            /root/.dogtag/pki-tomcat/ca_admin_cert.p12

      To check the status of the subsystem:
            systemctl status <email address hidden>

      To restart the subsystem:
            systemctl restart <email address hidden>

      The URL for the subsystem is:
            https://autopkgtest.debci:8443/ca

      PKI instances will be enabled upon system boot

    ==========================================================================

WARNING: Directory already exists: /etc/pki/pki-tomcat
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-kra-spawn.20210722222939.log
Installing KRA into /var/lib/pki/pki-tomcat.

    ==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

      Administrator's username:             kraadmin

      To check the status of the subsystem:
            systemctl status <email address hidden>

      To restart the subsystem:
            systemctl restart <email address hidden>

      The URL for the subsystem is:
            https://autopkgtest.debci:8443/kra

      PKI instances will be enabled upon system boot

    ==========================================================================

WARNING: Directory already exists: /etc/pki/pki-tomcat
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-ocsp-spawn.20210722223039.log
Installing OCSP into /var/lib/pki/pki-tomcat.

    ==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

      Administrator's username:             ocspadmin

      To check the status of the subsystem:
            systemctl status <email address hidden>

      To restart the subsystem:
            systemctl restart <email address hidden>

      The URL for the subsystem is:
            https://autopkgtest.debci:8443/ocsp

      PKI instances will be enabled upon system boot

    ==========================================================================

WARNING: Directory already exists: /etc/pki/pki-tomcat
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Loading deployment configuration from debian/tests/deploy.cfg.
Installation log: /var/log/pki/pki-tks-spawn.20210722223141.log
Installing TKS into /var/lib/pki/pki-tomcat.

    ==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

      Administrator's username:             tksadmin

      To check the status of the subsystem:
            systemctl status <email address hidden>

      To restart the subsystem:
            systemctl restart <email address hidden>

      The URL for the subsystem is:
            https://autopkgtest.debci:8443/tks

      PKI instances will be enabled upon system boot

    ==========================================================================

Loading deployment configuration from /var/lib/pki/pki-tomcat/tks/registry/tks/deployment.cfg.
Uninstallation log: /var/log/pki/pki-tks-destroy.20210722223248.log
Uninstalling TKS from /var/lib/pki/pki-tomcat.

Uninstallation complete.
WARNING: this 'OCSP' entry will NOT be deleted from security domain 'debci Security Domain'!
WARNING: security domain 'debci Security Domain' may be offline or unreachable!
ERROR: subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=OCSP&list=ocspList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.!
Loading deployment configuration from /var/lib/pki/pki-tomcat/ocsp/registry/ocsp/deployment.cfg.
Uninstallation log: /var/log/pki/pki-ocsp-destroy.20210722223255.log
Uninstalling OCSP from /var/lib/pki/pki-tomcat.

Uninstallation complete.
ERROR: unable to access security domain. Continuing .. HTTPSConnectionPool(host='autopkgtest.debci', port=8443): Max retries exceeded with url: /ca/rest/securityDomain/domainInfo (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x3ff8af5a2b0>: Failed to establish a new connection: [Errno 111] Connection refused')) 
WARNING: this 'KRA' entry will NOT be deleted from security domain 'debci Security Domain'!
WARNING: security domain 'debci Security Domain' may be offline or unreachable!
ERROR: subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=KRA&list=kraList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.!
Loading deployment configuration from /var/lib/pki/pki-tomcat/kra/registry/kra/deployment.cfg.
Uninstallation log: /var/log/pki/pki-kra-destroy.20210722223257.log
Uninstalling KRA from /var/lib/pki/pki-tomcat.

Uninstallation complete.
WARNING: this 'CA' entry will NOT be deleted from security domain 'debci Security Domain'!
WARNING: security domain 'debci Security Domain' may be offline or unreachable!
ERROR: subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-tomcat', '-p', '~EGeO^i!Ai4^', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=CA&list=caList&host=autopkgtest.debci&sport=8443&ncsport=8443&adminsport=8443&agentsport=8443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'autopkgtest.debci:8443']' returned non-zero exit status 6.!
Loading deployment configuration from /var/lib/pki/pki-tomcat/ca/registry/ca/deployment.cfg.
Uninstallation log: /var/log/pki/pki-ca-destroy.20210722223258.log
Uninstalling CA from /var/lib/pki/pki-tomcat.

Uninstallation complete.
>>>> All done!
autopkgtest [22:32:59]: test pkispawn: -----------------------]
autopkgtest [22:33:00]: test pkispawn:  - - - - - - - - - - results - - - - - - - - - -
pkispawn             PASS
autopkgtest [22:33:00]: @@@@@@@@@@@@@@@@@@@@ summary
pkispawn             PASS