Ubuntu
openssl package

[UBUNTU 22.04] openssl with ibmca engine configured dumps core when creating a new certificate

  1. Jammy (22.04)
  2. Bug #2023545
Bug #2023545 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
In Progress
High
Skipper Bug Screeners
openssl (Ubuntu)
In Progress
High
Adrien Nader
Jammy
In Progress
High
Adrien Nader
Ubuntu jammy-updates
Lunar
Fix Released
Undecided
Unassigned

Bug Description

=== SRU information ===
[Meta]
This bug is part of a series of four bugs for a single SRU.
The "central" bug with the global information and debdiff is http://pad.lv/2033422

[Impact]
Openssl using an engine dumps core upon certificate creation; other operations are probably affected too. Overall, engines are likely mostly unusable.

[Test plan]
An engine is needed to test the fix and I don't think we have many in the archive. This complicates reproducing the issue. I have been relying on user reports which have been very detailled and helpful.
The issue has also been reported independently and with another engine (devcrypto).
The issue is fixed in openssl 3.0.8 which landed in lunar.

[Where problems could occur]
I don't pretend to understand the lifecycle of providers in openssl3 but the patch is simple and has been widely tested by now, including on ubuntu. Thus, I see little chance an unexpected problem would occur with it.

[Patches]
The patches come directly from upstream and apply cleanly.

https://github.com/openssl/openssl/issues/18578

* https://git.launchpad.net/~adrien-n/ubuntu/+source/openssl/tree/debian/patches/jammy-sru-0001-Release-the-drbg-in-the-global-default-context-befor.patch?h=jammy-sru&id=04ef023920ab08fba214817523fba897527dfff0

=== Original description ===

openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'

---Problem Description---
OpenSSL with ibmca engine configured dumps core when creating a new certificate.

# openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
# openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
Segmentation fault (core dumped)

# journalctl
Jun 07 13:06:08 SYSTEM kernel: User process fault: interruption code 003b ilc:2 in libc.so.6[3ffae080000+1ca000]
Jun 07 13:06:08 SYSTEM kernel: Failing address: 0000000000000000 TEID: 0000000000000800
Jun 07 13:06:08 SYSTEM kernel: Fault in primary space mode while using user ASCE.
Jun 07 13:06:08 SYSTEM kernel: AS:000000009c2941c7 R3:0000000000000024
Jun 07 13:06:08 SYSTEM kernel: CPU: 2 PID: 2344 Comm: openssl Kdump: loaded Not tainted 5.15.0-73-generic #80-Ubuntu
Jun 07 13:06:08 SYSTEM kernel: Hardware name: IBM 3931 A01 703 (z/VM 7.3.0)
Jun 07 13:06:08 SYSTEM kernel: User PSW : 0705000180000000 000003ffae11c708
Jun 07 13:06:08 SYSTEM kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 RI:0 EA:3
Jun 07 13:06:08 SYSTEM kernel: User GPRS: 0000000000000007 000003ffae11c6f0 0000000000000000 000002aa3289f9d0
Jun 07 13:06:08 SYSTEM kernel: 000002aa1825980f 000002aa3289f9d0 0000000000000000 000002aa328a4300
Jun 07 13:06:08 SYSTEM kernel: 000003ffae870720 000003ffae657128 000002aa000003ff 0000000000000000
Jun 07 13:06:08 SYSTEM kernel: 000003ffae24dd10 000003ffae657120 000003ffae437c22 000003ffec2fe000
Jun 07 13:06:08 SYSTEM kernel: User Code: 000003ffae11c6fc: b90400b2 lgr %r11,%r2
                                                      000003ffae11c700: 47000000 bc 0,0
                                                     #000003ffae11c704: b24f00a0 ear %r10,%a0
                                                     >000003ffae11c708: 58102018 l %r1,24(%r2)
                                                      000003ffae11c70c: ebaa0020000d sllg %r10,%r10,32
                                                      000003ffae11c712: b24f00a1 ear %r10,%a1
                                                      000003ffae11c716: 5910a0d0 c %r1,208(%r10)
                                                      000003ffae11c71a: a7840033 brc 8,000003ffae11c780
Jun 07 13:06:08 SYSTEM kernel: Last Breaking-Event-Address:
Jun 07 13:06:08 SYSTEM kernel: [<000003ffae33242c>] 0x3ffae33242c
Jun 07 13:06:08 SYSTEM systemd[1]: Started Process Core Dump (PID 2345/UID 0).
Jun 07 13:06:08 SYSTEM systemd-coredump[2350]: Process 2344 (openssl) of user 0 dumped core.

                                                           Found module linux-vdso64.so.1 with build-id: bcfab8ac8dbd44c758c3c5494e2952db16905d2e
                                                           Found module libica.so.4 with build-id: 0cc5ace50644dfba6d0ecf4f783477cd04a55731
                                                           Found module ibmca.so with build-id: 27daaf0ed1857fdad3761c2b3db21020999eee08
                                                           Found module ld64.so.1 with build-id: 31d4856f0ba9ea058c91a34f4d684ae0fe01964c
                                                           Found module libc.so.6 with build-id: 74250317950da91d3345f258cb2dd12d22c3f2e5
                                                           Found module libcrypto.so.3 with build-id: a27f20e6cf293f214d459530ce2c0b2b52fdbdb4
                                                           Found module libssl.so.3 with build-id: e2c031c3dac06b5ce43bdea022aee7989f78dde4
                                                           Found module openssl with build-id: ed0fe325182e99d135ee6b08e6d90a9d1c42af7f
                                                           Stack trace of thread 2344:
                                                           #0 0x000003ffae11c708 __pthread_rwlock_wrlock_full64 (libc.so.6 + 0x9c708)
                                                           #1 0x000003ffae437c22 CRYPTO_THREAD_write_lock (libcrypto.so.3 + 0x1b7c22)
                                                           #2 0x000003ffae3e3472 ENGINE_finish (libcrypto.so.3 + 0x163472)
                                                           #3 0x000003ffae406844 EVP_CIPHER_CTX_reset (libcrypto.so.3 + 0x186844)
                                                           #4 0x000003ffae40688c EVP_CIPHER_CTX_free (libcrypto.so.3 + 0x18688c)
                                                           #5 0x000003ffae4f903c n/a (libcrypto.so.3 + 0x27903c)
                                                           #6 0x000003ffae40ca98 EVP_RAND_CTX_free (libcrypto.so.3 + 0x18ca98)
                                                           #7 0x000003ffae461a92 n/a (libcrypto.so.3 + 0x1e1a92)
                                                           #8 0x000003ffae430b9c CRYPTO_free_ex_data (libcrypto.so.3 + 0x1b0b9c)
                                                           #9 0x000003ffae4293ca n/a (libcrypto.so.3 + 0x1a93ca)
                                                           #10 0x000003ffae4335e8 OPENSSL_cleanup (libcrypto.so.3 + 0x1b35e8)
                                                           #11 0x000003ffae0cb6cc __run_exit_handlers (libc.so.6 + 0x4b6cc)
                                                           #12 0x000003ffae0cb790 __GI_exit (libc.so.6 + 0x4b790)
                                                           #13 0x000002aa31847c06 main (openssl + 0x47c06)
                                                           #14 0x000003ffae0aa712 __libc_start_call_main (libc.so.6 + 0x2a712)
                                                           #15 0x000003ffae0aa7f0 __libc_start_main_impl (libc.so.6 + 0x2a7f0)
                                                           #16 0x000002aa31848070 n/a (openssl + 0x48070)
Jun 07 13:06:08 SYSTEM systemd[1]: systemd-coredump@12-2345-0.service: Deactivated successfully.
Jun 07 13:06:31 SYSTEM kernel: User process fault: interruption code 003b ilc:2 in libc.so.6[3ffbee00000+1ca000]
Jun 07 13:06:31 SYSTEM kernel: Failing address: 0000000000000000 TEID: 0000000000000800
Jun 07 13:06:31 SYSTEM kernel: Fault in primary space mode while using user ASCE.
Jun 07 13:06:31 SYSTEM kernel: AS:000000009c2941c7 R3:0000000000000024
Jun 07 13:06:31 SYSTEM kernel: CPU: 2 PID: 2356 Comm: openssl Kdump: loaded Not tainted 5.15.0-73-generic #80-Ubuntu
Jun 07 13:06:31 SYSTEM kernel: Hardware name: IBM 3931 A01 703 (z/VM 7.3.0)
Jun 07 13:06:31 SYSTEM kernel: User PSW : 0705000180000000 000003ffbee9c708
Jun 07 13:06:31 SYSTEM kernel: R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:1 AS:0 CC:0 PM:0 RI:0 EA:3
Jun 07 13:06:31 SYSTEM kernel: User GPRS: 0000000000000007 000003ffbee9c6f0 0000000000000000 000002aa176569d0
Jun 07 13:06:31 SYSTEM kernel: 000002aa3dc356c6 000002aa176569d0 0000000000000000 000002aa1765b300
Jun 07 13:06:31 SYSTEM kernel: 000003ffbf5f0720 000003ffbf3d7128 000002aa000003ff 0000000000000000
Jun 07 13:06:31 SYSTEM kernel: 000003ffbefcdd10 000003ffbf3d7120 000003ffbf1b7c22 000003ffc4efdd40
Jun 07 13:06:31 SYSTEM kernel: User Code: 000003ffbee9c6fc: b90400b2 lgr %r11,%r2
                                                      000003ffbee9c700: 47000000 bc 0,0
                                                     #000003ffbee9c704: b24f00a0 ear %r10,%a0
                                                     >000003ffbee9c708: 58102018 l %r1,24(%r2)
                                                      000003ffbee9c70c: ebaa0020000d sllg %r10,%r10,32
                                                      000003ffbee9c712: b24f00a1 ear %r10,%a1
                                                      000003ffbee9c716: 5910a0d0 c %r1,208(%r10)
                                                      000003ffbee9c71a: a7840033 brc 8,000003ffbee9c780
Jun 07 13:06:31 SYSTEM kernel: Last Breaking-Event-Address:
Jun 07 13:06:31 SYSTEM kernel: [<000003ffbf0b242c>] 0x3ffbf0b242c
Jun 07 13:06:32 SYSTEM systemd[1]: Started Process Core Dump (PID 2357/UID 0).
Jun 07 13:06:32 SYSTEM systemd-coredump[2362]: Process 2356 (openssl) of user 0 dumped core.

                                                           Found module linux-vdso64.so.1 with build-id: bcfab8ac8dbd44c758c3c5494e2952db16905d2e
                                                           Found module libica.so.4 with build-id: 0cc5ace50644dfba6d0ecf4f783477cd04a55731
                                                           Found module ibmca.so with build-id: 27daaf0ed1857fdad3761c2b3db21020999eee08
                                                           Found module ld64.so.1 with build-id: 31d4856f0ba9ea058c91a34f4d684ae0fe01964c
                                                           Found module libc.so.6 with build-id: 74250317950da91d3345f258cb2dd12d22c3f2e5
                                                           Found module libcrypto.so.3 with build-id: a27f20e6cf293f214d459530ce2c0b2b52fdbdb4
                                                           Found module libssl.so.3 with build-id: e2c031c3dac06b5ce43bdea022aee7989f78dde4
                                                           Found module openssl with build-id: ed0fe325182e99d135ee6b08e6d90a9d1c42af7f
                                                           Stack trace of thread 2356:
                                                           #0 0x000003ffbee9c708 __pthread_rwlock_wrlock_full64 (libc.so.6 + 0x9c708)
                                                           #1 0x000003ffbf1b7c22 CRYPTO_THREAD_write_lock (libcrypto.so.3 + 0x1b7c22)
                                                           #2 0x000003ffbf163472 ENGINE_finish (libcrypto.so.3 + 0x163472)
                                                           #3 0x000003ffbf186844 EVP_CIPHER_CTX_reset (libcrypto.so.3 + 0x186844)
                                                           #4 0x000003ffbf18688c EVP_CIPHER_CTX_free (libcrypto.so.3 + 0x18688c)
                                                           #5 0x000003ffbf27903c n/a (libcrypto.so.3 + 0x27903c)
                                                           #6 0x000003ffbf18ca98 EVP_RAND_CTX_free (libcrypto.so.3 + 0x18ca98)
                                                           #7 0x000003ffbf1e1a92 n/a (libcrypto.so.3 + 0x1e1a92)
                                                           #8 0x000003ffbf1b0b9c CRYPTO_free_ex_data (libcrypto.so.3 + 0x1b0b9c)
                                                           #9 0x000003ffbf1a93ca n/a (libcrypto.so.3 + 0x1a93ca)
                                                           #10 0x000003ffbf1b35e8 OPENSSL_cleanup (libcrypto.so.3 + 0x1b35e8)
                                                           #11 0x000003ffbee4b6cc __run_exit_handlers (libc.so.6 + 0x4b6cc)
                                                           #12 0x000003ffbee4b790 __GI_exit (libc.so.6 + 0x4b790)
                                                           #13 0x000002aa161c7c06 main (openssl + 0x47c06)
                                                           #14 0x000003ffbee2a712 __libc_start_call_main (libc.so.6 + 0x2a712)
                                                           #15 0x000003ffbee2a7f0 __libc_start_main_impl (libc.so.6 + 0x2a7f0)
                                                           #16 0x000002aa161c8070 n/a (openssl + 0x48070)

Contact Information = <email address hidden> <email address hidden>

---uname output---
Linux SYSTEM 5.15.0-73-generic #80-Ubuntu SMP Mon May 15 15:23:03 UTC 2023 s390x s390x s390x GNU/Linux

Machine Type = Manufacturer: IBM Type: 3931 Model: 703 A01

---Steps to Reproduce---
1. Configure openssl to be used with ibmca engine.
2. run the following command:
    # openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
3. Check the syslog

Userspace tool common name: openssl

The userspace tool has the following bit modes: 64

Userspace rpm: openssl 3.0.2-0ubuntu1.10 s390x

Userspace tool obtained from project website: na

== Comment: #1 - Grgo ===============================================
Further investigations point to this open issue on openssl repository https://github.com/openssl/openssl/issues/18578

The default_algorithms declaration of RAND and others has no effect on the behaviour of this problem.

== Comment: #2 - Ingo - 2023-06-12 06:05:20 =========================
The OpenSSL fix for the mentioned issue https://github.com/openssl/openssl/issues/18578 is commit
https://github.com/openssl/openssl/commit/a88e97fcace01ecf557b207f04328a72df5110df in the master branch.
The corresponding commit for the openssl-3.0 branch is
https://github.com/openssl/openssl/commit/d0f8056c47f7aea40a34815fe459404f14501e81
This commit is included in OpenSSL 3.0.8.

Please include this commit into the OpenSSL package shipped with 22.04 (and later releases).

See original description
bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-202749 severity-high targetmilestone-inin22045
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
affects: linux (Ubuntu) → openssl (Ubuntu)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in openssl (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
importance: Undecided → High
Changed in ubuntu-z-systems:
importance: Undecided → High
tags: added: rls-jj-incoming
Julian Andres Klode (juliank)
tags: added: foundations-todo
removed: rls-jj-incoming
Adrien Nader (adrien-n)
Changed in openssl (Ubuntu):
assignee: nobody → Adrien Nader (adrien-n)
milestone: none → jammy-updates
status: New → In Progress
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → In Progress
Adrien Nader (adrien-n)
Changed in openssl (Ubuntu Jammy):
status: New → In Progress
Changed in openssl (Ubuntu Lunar):
status: New → Fix Released
Changed in openssl (Ubuntu Jammy):
importance: Undecided → High
assignee: nobody → Adrien Nader (adrien-n)
milestone: none → jammy-updates
Changed in openssl (Ubuntu):
milestone: jammy-updates → none
Revision history for this message
Adrien Nader (adrien-n) wrote :

I've created a PPA for Jammy that incorporates the fix mentionned. The details are available at https://launchpad.net/~adrien-n/+archive/ubuntu/openssl-jammy-sru . Could you test it and confirm your issue is solved?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla
Download full text (3.7 KiB)

------- Comment From <email address hidden> 2023-09-14 03:15 EDT-------
First, reproduced the behaviour of the problem as described and encountered the segmentation fault situation as described.
Wrote out the coredump with coredumpctl dump and saw from the call stack the reported problem was reproduced.

The installed libssl3 and openssl versions were:
# dpkg -l libssl3 openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-====================================================
ii libssl3:s390x 3.0.2-0ubuntu1.10 s390x Secure Sockets Layer toolkit - shared libraries
ii openssl 3.0.2-0ubuntu1.10 s390x Secure Sockets Layer toolkit - cryptographic utility

Next, added the ppa repository as instructed.

# apt list --upgradable
Listing... Done
libssl-dev/jammy 3.0.2-0ubuntu1.11~ppa2 s390x [upgradable from: 3.0.2-0ubuntu1.10]
libssl3/jammy 3.0.2-0ubuntu1.11~ppa2 s390x [upgradable from: 3.0.2-0ubuntu1.10]
openssl/jammy 3.0.2-0ubuntu1.11~ppa2 s390x [upgradable from: 3.0.2-0ubuntu1.10]

Successfully updated the three required packages to the version provided by the ppa package:
# apt upgrade libssl3 openssl libssl-dev
The following packages will be upgraded:
libssl-dev libssl3 openssl
3 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.
Need to get 5,038 kB of archives.
After this operation, 5,120 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
...

Cross-checked the libssl3 and openssl packages had been successfully upgraded:
# dpkg -l libssl3 openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-======================-============-====================================================
ii libssl3:s390x 3.0.2-0ubuntu1.11~ppa2 s390x Secure Sockets Layer toolkit - shared libraries
ii openssl 3.0.2-0ubuntu1.11~ppa2 s390x Secure Sockets Layer toolkit - cryptographic utility

Next checked the ibmca engine is still enabled, and subsequently was able to run the key generation request without encountering the segmentation fault.
Thanks for providing the fix.
Please integrate.

Thanks.

# openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
# openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
# openssl req -new -newkey rsa:2048 -x509 -sha256 -nodes -out __cert.pem -keyout __key.pem --subj '/CN=US'
..+..+.+......+...+.....+.+..+...+.........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+..+.+.....+.......+........+...+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.....................+.........+...+..+...+......+..........+.....+......+.+..+......+.+....................+...+...+.............+....

Read more...

Revision history for this message
Adrien Nader (adrien-n) wrote :

Thanks a lot for taking the time to test and provide feedback. I'll continue with the SRU process; landing will probably take a few weeks.

Adrien Nader (adrien-n)
description: updated
Adrien Nader (adrien-n)
description: updated
Adrien Nader (adrien-n)
description: updated
Adrien Nader (adrien-n)
description: updated
Adrien Nader (adrien-n)
tags: removed: foundations-todo
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Hi Adrien,

You subscribed ubuntu-sponsors, do you have any debdiff or MP to be reviewed? Sorry, I did not find anything ready to review. Without that I believe the best way is to unsubscribe ubuntu-sponsors until there is something ready to be uploaded.

Revision history for this message
Adrien Nader (adrien-n) wrote :

Hi Lucas,

Sorry, this is part of an SRU with 4 patches but that we've decided to hold back for a bit (a few days after the current release). I've removed ubuntu-sponsors from the "main" LP bug (link near the top of the bug report) but not from the others. I'll do it now and I think maybe it's better to only add ~ubuntu-sponsors to that main ticket.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.