OpenSSL 3 performance regression

We discussed this elsewhere but it's worth updating this ticket too.

22.04 and 24.04 will most likely both be on openssl 3.0. It seems unlikely there is another LTS in time for 24.04.

There are several tickets upstream about performance regression and I'm not aware of a few specific and isolated ones that reclaim that performance (I'm not even sure HEAD has reclaimed it yet). The more time passes by, the more difficult backports will be, especially since some of the changes seem to be related to the architecture.

Overall I'm not very confident we can tackle this.

Upstream has an umbrella bug that covers this and other associated performance related problems:

https://github.com/openssl/openssl/issues/17627

I'm switching to bug to Confirmed but I still don't really know if we're going to be able to address this while on 3.0.

I'll try to discuss with at least one person involved in September but I also really wish there's a new LTS for 24.04.

The perf regression is easy to demonstrate using the 'main.py' script attached to this bug, also available here: https://pastebin.ubuntu.com/p/XzV766BpHd/

Some results -

ubuntu@focal-vm:~$ /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 20.04.6 LTS
Python Version: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
OpenSSL Version: OpenSSL 1.1.1f 31 Mar 2020
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 1.65 s

root@jammy-vm:~# /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.0.2 15 Mar 2022
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 5.91 s

Both the above tests were done on a 1core/1G VM. Interestingly, when adding resources it exacerbates the problem, example using 8core/4G VMs (same hardware):

ubuntu@sslfocal-vm1:~$ /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 20.04.6 LTS
Python Version: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
OpenSSL Version: OpenSSL 1.1.1f 31 Mar 2020
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 1.77 s

ubuntu@ssljammy-vm1:~$ /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.0.2 15 Mar 2022
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 12.69 s

Also, when replacing the distro libssl and libcrypto with ones built from upstream master, there is a massive improvement, close to 1.1 perf:

1core/1G:
root@jammy-vm:~# /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.2.0-dev
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 2.15 s
root@jammy-vm:~#

8core/4G:
root@ssljammy-vm1:~# /usr/bin/time -f "\n\nElapsed: %e s" python3 main.py
Distro: Ubuntu 22.04.2 LTS
Python Version: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
OpenSSL Version: OpenSSL 3.2.0-dev
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Elapsed: 1.87 s
root@ssljammy-vm1:~#

Script for testing OpenSSL performance, 100 requests using 30 workers.

Rafael, could you do these tests on Lunar or Mantic too?

Hi Adrien, see results below - lunar and mantic show similar performance, both a significant improvement over jammy. All tests were run using same underlying hardware, 8vcpu/4G VMs.

ubuntu@openssl-jammy:~/openssl_python_ubuntu$ time python3 main.py
Distro: Ubuntu 22.04.3 LTS
Python Version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
OpenSSL Version: OpenSSL 3.0.2 15 Mar 2022
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m12.917s
user 0m21.726s
sys 0m10.047s
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 13.53 User: 22.79
Real: 13.59 User: 22.64
Real: 12.62 User: 22.84
Real: 12.48 User: 21.97
Real: 11.96 User: 21.54
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

ubuntu@openssl-lunar:~/openssl_python_ubuntu$ time python3 main.py
Distro: Ubuntu 23.04
Python Version: 3.11.2 (main, May 30 2023, 17:45:26) [GCC 12.2.0]
OpenSSL Version: OpenSSL 3.0.8 7 Feb 2023
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m2.492s
user 0m13.588s
sys 0m0.150s
ubuntu@openssl-lunar:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 2.36 User: 13.56
Real: 2.68 User: 14.76
Real: 2.55 User: 13.60
Real: 2.81 User: 15.37
Real: 2.51 User: 13.16
ubuntu@openssl-lunar:~/openssl_python_ubuntu$

ubuntu@openssl-mantic:~/openssl_python_ubuntu$ time python3 main.py
Distro: Ubuntu Mantic Minotaur (development branch)
Python Version: 3.11.4 (main, Jun 7 2023, 10:13:09) [GCC 12.3.0]
OpenSSL Version: OpenSSL 3.0.10 1 Aug 2023
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
real 0m2.324s
user 0m14.769s
sys 0m0.147s
ubuntu@openssl-mantic:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 2.63 User: 14.47
Real: 2.57 User: 14.82
Real: 2.27 User: 13.22
Real: 2.39 User: 12.82
Real: 2.71 User: 15.10
ubuntu@openssl-mantic:~/openssl_python_ubuntu$

Tested some additional patches, including [1] (PR linked in bug desc) plus [2] and [3]. PPAs for each at [4], [5] and [6]. All tests on same hardware and VM 8vcpu/4G.

Baseline:
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 13.62 User: 24.44
Real: 13.84 User: 22.89
Real: 14.12 User: 23.71
Real: 13.72 User: 23.94
Real: 13.42 User: 24.31
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 3.20 User: 13.89
Real: 2.43 User: 15.29
Real: 2.72 User: 14.88
Real: 2.77 User: 14.56
Real: 2.43 User: 15.46
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1] + PR#17881[2]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 2.02 User: 7.92
Real: 1.93 User: 7.87
Real: 1.89 User: 7.88
Real: 2.09 User: 8.46
Real: 1.89 User: 7.80
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1] + PR#17921[3]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 1.64 User: 5.30
Real: 1.74 User: 6.08
Real: 1.74 User: 5.80
Real: 1.69 User: 6.50
Real: 1.70 User: 5.68
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

PR#18151[1] + PR#17881[2] + PR#17921[3]
ubuntu@openssl-jammy:~/openssl_python_ubuntu$ for i in {0..4};do /usr/bin/time -f "Real: %e User: %U" python3 main.py > /dev/null;done
Real: 1.55 User: 3.79
Real: 1.64 User: 3.78
Real: 1.86 User: 4.36
Real: 1.73 User: 3.81
Real: 1.81 User: 3.72
ubuntu@openssl-jammy:~/openssl_python_ubuntu$

[1] https://github.com/openssl/openssl/pull/18151
[2] https://github.com/openssl/openssl/pull/17881
[3] https://github.com/openssl/openssl/pull/17921
[4] https://launchpad.net/~rafael.lopez/+archive/ubuntu/lp2009544
[5] https://launchpad.net/~rafael.lopez/+archive/ubuntu/lp2009544-2
[6] https://launchpad.net/~rafael.lopez/+archive/ubuntu/lp2009544-3