Debian
openjdk-7 package

openjdk-6-jdk ssl negotiation incompatibility

Bug #1006776 reported by David Peall
124
This bug affects 26 people
Affects Status Importance Assigned to Milestone
CentOS
Fix Released
Critical
openjdk-6 (Ubuntu)
Confirmed
Undecided
Unassigned
openjdk-7 (Debian)
New
Undecided
Unassigned
openjdk-7 (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Ununtu 12.04 LTS

openjdk-6-jdk:
  Installed: 6b24-1.11.1-4ubuntu3
  Candidate: 6b24-1.11.1-4ubuntu3
  Version table:
 *** 6b24-1.11.1-4ubuntu3 0
        500 http://za.archive.ubuntu.com/ubuntu/ precise-updates/main i386 Packages
        100 /var/lib/dpkg/status
     6b24-1.11.1-4ubuntu2 0
        500 http://za.archive.ubuntu.com/ubuntu/ precise/main i386 Packages

From the OpenSSL client:

openssl version
OpenSSL 1.0.1 14 Mar 2012

openssl s_client -connect localhost:3121
CONNECTED(00000003)
3077671112:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:724:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 226 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

From the OpenJDK6 server (broken):

Allow unsafe renegotiation: true
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
pool-2-thread-4, setSoTimeout(20000) called
pool-2-thread-4, READ: TLSv1 Handshake, length = 221
*** ClientHello, TLSv1.1
RandomCookie: GMT: 1321675259 bytes = { 184, 44, 25, 155, 123, 0, 221, 149, 99, 164, 30, 145, 14, 82, 5, 146, 179, 15, 178, 161, 250, 169, 115, 69, 239, 126, 131, 196 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, Unknown 0xc0:0x22, Unknown 0xc0:0x21, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, Unknown 0x0:0x88, Unknown 0x0:0x87, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, Unknown 0x0:0x84, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, Unknown 0xc0:0x1c, Unknown 0xc0:0x1b, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, Unknown 0xc0:0x1f, Unknown 0xc0:0x1e, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, Unknown 0x0:0x9a, Unknown 0x0:0x99, Unknown 0x0:0x45, Unknown 0x0:0x44, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, Unknown 0x0:0x96, Unknown 0x0:0x41, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_MD5, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5, SSL_RSA_EXPORT_WITH_RC4_40_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 1, 0 }
Extension ec_point_formats, formats: [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
Extension elliptic_curves, curve names: {sect571r1, sect571k1, secp521r1, sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1, sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2, secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1, secp160r2}
Unsupported extension type_35, data:
Unsupported extension type_15, data: 01
***
pool-2-thread-4, handling exception: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DOMAIN_PARAMS_INVALID
pool-2-thread-4, SEND TLSv1 ALERT: fatal, description = internal_error
pool-2-thread-4, WRITE: TLSv1 Alert, length = 2
pool-2-thread-4, called closeSocket()
pool-2-thread-4, IOException in getSession(): javax.net.ssl.SSLException: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DOMAIN_PARAMS_INVALID
Finalizer, called close()
Finalizer, called closeInternal(true)
Finalizer, called close()
Finalizer, called closeInternal(true)
Finalizer, called close()
Finalizer, called closeInternal(true)

From the sun-jdk server(works):

Allow unsafe renegotiation: true
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
pool-2-thread-1, setSoTimeout(20000) called
pool-2-thread-1, READ: TLSv1 Handshake, length = 221
*** ClientHello, TLSv1.1
RandomCookie: GMT: 1321675506 bytes = { 188, 132, 89, 108, 237, 169, 129, 49, 160, 33, 112, 237, 203, 27, 146, 187, 53, 152, 148, 219, 10, 93, 44, 51, 49, 209, 241, 18 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, Unknown 0xc0:0x22, Unknown 0xc0:0x21, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, Unknown 0x0:0x88, Unknown 0x0:0x87, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, Unknown 0x0:0x84, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, Unknown 0xc0:0x1c, Unknown 0xc0:0x1b, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, Unknown 0xc0:0x1f, Unknown 0xc0:0x1e, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, Unknown 0x0:0x9a, Unknown 0x0:0x99, Unknown 0x0:0x45, Unknown 0x0:0x44, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, Unknown 0x0:0x96, Unknown 0x0:0x41, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_MD5, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5, SSL_RSA_EXPORT_WITH_RC4_40_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 1, 0 }
Extension ec_point_formats, formats: [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
Extension elliptic_curves, curve names: {sect571r1, sect571k1, secp521r1, sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1, sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2, secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1, secp160r2}
Unsupported extension type_35, data:
Unsupported extension type_15, data: 01
***
%% Created: [Session-1, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA]
*** ServerHello, TLSv1
RandomCookie: GMT: 1321675506 bytes = { 141, 15, 202, 217, 253, 174, 240, 169, 172, 62, 151, 132, 183, 87, 204, 146, 37, 174, 38, 204, 18, 234, 112, 30, 174, 165, 57, 117 }
Session ID: {79, 199, 43, 242, 167, 217, 237, 76, 85, 242, 195, 126, 53, 209, 252, 103, 58, 71, 185, 6, 181, 52, 206, 70, 75, 13, 117, 143, 21, 183, 5, 142}
Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
***
Cipher suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
*** Certificate chain

See original description
David Peall (dkpeall)
tags: added: precise
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openjdk-6 (Ubuntu):
status: New → Confirmed
Revision history for this message
Christoph W (wech) wrote :

Same problem exists in openjdk-7u3-2.1.1

Launchpad Janitor (janitor)
Changed in openjdk-7 (Ubuntu):
status: New → Confirmed
Christoph W (wech)
affects: openjdk → openjdk-7 (Ubuntu)
Revision history for this message
Christoph W (wech) wrote :

Same as https://bugs.launchpad.net/ubuntu/+source/openjdk-7/+bug/989240

sun.security.pkcs11.wrapper.PKCS11Exception of CKR_DOMAIN_PARAMS_INVALID while creating private

    // Called by ServerHandshaker for ephemeral ECDH
    ECDHCrypt(String curveName, SecureRandom random) {
        try {
            KeyPairGenerator kpg = JsseJce.getKeyPairGenerator("EC");
            ECGenParameterSpec params = new ECGenParameterSpec(curveName);
            kpg.initialize(params, random);
            KeyPair kp = kpg.generateKeyPair(); <<<<<<<< ***BOOOM

I expierience the crash in jetty7 when connecting with libssl>=1.0.0.

It does work fine when I run jetty sun/oracles jdk7u4 and it also worked on ubuntu 8.04 lts with openjdk6b18, but not on ubuntu 10.04 lts with openjdk6b20 or on 12.04 with openjdk6b24 or openjdk7u3.

The ciphersuite chosen in my case is TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 in my case. When I disable all Elliptic Curve cipher suites trough jettys ssl configuration, the problem gets away.

This bug had been fixed for openjdk-6 - 6b18-1.8-0ubuntu1 (see https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/556549), but produced another bug so was undone it seems (https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/580982)

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openjdk-7 (Ubuntu):
status: New → Confirmed
Revision history for this message
Christoph W (wech) wrote :

Out of curiosity I've further examined the problem. Here my results.

It seems that the used libnss3 only supports these 3 elliptic curves: secp256r1 secp384r1 and secp521r1

(See source package for libnss3 on ubuntu 12.04
   openjdk-6-src/nss-3.13.1.with.ckbi.1.88/mozilla/security/nss/freebl/ecl/ecl-curve.h:

/* mapping between ECCurveName enum and pointers to ECCurveParams */
static const ECCurveParams *ecCurve_map[] = {
        NULL, /* ECCurve_noName */
        NULL, /* ECCurve_NIST_P192 */
        NULL, /* ECCurve_NIST_P224 */
        &ecCurve_NIST_P256, /* ECCurve_NIST_P256 */
        &ecCurve_NIST_P384, /* ECCurve_NIST_P384 */
        &ecCurve_NIST_P521, /* ECCurve_NIST_P521 */
        NULL, /* ECCurve_NIST_K163 */
        NULL, /* ECCurve_NIST_B163 */
…
    all following are NULL too
}

)

But sun.security.ssl.HelloExtensions.isSupported() always returns true (because "fips mode" is false) - for every existing or non existing curve ID.

OpenSSL in ssl client mode suggests curves in the following order: curve names: {sect571r1, sect571k1, secp521r1, sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1, sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2, secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1, secp160r2}

Because HelloExtensions.isSupported() now says true for the first one (Index 14 = sect571r1), this one is being chosen, but as libnss3 does not support it this leads to

-> SECFailure in gf_populate_params in ecdecode.c:182
-> SECFailure with SetError(SEC_ERROR_UNSUPPORTED_ELLIPTIC_CURVE); in EC_FillParams:597
-> rv 304 (CKR_DOMAIN_PARAMS_INVALID) at Java_sun_security_pkcs11_wrapper_PKCS11_C_1GenerateKeyPair:167
167 rv = (*ckpFunctions->C_GenerateKeyPair)(ckSessionHandle, &ckMechanism,
168 ckpPublicKeyAttributes, ckPublicKeyAttributesLength,
169 ckpPrivateKeyAttributes, ckPrivateKeyAttributesLength,
170 ckpPublicKeyHandle, ckpPrivateKeyHandle);

which prouces a PKCS11Exception in sun.security.pkcs11.P11KeyPairGenerator:314
long[] keyIDs = token.p11.C_GenerateKeyPair
                (session.id(), new CK_MECHANISM(mechanism),
                publicKeyTemplate, privateKeyTemplate);

which is being converted into a ProviderException which is catched in the Handshakers DelegatedTask and being remembered als "thrown" until the unwrap() and then thrown.

call stack at this point:
P11KeyPairGenerator.generateKeyPair() line: 314
KeyPairGenerator$Delegate.generateKeyPair() line: 687
ECDHCrypt.<init>(String, SecureRandom) line: 63
ServerHandshaker.setupEphemeralECDHKeys() line: 1204
ServerHandshaker.trySetCipherSuite(CipherSuite) line: 1058
ServerHandshaker.chooseCipherSuite(HandshakeMessage$ClientHello) line: 887

This prevents the response from the server being created and the HandshakeResponse stays in NEEDS_UNWRAP mode as nothing can be sent back to the client and the connection hangs and times out.

Revision history for this message
Christoph W (wech) wrote :

*Ouch*

I just realized that it is sufficient to modify /etc/java-6-openjdk/security/java.security so it uses the sun Elliptic Curve impelementation wich is also included in openjdk.

Just change the line
security.provider.9=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg
to
security.provider.9=sun.security.ec.SunEC
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

and move the Sun EC implementation, which provides all curves, in front of the pkcs11 implementation so it is being tried first when creating the key pair.

Revision history for this message
Christoph W (wech) wrote :

hm. This worked on ubuntu 12.04, but not on 10.04 with openjdk-6_6b20-1.9.13 *sigh*

Revision history for this message
Christoph W (wech) wrote :

Hm. Now I get a java.lang.ClassNotFoundException: sun.security.ec.SunEC even on openjdk7. I was sure this worked before.

It seems the SunEC provider was added in jdk7, so no way to get this to work in jdk6, but it should work on openjdk7 at least.
http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#SunEC

The reason it worked on openjdk-6 yesterday was plainly, that I commented out the pkcs11 provider and the SunEC provider is not available so EllipticCurve was disabled completely when negotiating the used cipher.

So a potential workaround for people affected by this problem would be to comment out the line

#security.provider.9=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

But of course this has the potential problem that EC is no longer available at all which might affect other java applications too.

In my opinion it would be best to fix sun.security.ssl.HelloExtensions.isSupported() so it returns only true for secp256r1 secp384r1 and secp521r1 when the SunEC Provider is not available.

I guess somebody else knows better than me how to implement this in a generic way. For example I don't know why the SunEC provider is not included in IceadTea. Is this a licensing issue for oss? For libnss3 it seems to be a patent issue with the other curves from what I've read, but I'm also not sure about this. Otherwise the best solution at all would be to add the missing curves - but I guess there is a reason they were removed.

The most-non-generic but easiest working solution would probably be trough a patch to the openjdk sources for IcedTea which does a hard check for all curves ones implemented in libnss3 and only returns true for them.

Revision history for this message
David Peall (dkpeall) wrote :
Download full text (6.1 KiB)

#
# List of providers and their preference orders (see above):
#
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=com.sun.net.ssl.internal.ssl.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider
security.provider.6=com.sun.security.sasl.Provider
security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.8=sun.security.smartcardio.SunPCSC
security.provider.9=sun.security.ec.SunEC
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

Not working...

Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
matching alias: fmsrns
pool-2-thread-1, called closeSocket()
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
MessageManager, setSoTimeout(20000) called
[Raw read]: length = 5
0000: 16 03 01 00 DD .....
[Raw read]: length = 221
0000: 01 00 00 D9 03 02 4F CD B8 7F 37 09 DE 24 83 80 ......O...7..$..
0010: 94 0D 69 FC 1F 57 91 E8 27 B8 B5 57 4A 2A 1C 9B ..i..W..'..WJ*..
0020: 5D 34 50 CE B4 05 00 00 66 C0 14 C0 0A C0 22 C0 ]4P.....f.....".
0030: 21 00 39 00 38 00 88 00 87 C0 0F C0 05 00 35 00 !.9.8.........5.
0040: 84 C0 12 C0 08 C0 1C C0 1B 00 16 00 13 C0 0D C0 ................
0050: 03 00 0A C0 13 C0 09 C0 1F C0 1E 00 33 00 32 00 ............3.2.
0060: 9A 00 99 00 45 00 44 C0 0E C0 04 00 2F 00 96 00 ....E.D...../...
0070: 41 C0 11 C0 07 C0 0C C0 02 00 05 00 04 00 15 00 A...............
0080: 12 00 09 00 14 00 11 00 08 00 06 00 03 00 FF 02 ................
0090: 01 00 00 49 00 0B 00 04 03 00 01 02 00 0A 00 34 ...I...........4
00A0: 00 32 00 0E 00 0D 00 19 00 0B 00 0C 00 18 00 09 .2..............
00B0: 00 0A 00 16 00 17 00 08 00 06 00 07 00 14 00 15 ................
00C0: 00 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0F ................
00D0: 00 10 00 11 00 23 00 00 00 0F 00 01 01 .....#.......
MessageManager, READ: TLSv1 Handshake, length = 221
*** ClientHello, TLSv1.1
RandomCookie: GMT: 1322039423 bytes = { 55, 9, 222, 36, 131, 128, 148, 13, 105, 252, 31, 87, 145, 232, 39, 184, 181, 87, 74, 42, 28, 155, 93, 52, 80, 206, 180, 5 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, Unknown 0xc0:0x22, Unknown 0xc0:0x21, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, Unknown 0x0:0x88, Unknown 0x0:0x87, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, Unknown 0x0:0x84, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, Unknown 0xc0:0x1c, Unknown 0xc0:0x1b, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, Unknown 0xc0:0x1f, Unknown 0xc0:0x1e, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CB...

Read more...

Revision history for this message
Christoph W (wech) wrote :

yes. I've updated my suggestion above. Please comment out the line like this:

#security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

This completely disables Elliptic Curves so another method is chosen. It seems SunEC is not available in iced tea builds currently.

Revision history for this message
Christoph W (wech) wrote :

btw: the reason it stopped working with openssl1.0 is, that openssl0.9.8 did only send these ciphers:
DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DES-CBC3-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES128-SHA:RC4-SHA:RC4-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC4-MD5

while 1.0 extended them to include EllipticCurve-Chipers:
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SRP-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC4-MD5

(actual output on ubuntu 8.04 and 12.04 of command "openssl ciphers -tls1" )

Revision history for this message
David Peall (dkpeall) wrote :

Removing
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg
Has solved the problem for us for now.

Riccardo Gerosa (rgerosa)
description: updated
Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

Created attachment 815006
SSL debug from tomcat6's catalina.out

Description of problem:
Using RHEL 6.5, tomcat6 and java-1.7.0-openjdk, I get the following exception in the catalina.out:

%% Initialized: [Session-1, SSL_NULL_WITH_NULL_NULL]
matching alias: tomcat
http-8443-1, handling exception: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DOMAIN_PARAMS_INVALID
%% Invalidated: [Session-1, SSL_NULL_WITH_NULL_NULL]

The client gets an SSL error. If I use java-1.6.0 the problem goes away. If I used the java.security and nss.cfg from java-1.60 with java-1.7.0 it works fine as well.

This is *not* a problem with java-1.7.0-1.7.0.60 on Fedora 18 or 19. Looking in dist-git we enabled nss for RHEL 6.5:

%global enable_nss 1

That is the only difference I have found between the JREs.

Version-Release number of selected component (if applicable):

I have tried both version of the jdk with no success.

java-1.7.0-openjdk-1.7.0.40-2.4.2.5.el6.x86_64
java-1.7.0-openjdk-1.7.0.45-2.4.3.0.el6.x86_64
nss-softokn-freebl-3.14.3-9.el6.x86_64
nss-util-3.15.1-3.el6.x86_64
mod_dnssd-0.6-2.el6.x86_64
python-nss-0.13-1.el6.x86_64
openssl-1.0.1e-15.el6.x86_64
openssh-askpass-5.3p1-94.el6.x86_64
openssl-devel-1.0.1e-15.el6.x86_64
openssh-server-5.3p1-94.el6.x86_64
openssh-clients-5.3p1-94.el6.x86_64
nss-3.15.1-15.el6.x86_64
nss-tools-3.15.1-15.el6.x86_64
openssh-5.3p1-94.el6.x86_64
nss-softokn-3.14.3-9.el6.x86_64
nss-sysinit-3.15.1-15.el6.x86_64

How reproducible:
On my RHEL 6.5 guest with the above rpm versions, I can recreate it at will.

Steps to Reproduce:
1. Installed RHEL 6.5
2. Ensure java-1.7.0-openjdk is installed
3. Install Subscription Asset Manager (SAM) 1.3
4. try to connect using rest-client
4a. scl enable ruby193 'irb'
4b. > require 'rest-client'
4c. > RestClient.get("https://localhost:8443/candlepin/status")

Actual results:
client gets an error.

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert internal error
        from /opt/rh/ruby193/root/usr/share/ruby/net/http.rb:800:in `connect'

Tomcat spews out:
%% Initialized: [Session-1, SSL_NULL_WITH_NULL_NULL]
http-8443-1, handling exception: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DOMAIN_PARAMS_INVALID
%% Invalidated: [Session-1, SSL_NULL_WITH_NULL_NULL]

Expected results:
The JSON output from the Candlepin java app. Like I said above if I switch to java-1.6 or use the configs from 1.6 with java-1.7, it works fine.

Additional info:

Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

This came about while investigating BZ 1010111

Revision history for this message
In , dbhole (dbhole-redhat-bugs) wrote :

Assigning to Andrew to take a look.

Revision history for this message
In , dbhole (dbhole-redhat-bugs) wrote :

Hi Jesus, can you try a workaround while we try to fix this issue?

You need to edit /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/security/java.security

In there, change the priorities to make sun.security.provider.Sun #1 and so on i.e. change:

security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
...
...
security.provider.10=sun.security.smartcardio.SunPCSC

to:

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
...
...
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

(In reply to Deepak Bhole from comment #3)
> Hi Jesus, can you try a workaround while we try to fix this issue?
>
> You need to edit
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/security/java.
> security
>
> In there, change the priorities to make sun.security.provider.Sun #1 and so
> on i.e. change:
>
> security.provider.1=sun.security.pkcs11.SunPKCS11
> ${java.home}/lib/security/nss.cfg
> security.provider.2=sun.security.provider.Sun
> security.provider.3=sun.security.rsa.SunRsaSign
> ...
> ...
> security.provider.10=sun.security.smartcardio.SunPCSC
>
> to:
>
> security.provider.1=sun.security.provider.Sun
> security.provider.2=sun.security.rsa.SunRsaSign
> ...
> ...
> security.provider.10=sun.security.pkcs11.SunPKCS11
> ${java.home}/lib/security/nss.cfg

my java.security now has the following:

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

Same error:

%% Initialized: [Session-4, SSL_NULL_WITH_NULL_NULL]
http-8443-2, handling exception: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DOMAIN_PARAMS_INVALID
%% Invalidated: [Session-4, SSL_NULL_WITH_NULL_NULL]

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Yes, CKR_DOMAIN_PARAMS_INVALID is an error coming from NSS, not OpenJDK.

Jesus, what happens if you remove the security.provider.10 line altogether?

Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

Using the following providers in java.security:

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
# the NSS security provider was not enabled for this build; it can be enabled
# if NSS (libnss3) is available on the machine. The nss.cfg file may need
# editing to reflect the location of the NSS installation.
#security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

I no longer get the CKR_DOMAIN_PARAMS_INVALID:

%% Initialized: [Session-4, SSL_NULL_WITH_NULL_NULL]
%% Negotiating: [Session-4, TLS_RSA_WITH_AES_256_CBC_SHA]
*** ServerHello, TLSv1.2

And when I connect with RestClient I get the actual JSON response I expected:

RestClient.get("https://localhost:8443/candlepin/status")
=> "{\"result\":true,\"version\":\"0.8.26\",\"rulesVersion\":\"4.2\",\"release\":\"1\",\"standalone\":true,\"timeUTC\":\"2013-10-25T18:17:16.391+0000\",\"managerCapabilities\":[\"cores\",\"ram\",\"instance_multiplier\",\"derived_product\",\"cert_v3\"],\"rulesSource\":\"DEFAULT\"}"

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Interesting. What is happening here is the following:

1. The client tries to get a crypto provider that supports SSL_NULL_WITH_NULL_NULL.
2. If the NSS provider is not present, no provider responds to this request and the client requests TLS_RSA_WITH_AES_256_CBC_SHA instead.
3. If the NSS provider is present, instead of saying it doesn't support the algorithm it throws an exception which is carried up to the client.

So the NSS provider is giving the wrong response.

Revision history for this message
In , dbhole (dbhole-redhat-bugs) wrote :

So is this a bug in NSS then, or do you mean NSS provider as in OpenJDK code that interfaces with NSS that is misinterpreting what low-level NSS libraries are saying?

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Seems like the latter, but I won't know 100% until I have time to look into this fully.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

I think https://bugzilla.redhat.com/show_bug.cgi?id=1022950 is related, if not the same issue.

This is the difference on Jesus' machine when the PKCS11 NSS provider is enabled and when it isn't:

+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
+TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 TLS_RSA_WITH_AES_256_CBC_SHA256
+TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
+TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 TLS_RSA_WITH_AES_256_CBC_SHA
+TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 TLS_RSA_WITH_AES_128_CBC_SHA256
+TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 TLS_RSA_WITH_AES_128_CBC_SHA
+TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
+TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
+TLS_ECDHE_RSA_WITH_RC4_128_SHA
 SSL_RSA_WITH_RC4_128_SHA
+TLS_ECDH_ECDSA_WITH_RC4_128_SHA
+TLS_ECDH_RSA_WITH_RC4_128_SHA
+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
+TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
 SSL_RSA_WITH_3DES_EDE_CBC_SHA
+TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
 SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
 SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
 SSL_RSA_WITH_RC4_128_MD5
 TLS_EMPTY_RENEGOTIATION_INFO_SCSV
 TLS_DH_anon_WITH_AES_256_CBC_SHA256
+TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 TLS_DH_anon_WITH_AES_256_CBC_SHA
 TLS_DH_anon_WITH_AES_128_CBC_SHA256
+TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 TLS_DH_anon_WITH_AES_128_CBC_SHA
+TLS_ECDH_anon_WITH_RC4_128_SHA
 SSL_DH_anon_WITH_RC4_128_MD5
+TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
 SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
 TLS_RSA_WITH_NULL_SHA256
+TLS_ECDHE_ECDSA_WITH_NULL_SHA
+TLS_ECDHE_RSA_WITH_NULL_SHA
 SSL_RSA_WITH_NULL_SHA
+TLS_ECDH_ECDSA_WITH_NULL_SHA
+TLS_ECDH_RSA_WITH_NULL_SHA
+TLS_ECDH_anon_WITH_NULL_SHA
 SSL_RSA_WITH_NULL_MD5
 SSL_RSA_WITH_DES_CBC_SHA
 SSL_DHE_RSA_WITH_DES_CBC_SHA

So, with it enabled, the SSL connection is trying to use TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 and failing because NSS doesn't actually support it.

I didn't get the ECC algorithms on my local RHEL machine (latest 6.4). Has there been a change in NSS?

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

* Sun May 26 2013 Elio Maldonado <email address hidden> - 3.14.3-5
- Syncup with uptream changes for aes gcm and ecc suiteb
- Enable ecc support for suite b
- Apply several upstream AES GCM fixes
- Use the pristine nss upstream sources with ecc included
- Export NSS_ENABLE_ECC=1 in both the build and the check sections
- Make failed requests for unsupoprted ssl pkcs 11 bypass non fatal
- Resolves: rhbz#882408 - NSS_NO_PKCS11_BYPASS must preserve ABI
- Related: rhbz#918950 - rebase nss to 3.14.3

6.4 is on 3.14.3-4, the change before the above.
6.5 is on 3.15.1-15, so includes the above.

Revision history for this message
In , hkario (hkario-redhat-bugs) wrote :

(In reply to Andrew John Hughes from comment #16)
> I think https://bugzilla.redhat.com/show_bug.cgi?id=1022950 is related, if
> not the same issue.
>
> This is the difference on Jesus' machine when the PKCS11 NSS provider is
> enabled and when it isn't:
>
>[snip]
>
> So, with it enabled, the SSL connection is trying to use
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 and failing because NSS doesn't
> actually support it.
>
> I didn't get the ECC algorithms on my local RHEL machine (latest 6.4). Has
> there been a change in NSS?

yes, NSS in 6.5 introduced support for TLSv1.2 and ECC.
But the support is not complete.

In case of TLSv1.2 two features are not supported:
 * GCM
 * SHA384 as MAC
In case of ECC, only three curves are supported: nistp256, nistp384, nistp521.

so TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 won't work

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Yeah, that's what I just worked out, tracing through the code :)

Can you tell us which of the additions in the list above should work, if any? It looks like we're going to need to find where this list is created and patch out the ones our NSS doesn't support.

We came across an issue with the curves before. Oracle's upstream test cases for this provider expect additional curves. See:

http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=469

Revision history for this message
In , hkario (hkario-redhat-bugs) wrote :

There are basically only four suites that won't work:

> +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
> +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
> +TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
> +TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384

rest of additions should work.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Created attachment 821065
Specify only the three supported curves

The attached patch should resolve the issue.

Looking at the code, I don't think NSS is being used for its TLS 1.2 implementation at all:

"* TLS 1.2 uses a different hash algorithm than 1.0/1.1 for the PRF calculations. As of 2010, there is no PKCS11-level support for TLS 1.2 PRF calculations, and no known OS's have an internal variant we could use. Therefore for TLS 1.2, we are updating JSSE to request different provider algorithms (e.g. "SunTls12Prf"), and currently only SunJCE has these TLS 1.2 algorithms. If we reused the names such as "SunTlsPrf", the PKCS11 providers would need be updated to fail correctly when presented with the wrong version number (via Provider.Service.supportsParameters()), and we would also need to add the appropriate supportsParamters() checks into KeyGenerators (not currently there).In the future, if PKCS11 support is added, we will restructure this."

which is where the mismatch occurs. Instead of probing providers for which curves they support, the OpenJDK TLS code hardcodes a set list and uses that in its implementation of the Hello ECC curves extension.

The fix just cuts that list down to the three NSS curves. I don't know why this wasn't designed so as to use an API to ask providers for the curves they support. We obviously can't add it to 7 now.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Why did you change the assignment?

Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

As Lukas mentioned above, using the java from comment #23, java-1.7.0-openjdk-1.7.0.45-2.4.3.4.el6.x86_64 I get:

candlepin FAIL SSL_connect returned=1 errno=0 state=SSLv3 read server key exchange B: EC lib
candlepin_auth FAIL SSL_connect returned=1 errno=0 state=SSLv3 read server key exchange B: EC lib

This is slightly different than the *old* message which was a generic tlsv1 internal error:

candlepin FAIL SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert internal error
candlepin_auth FAIL SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert internal error

Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

Using the RestClient method I get the following:

# scl enable ruby193 'irb'
irb> require 'rest-client'
irb> RestClient.get("https://localhost:8443/candlepin/status")
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server key exchange B: EC lib

Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

Created attachment 823039
SSL debug from tomcat6 for comment #26

The third attachment contains the SSL debug from tomcat for the comment run on comment #26.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

I'm a little baffled that this:

Extension elliptic_curves, curve names: {sect571r1, sect571k1, secp521r1, sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1, sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2, secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1, secp160r2}

is still present in the output. Is this definitely from the current run? Have both the server and client been restarted?

Revision history for this message
In , jesusr (jesusr-redhat-bugs) wrote :

(In reply to Andrew John Hughes from comment #28)
> I'm a little baffled that this:
>
> Extension elliptic_curves, curve names: {sect571r1, sect571k1, secp521r1,
> sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1,
> sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2,
> secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1,
> secp160r2}
>
> is still present in the output. Is this definitely from the current run?
> Have both the server and client been restarted?

Yeah, I rebooted the machine completely. Same result.

java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.4.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

rpm -q nss
nss-3.15.1-15.el6.x86_64

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Yeah, I don't think that's the cause, just wondering why it shows up.

Once I can upgrade to 6.5, I'll start testing/debugging this locally.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Ok, I have 6.5 now (after updating nearly broke my entire install...) but:

]# yum install katello-headpin-all
Loaded plugins: auto-update-debuginfo, presto, product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
Setting up Install Process
No package katello-headpin-all available.

Is there an extra repo I need for this?

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

# /usr/bin/java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

# scl enable ruby193 'irb'
irb(main):001:0> require 'rest-client'
=> true
irb(main):002:0> RestClient.get("https://localhost:8443/candlepin/status")
=> "{\"result\":true,\"version\":\"0.8.26\",\"rulesVersion\":\"4.2\",\"release\":\"1\",\"standalone\":true,\"timeUTC\":\"2013-11-28T00:00:16.894+0000\",\"managerCapabilities\":[\"cores\",\"ram\",\"instance_multiplier\",\"derived_product\",\"cert_v3\"],\"rulesSource\":\"DEFAULT\"}"

# rpm -q nss
nss-3.15.1-15.el6.x86_64

# rpm -q java-1.7.0-openjdk
java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

I'm not seeing the error.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

I've managed to reproduce it on there too, but I can't see any obvious differences. It's mystifying.

Are both ends of the transmission Java clients? I did notice that lsof shows postgres, sendmail, cups and ssh to have references to a deleted version of NSS.

If anyone has a simpler reproducer, it would be very welcome :)

As a workaround, there is a property we could change the default of and thus turn off ECC. Thoughts?

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Ok, I'm pretty sure I've found the issue here and it's nothing to do with either Java or even NSS.

I can access the URL fine on both machines with wget and curl. The same issue I patched earlier has also been fixed in OpenSSL (used by your ruby client) recently:

* Thu Oct 31 2013 Tomáš Mráz <email address hidden> 1.0.1e-16
- do not advertise ECC curves we do not support
- fix CPU identification on Cyrix CPUs

My box:

Name : openssl Relocations: (not relocatable)
Version : 1.0.1e Vendor: Red Hat, Inc.
Release : 16.el6_5 Build Date: Mon 04 Nov 2013 16:10:09 GMT

Box from Lukas:

Name : openssl Relocations: (not relocatable)
Version : 1.0.1e Vendor: Red Hat, Inc.
Release : 15.el6 Build Date: Fri 27 Sep 2013 10:13:23 EDT

After a yum upgrade of openssl, surprise, surprise...

irb(main):001:0> require 'rest-client'
=> true
irb(main):002:0> RestClient.get("https://localhost:8443/candlepin/status")
=> "{\"result\":true,\"version\":\"0.8.26\",\"rulesVersion\":\"4.2\",\"release\":\"1\",\"standalone\":true,\"timeUTC\":\"2013-12-04T14:06:23.328+0000\",\"managerCapabilities\":[\"cores\",\"ram\",\"instance_multiplier\",\"derived_product\",\"cert_v3\"],\"rulesSource\":\"DEFAULT\"}"

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

i.e. basically your OpenSSL implementation was saying it supports nist512 when it doesn't. Thus, when the Java side sent it data encrypted with a nist512 curve, it failed.

Revision history for this message
In , dbhole (dbhole-redhat-bugs) wrote :

Jesus, can you confirm that everything works now on your end with the new OpenSSL? If so, we will close this issue as NOTABUG.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Deepak, there was still a bug to begin with. It just means our patch did fix it.

Revision history for this message
In , dbhole (dbhole-redhat-bugs) wrote :

(In reply to Andrew John Hughes from comment #40)
> Deepak, there was still a bug to begin with. It just means our patch did
> fix it.

Ah, right, sorry.

Jesus, if this works for you then, can you please close it as CURRENTRELEASE?

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Yes that is correct. The patch changes it to the shorter output to match NSS.

Revision history for this message
In , rhbugzilla.5.nebuchadnezar (rhbugzilla.5.nebuchadnezar-redhat-bugs) wrote :

Could you guys please send the bugfix, or a link to this bugreport to the openssl guys? Your bugfix is quite worthless to the rest of the world if you don't report this bug upstream and keeping the fix to yourselves. I had to disable nss via. the java.security file in order to use mcabber with my jabber server, because I had this exact exception on a debian box with openfire running on top of it.

Revision history for this message
In , dbhole (dbhole-redhat-bugs) wrote :

(In reply to rhbugzilla.5.nebuchadnezar from comment #46)
> Could you guys please send the bugfix, or a link to this bugreport to the
> openssl guys? Your bugfix is quite worthless to the rest of the world if you
> don't report this bug upstream and keeping the fix to yourselves. I had to
> disable nss via. the java.security file in order to use mcabber with my
> jabber server, because I had this exact exception on a debian box with
> openfire running on top of it.

The fix is not applicable to upstream. Upstream does does not remove support for certain Elliptic Curves. That is something specific to RHEL packages.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

(In reply to rhbugzilla.5.nebuchadnezar from comment #46)
> Could you guys please send the bugfix, or a link to this bugreport to the
> openssl guys? Your bugfix is quite worthless to the rest of the world if you
> don't report this bug upstream and keeping the fix to yourselves. I had to
> disable nss via. the java.security file in order to use mcabber with my
> jabber server, because I had this exact exception on a debian box with
> openfire running on top of it.

This bug is not about OpenSSL. You need to comment on their bug report.

Revision history for this message
In , rhbugzilla.5.nebuchadnezar (rhbugzilla.5.nebuchadnezar-redhat-bugs) wrote :

It is clearly about openssl, or nss. At least the same reason why this bug report was opened. The workaround proposed here, worked for openfire too. It is something about the handling of tls 1.2. Openfire can't do anything about it, if the error is somewhere below the jvm, isn't it. Unfortunately I'm not well versed with nss/openssl, so I don't really know to whom I should report this, but I figured, due to the fact that the workaround provided here worked too, I can only conclude that the source of the problem is the same; or at least something very similar. Sorry, if I sounded angry before, but it looked to me like, .. nah don't bother with reporting this upstream, we just patch it ourselves, but whatever; I'm a bit helpless here to whom this should be reported. At least, the workaround helped.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

(In reply to rhbugzilla.5.nebuchadnezar from comment #50)
> It is clearly about openssl, or nss. At least the same reason why this bug
> report was opened. The workaround proposed here, worked for openfire too. It
> is something about the handling of tls 1.2. Openfire can't do anything about
> it, if the error is somewhere below the jvm, isn't it. Unfortunately I'm not
> well versed with nss/openssl, so I don't really know to whom I should report
> this, but I figured, due to the fact that the workaround provided here
> worked too, I can only conclude that the source of the problem is the same;
> or at least something very similar. Sorry, if I sounded angry before, but it
> looked to me like, .. nah don't bother with reporting this upstream, we just
> patch it ourselves, but whatever; I'm a bit helpless here to whom this
> should be reported. At least, the workaround helped.

This is a bug related to java-1.7.0-openjdk, so no it's not "clearly about openssl or nss".

Revision history for this message
In , dbhole (dbhole-redhat-bugs) wrote :

As far as RHEL is concerned, this issue is "resolved", in the sense that we have reverted the use of NSS as the default provider due to other issues with it. Marking it so.

Revision history for this message
Thorsten Glaser (mirabilos) wrote :

Just debugged this on OpenJDK 7, Tomcat 7, Debian wheezy.

The workaround to comment out the line

security.provider.9=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

worked. The line

#security.provider.9=sun.security.ec.SunEC

was already commented out, since this is the nōn-free part AFAICT.

Revision history for this message
In , tg (tg-redhat-bugs) wrote :

This is also tracked on Launchpad:

https://bugs.launchpad.net/centos/+bug/1006776

Thanks for the additional information regarding that this is caused by a change in NSS (not Java™ or so) adding partial/incomplete ECC support.

Revision history for this message
Thorsten Glaser (mirabilos) wrote :

Also tracked in Red Hat: https://bugzilla.redhat.com/show_bug.cgi?id=1022017

The bug is apparently caused by a change in NSS (not Java™) adding partial/incomplete ECC support.

Revision history for this message
Andrew John Hughes (ahughes) wrote :

Some feedback on this bug as I worked on it from the RH side.

* The OpenJDK Java code for elliptic curve support in SSL uses a list of curves based on the in-tree code used by Oracle.
* The actual curves supported by the system version of NSS used on most distributions is much shorter; it's basically three approved NIST curves. I assume this is for legal reasons which I won't try and go into further.
* This bug comes about because of the mismatch between the two. The Java code says it supports one set of curves, but in practice, the underlying provider code doesn't.
* This is true whether the combination of NSS with the PKCS11 provider is used (the Ubuntu setup above and what Red Hat were trying in its bug) or with the SunEC provider (support for this is coming in IcedTea 2.5.0).

The next set of releases (2.4.8, 2.5.0) will include a patch which fixes the list of curves on the Java side to match the list used by system NSS. This bug should then be properly resolved and the above workaround will not be needed.

Looking to the future, the SunEC provider may be a better solution for providing elliptic curve support - it matches what Oracle use in their packages - but doing so with a system NSS will require 3.16.1, in order to expose necessary APIs. Even then, it is necessary to link against a static NSS library at build-time.

Revision history for this message
In , ahughes (ahughes-redhat-bugs) wrote :

Re-opening this as we appear to have regressed on this issue with the java-1.8.0-openjdk package. We now ship the SunEC provider which also uses NSS (see bug 1208307) and the same three curves, but the patch from comment #21 was never forward-ported to the OpenJDK 8 packages. It was already in the OpenJDK 7 packages when SunEC support was added, so it was missed as not being one of the changes we made then.

Revision history for this message
In , gfrankliu (gfrankliu-redhat-bugs) wrote :

I saw upstream tracks this issue in https://bugs.openjdk.java.net/browse/JDK-8160342 and marka as fix version 9.1 [ 18400 ]

Revision history for this message
In , errata-xmlrpc (errata-xmlrpc-redhat-bugs) wrote :

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0571.html

Bug Watch Updater (bug-watch-updater)
Changed in centos:
importance: Unknown → Critical
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.