Comment 6 for bug 1006776

Out of curiosity I've further examined the problem. Here my results.

It seems that the used libnss3 only supports these 3 elliptic curves: secp256r1 secp384r1 and secp521r1

(See source package for libnss3 on ubuntu 12.04
   openjdk-6-src/nss-3.13.1.with.ckbi.1.88/mozilla/security/nss/freebl/ecl/ecl-curve.h:

/* mapping between ECCurveName enum and pointers to ECCurveParams */
static const ECCurveParams *ecCurve_map[] = {
        NULL, /* ECCurve_noName */
        NULL, /* ECCurve_NIST_P192 */
        NULL, /* ECCurve_NIST_P224 */
        &ecCurve_NIST_P256, /* ECCurve_NIST_P256 */
        &ecCurve_NIST_P384, /* ECCurve_NIST_P384 */
        &ecCurve_NIST_P521, /* ECCurve_NIST_P521 */
        NULL, /* ECCurve_NIST_K163 */
        NULL, /* ECCurve_NIST_B163 */
…
    all following are NULL too
}

)

But sun.security.ssl.HelloExtensions.isSupported() always returns true (because "fips mode" is false) - for every existing or non existing curve ID.

OpenSSL in ssl client mode suggests curves in the following order: curve names: {sect571r1, sect571k1, secp521r1, sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1, sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2, secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1, secp160r2}

Because HelloExtensions.isSupported() now says true for the first one (Index 14 = sect571r1), this one is being chosen, but as libnss3 does not support it this leads to

-> SECFailure in gf_populate_params in ecdecode.c:182
-> SECFailure with SetError(SEC_ERROR_UNSUPPORTED_ELLIPTIC_CURVE); in EC_FillParams:597
-> rv 304 (CKR_DOMAIN_PARAMS_INVALID) at Java_sun_security_pkcs11_wrapper_PKCS11_C_1GenerateKeyPair:167
167 rv = (*ckpFunctions->C_GenerateKeyPair)(ckSessionHandle, &ckMechanism,
168 ckpPublicKeyAttributes, ckPublicKeyAttributesLength,
169 ckpPrivateKeyAttributes, ckPrivateKeyAttributesLength,
170 ckpPublicKeyHandle, ckpPrivateKeyHandle);

which prouces a PKCS11Exception in sun.security.pkcs11.P11KeyPairGenerator:314
long[] keyIDs = token.p11.C_GenerateKeyPair
                (session.id(), new CK_MECHANISM(mechanism),
                publicKeyTemplate, privateKeyTemplate);

which is being converted into a ProviderException which is catched in the Handshakers DelegatedTask and being remembered als "thrown" until the unwrap() and then thrown.

call stack at this point:
P11KeyPairGenerator.generateKeyPair() line: 314
KeyPairGenerator$Delegate.generateKeyPair() line: 687
ECDHCrypt.<init>(String, SecureRandom) line: 63
ServerHandshaker.setupEphemeralECDHKeys() line: 1204
ServerHandshaker.trySetCipherSuite(CipherSuite) line: 1058
ServerHandshaker.chooseCipherSuite(HandshakeMessage$ClientHello) line: 887

This prevents the response from the server being created and the HandshakeResponse stays in NEEDS_UNWRAP mode as nothing can be sent back to the client and the connection hangs and times out.