OpenSSL

12.04/openssl refusing some verisign certified sites

Bug #1014640 reported by SeanBoran
286
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenSSL
Confirmed
Unknown
ca-certificates (Ubuntu)
Fix Released
Undecided
Unassigned
openssl (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

Summary: SSL refuses to work with some https sites on both 12.04, 13.04, 13.10, for fresh and updated installations. It is an issue with OpenSSL's handling of certificates..

FIX:
Fixed in Ubuntu 14.04 apparently.
Openssl upstream, see http://rt.openssl.org/Ticket/Display.html?id=2732

WORKAROUND:
1) Copy the Root CA from Symantec's website https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR1556
2) Paste the contents into a file under "/usr/local/share/ca-certificates/" and Update:
$ sudo vi /usr/local/share/ca-certificates/<anyname>.crt
$ sudo update-ca-certificates

# You should see output similar to this:
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.

---- Original post ----
After upgrading a 10.04 server to 12.04, SSL refuses to work with some sites.
On 10.04,
curl -v https://cs.directnet.com/dn/c/cls/auth?language=de
works fine, on 12.04 it says:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

This happens on some very well know bank sites , another example is https://postfinance.ch.
Hence I think

Analysis:
- test on an 10.04 upgraded to 12.04 and also a 12.04 fresh server installation
- curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
- Calling ssl directly:
openssl s_client -host cs.directnet.com -port 443
 says "self signed certificate in certificate chain", and the chain shown is:

Certificate chain
 0 s:/1.3.6.1.4.1.311.60.2.1.3=CH/businessCategory=Private Organization/serialNumber=CH-020.3.906.075-9/C=CH/postalCode=8001/ST=Zuerich/L=Zuerich/street=Paradeplatz 8/O=Credit Suisse Group AG/CN=cs.directnet.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
 3 s:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority

Now there are lots of certificates in /usr/share/ca-certificates/mozilla (148 of them, there were 123 in Lucid 10.04).

Search the existing openssl/12.04 issues I came across ciper issues, but didnt' notice a bus for certs.
Since this affects well know sites it would seems to be quite an important issue?

See original description
Revision history for this message
SeanBoran (sean-boran) wrote :

I found a fix:
copy Verisign_Class_3_Public_Primary_Certification_Authority.crt from a lucid/10.04 system (in /usr/share/ca-certificates/mozilla)

Sounds a bit whacky, but invalid certs have been delivered with 12.04?

I did a diff, the changes between 10.04 and 12.04 in that directory for verisign are:
New:
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.crt
VeriSign_Universal_Root_Certification_Authority.crt

Changed:
Verisign_Class_1_Public_Primary_Certification_Authority.crt
Verisign_Class_3_Public_Primary_Certification_Authority.crt

Revision history for this message
Peter (peter-md) wrote :
Download full text (4.7 KiB)

There does appear to be an issue with 12.04.

From a 10.04 server:

=========================================================================
# strace -o /tmp/foo.out curl -Iv https://test.sagepay.com
* About to connect() to test.sagepay.com port 443 (#0)
* Trying 195.170.169.8... connected
* Connected to test.sagepay.com (195.170.169.8) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* subject: 1.3.6.1.4.1.311.60.2.1.3=GB; 2.5.4.15=Private Organization; serialNumber=01045967; C=GB; ST=TYNE AND WEAR; L=Newcastle Upon Tyne; O=Sage (UK) Limited; OU=Sage; OU=Terms of use at www.verisign.co.uk/rpa (c)05; OU=Authenticated by VeriSign; OU=Member, VeriS
* start date: 2011-02-01 00:00:00 GMT
* expire date: 2013-01-31 23:59:59 GMT
* common name: test.sagepay.com (matched)
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)06; CN=VeriSign Class 3 Extended Validation SSL SGC CA
* SSL certificate verify ok.
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: test.sagepay.com
> Accept: */*
>
 < HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 16 Aug 2012 10:56:46 GMT
Date: Thu, 16 Aug 2012 10:56:46 GMT
< Server: Microsoft-IIS/6.0
Server: Microsoft-IIS/6.0
< P3P: CP="CUR"
P3P: CP="CUR"
< X-Powered-By: ASP.NET
X-Powered-By: ASP.NET
< Content-Length: 1397
Content-Length: 1397
< Content-Type: text/html
Content-Type: text/html
< Set-Cookie: ASPSESSIONIDCUDDCSTT=JOANLLECHFAFLAGOCJGBIFPI; secure; path=/
Set-Cookie: ASPSESSIONIDCUDDCSTT=JOANLLECHFAFLAGOCJGBIFPI; secure; path=/
< Cache-control: private
Cache-control: private

<
* Connection #0 to host test.sagepay.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

# grep ssl /tmp/foo.out
open("/lib/i686/cmov/libssl.so.0.9.8", O_RDONLY) = 3
stat64("/etc/ssl/certs/7651b327.0", {st_mode=S_IFREG|0644, st_size=834, ...}) = 0
open("/etc/ssl/certs/7651b327.0", O_RDONLY|O_LARGEFILE) = 4
stat64("/etc/ssl/certs/7651b327.1", 0xbfcb697c) = -1 ENOENT (No such file or directory)

# readlink -f /etc/ssl/certs/7651b327.0
/usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt
==================================================================

And then the result from 12.04 (tested on two different servers have ca-certificates installed):

==================================================================

# strace -o /tmp/foo.out curl -Iv https://test.sagepay.com
* About to connect() to test.sagepay.com port 443 (#0)
* Trying 195.170.169.8... connected
* successfully set certificate verify locations:
* ...

Read more...

Revision history for this message
Peter (peter-md) wrote :
Download full text (4.9 KiB)

And the output from openssl on 12.04:

==================================================================

# openssl s_client -connect test.sagepay.com:443
CONNECTED(00000003)
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/1.3.6.1.4.1.311.60.2.1.3=GB/businessCategory=Private Organization/serialNumber=01045967/C=GB/ST=TYNE AND WEAR/L=Newcastle Upon Tyne/O=Sage (UK) Limited/OU=Sage/OU=Terms of use at www.verisign.co.uk/rpa (c)05/OU=Authenticated by VeriSign/OU=Member, VeriSign Trust Network/CN=test.sagepay.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGezCCBWOgAwIBAgIQYBuEUnm2OLRLJ8mztoWq7DANBgkqhkiG9w0BAQUFADCB
vjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNjE4MDYGA1UEAxMv
VmVyaVNpZ24gQ2xhc3MgMyBFeHRlbmRlZCBWYWxpZGF0aW9uIFNTTCBTR0MgQ0Ew
HhcNMTEwMjAxMDAwMDAwWhcNMTMwMTMxMjM1OTU5WjCCAVQxEzARBgsrBgEEAYI3
PAIBAxMCR0IxHTAbBgNVBA8TFFByaXZhdGUgT3JnYW5pemF0aW9uMREwDwYDVQQF
EwgwMTA0NTk2NzELMAkGA1UEBhMCR0IxFjAUBgNVBAgUDVRZTkUgQU5EIFdFQVIx
HDAaBgNVBAcUE05ld2Nhc3RsZSBVcG9uIFR5bmUxGjAYBgNVBAoUEVNhZ2UgKFVL
KSBMaW1pdGVkMQ0wCwYDVQQLFARTYWdlMTUwMwYDVQQLFCxUZXJtcyBvZiB1c2Ug
YXQgd3d3LnZlcmlzaWduLmNvLnVrL3JwYSAoYykwNTEiMCAGA1UECxMZQXV0aGVu
dGljYXRlZCBieSBWZXJpU2lnbjEnMCUGA1UECxMeTWVtYmVyLCBWZXJpU2lnbiBU
cnVzdCBOZXR3b3JrMRkwFwYDVQQDFBB0ZXN0LnNhZ2VwYXkuY29tMIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnXwhWyavqxapMnGLEu4ZYKfNvMxymqEj
vjIS4LKJqwebkW7WgmpJNO5lJVwBN/Ks+R1Zlap9jF7Zq6UwNytPb/k9vO/WYyKB
e3exKQcE5yRTRIS8mLWVdM1m02F1I6HCmNFNVWUb3LcI973LqqFwHzBFyZBTez/C
Nhdhmfnl/JRmRtsYMGnlVG6ssnafRENj9oL54adtOtHgqFlpkpDoO6oC6zegZLkz
q+dF+CX5HXj1DVby6ky0lUA2xexY5jtg+3HN+b1PRXVQoPH+8azUFiKCWOOPy9iw
qb8hUBoeJf5jf1vtZlFB0Goy8xjj7sdAqrfxRDkZFHqpTQSxtZMUdwIDAQABo4IB
2jCCAdYwCQYDVR0TBAIwADALBgNVHQ8EBAMCBaAwRAYDVR0gBD0wOzA5BgtghkgB
hvhFAQcXBjAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy52ZXJpc2lnbi5jb20v
Y3BzMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9FVkludGwtY3JsLnZlcmlzaWdu
LmNvbS9FVkludGwyMDA2LmNybDA0BgNVHSUELTArBggrBgEFBQcDAQYIKwYBBQUH
AwIGCWCGSAGG+EIEAQYKKwYBBAGCNwoDAzAfBgNVHSMEGDAWgBROQ8gddu83U3pP
8lhvlPM44tW93zBvBggrBgEFBQcBAQRjMGEwJAYIKwYBBQUHMAGGGGh0dHA6Ly9v
Y3NwLnZlcmlzaWduLmNvbTA5B...

Read more...

Revision history for this message
Kevin Pattison (kevpatts) wrote :

Any update on this issue? Testing now I'm still getting:

curl -v https://test.sagepay.com
* About to connect() to test.sagepay.com port 443 (#0)
* Trying 195.170.169.8... connected
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

information type: Public → Public Security
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openssl (Ubuntu):
status: New → Confirmed
Revision history for this message
uo (uo-li) wrote :

Similar problem on one of my 12.04 32bit Desktop installations.
postfinance.ch and sagepay.com are OK but
Firefox reports sec_error_unknown_issuer for
https://www1.pole-emploi.fr
On another 12.04 32bit Desktop machine this site is fine.

Revision history for this message
Chris Bainbridge (chris-bainbridge) wrote :

TLDR summary: run "c_rehash" as root to fix this issue.

I just ran into this issue (symptoms: "wget https://ev-root.digicert.com/", "openssl c_client ev-root.digicert.com" would fail) .

The problem is that the symbolic links that are supposed to exist in /etc/ssl/certs aren't there. Running "c_rehash" command recreates the links . Reinstallling ca-certificates does not fix this issue, because /usr/sbin/update-ca-certificates only runs c_rehash when /etc/ssl/certs/ca-certificates.crt is out of date (ie. when you added or removed some certificates).

I don't know why an Ubuntu 12.04 LTS system would be in this state, perhaps it only happens on systems that were upgraded from earlier Ubuntu installs, and for some reason c_rehash never got run.

$ wget https://ev-root.digicert.com/
--2013-06-07 19:55:03-- https://ev-root.digicert.com/
Resolving ev-root.digicert.com (ev-root.digicert.com)... 64.58.225.123
Connecting to ev-root.digicert.com (ev-root.digicert.com)|64.58.225.123|:443... connected.
ERROR: cannot verify ev-root.digicert.com's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1':
  Unable to locally verify the issuer's authority.
To connect to ev-root.digicert.com insecurely, use `--no-check-certificate'.

$ strace wget https://ev-root.digicert.com/

write(2, "Connecting to ev-root.digicert.c"..., 80Connecting to ev-root.digicert.com (ev-root.digicert.com)|64.58.225.123|:443... ) = 80
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("64.58.225.123")}, 16) = 0
.....
stat("/usr/lib/ssl/certs/244b5494.0", 0x7fff22ff0b60) = -1 ENOENT (No such file or directory)

$ c_rehash
....

$ ls -l /usr/lib/ssl/certs/244b5494.0
lrwxrwxrwx 1 root root 38 Jun 7 20:20 /usr/lib/ssl/certs/244b5494.0 -> DigiCert_High_Assurance_EV_Root_CA.pem

$ wget https://ev-root.digicert.com/
--2013-06-07 20:20:10-- https://ev-root.digicert.com/
Resolving ev-root.digicert.com (ev-root.digicert.com)... 64.58.225.123
Connecting to ev-root.digicert.com (ev-root.digicert.com)|64.58.225.123|:443... connected.
HTTP request sent, awaiting response... 200 OK

Revision history for this message
Chris Bainbridge (chris-bainbridge) wrote :

This bug might have been caused by changes in bug #854927 which stopped running "update-ca-certificates --fresh" on each install. So I'm guessing this might only affect people who went from Oneiric > Precise.

ca-certificates (20110502+nmu1ubuntu2) oneiric; urgency=low

  * Really only call --fresh on upgrade, instead of all the time; thanks to
    Adam Conrad for catching this in the queue.

Revision history for this message
SeanBoran (sean-boran) wrote :

"update-ca-certificates --fresh" or c_rehash does not fix the issue for me.

Whereas your example with ev-root.digicert.com is ok, the other tests I mentioned are not OK

wget https://postfinance.ch
--2013-06-10 15:36:43-- https://postfinance.ch/
Resolving postfinance.ch (postfinance.ch)... 194.41.226.14
Connecting to postfinance.ch (postfinance.ch)|194.41.226.14|:443... connected.
ERROR: cannot verify postfinance.ch's certificate, issued by `/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA':
  Unable to locally verify the issuer's authority.

wget https://cs.directnet.com
--2013-06-10 15:38:17-- https://cs.directnet.com/
Resolving cs.directnet.com (cs.directnet.com)... 198.240.216.7
Connecting to cs.directnet.com (cs.directnet.com)|198.240.216.7|:443... connected.
ERROR: cannot verify cs.directnet.com's certificate, issued by `/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA':
  Unable to locally verify the issuer's authority.

I'm not sure what systems I tested this on when reporting a year ago, but looking again now, its Ubuntu server 12.04, and most of my servers are upgraded from previous releases. Some are 32 bit, some 64 bit.

Next, went to a Ubuntu 12.04 system that was installed a month ago (i.e. no upgrades),
- "wget https://cs.directnet.com" also gives the above error
- and running "update-ca-certificates --fresh" or c_rehash does not change the result.
- running "curl -v https://test.sagepay.com" (Kevins case) fails too, as does "https://www1.pole-emploi.fr"

Finally also did an "apt-get update && apt-get upgrade" incase there were some patches that might be relevant. No difference though.

Revision history for this message
Wilken Haase (hibbelharry) wrote :

This problem also affects installs of 13.04 here, upgraded and fresh installations as far as I can see

Bug Watch Updater (bug-watch-updater)
Changed in openssl:
status: Unknown → Confirmed
SeanBoran (sean-boran)
description: updated
Revision history for this message
SeanBoran (sean-boran) wrote :

Fix:

Verisign's 2009 ROOT certificate is missing, so download it an install it.

1) Copy the Root CA from Symantec: https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR1556

2) Save to a file like verisign2009.crt in "/usr/local/share/ca-certificates/"
$ sudo vi /usr/local/share/ca-certificates/verisign2009.crt

3) Update the certificate stor
$ sudo update-ca-certificates

Now all of the above https site work fine, at least on 12.04.

The procedure is same for any custom CA certs that one needs to add. /usr/local/share/ca-certificates/ is normally empty.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

There seems to be a mismatch between the "VeriSign Class 3 Public Primary Certification Authority - G5" cert that is in Ubuntu, and the one that is at the end of the cert chain returned by www.postfinance.ch:

In Ubuntu:

VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18:da:d1:9e:26:7d:e8:bb:4a:21:58:cd:cc:6b:3b:4a
Validity Not Before: Nov 8 00:00:00 2006 GMT
         Not After : Jul 16 23:59:59 2036 GMT

www.postfinance.ch returns:

VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 57:bf:fb:03:fb:2c:46:d4:e1:9e:ce:e0:d7:43:7f:13
Validity Not Before: Wed Nov 08 00:00:00 UTC 2006
         Not After: Sun Nov 07 23:59:59 UTC 2021

This results in openssl not being able to validate the chain.
In theory, openssl should discover that the second to last cert in the postfinance.ch chain can be validated with the CA in Ubuntu like NSS and gnutls do, but it doesn't. See upstream openssl bug.

SeanBoran (sean-boran)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20130906ubuntu1

---------------
ca-certificates (20130906ubuntu1) trusty; urgency=low

  * mozilla/certdata2pem.py: Work around openssl issue by shipping both
    versions of the same signed roots. Previously, the script would simply
    overwrite the first one found in the certdata.txt with the later one
    since they both have the same CKA_LABEL, resulting in identical
    filenames. (LP: #1014640)
 -- Marc Deslauriers <email address hidden> Wed, 04 Dec 2013 16:53:53 -0500

Changed in ca-certificates (Ubuntu):
status: New → Fix Released
Louis Bouchard (louis)
Changed in openssl (Ubuntu):
status: Confirmed → Invalid
SeanBoran (sean-boran)
description: updated
Revision history for this message
Bartosz Kosiorek (gang65) wrote :

After install from Precise Proposed the problem was gone.
Verification done

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.