pyOpenSSL has no support for verifying a certificate

Is this a dupe of #385178 ? (Also see #400937)

lp:#385178 covers a lot of ground, but it doesn't seem to cover this.

Created a patch against bazaar trunk, rev 168. Please review and decide if this is how I could proceed (then I'd provide the necessary testcases etc.).

That seems to be missing support for intermediate certificates.

Cheers

Rich.

On 7 March 2013 09:27, Johannes Bauer <email address hidden> wrote:
> Created a patch against bazaar trunk, rev 168. Please review and decide
> if this is how I could proceed (then I'd provide the necessary testcases
> etc.).
>
> ** Patch added: "crt-verification.patch"
> https://bugs.launchpad.net/pyopenssl/+bug/892522/+attachment/3561785/+files/crt-verification.patch
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/892522
>
> Title:
> pyOpenSSL has no support for verifying a certificate
>
> Status in pyOpenSSL:
> New
>
> Bug description:
> Currently, whilst pyopenssl /can/ verify a certificate when connecting
> to an SSL server, it does not have a facility to verify a certificate
> chain on its own. The functionality required is provided in openssl
> (eg. the openssl verify command allows this to be done on the command
> line). Adding the required functionality is relatively
> straightforward, and I've implemented it for the Qt project before.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/pyopenssl/+bug/892522/+subscriptions

No, it isn't. It works fine with intermediate certificates. How do you arrive at that conclusion? Have you tried it out and have a case that fails although it should pass?

To explain a bit furhter what the code does: It inserts the issuer into an otherwise empty truststore. Then it tries to check the certificate with that. Now there are three cases:

1. If the direct signature issuer -> subject is broken, verify will fail at depth 0 and the function will throw an exception.
2. If the direct signature issuer -> subject is correct and issuer is a self-signed certificate, verify will succeed, the function returns None.
3. If the direct signature issuer -> subject is correct and the issuer is an intermediate CA, verify will fail at depth 1 (!), the function returns None (!).

Hope that clears it up,
Johannes

On 7 March 2013 13:33, Johannes Bauer <email address hidden> wrote:
> To explain a bit furhter what the code does: It inserts the issuer into
> an otherwise empty truststore. Then it tries to check the certificate
> with that. Now there are three cases:
>
> 1. If the direct signature issuer -> subject is broken, verify will fail at depth 0 and the function will throw an exception.
> 2. If the direct signature issuer -> subject is correct and issuer is a self-signed certificate, verify will succeed, the function returns None.
> 3. If the direct signature issuer -> subject is correct and the issuer is an intermediate CA, verify will fail at depth 1 (!), the function returns None (!).

I was thinking of the case where the root CA has issued a cert using
an intermediate. An SSL server in this case would transmit the leaf
and the intermediate. I don't see that this API would let you test the
leaf against an existing trust store. In terms of the openssl command
line, this would be something like:

openssl verify -CApath /etc/ssl/certs -untrusted intermediate.pem leaf.pem

Given what you've said above, it sounds like you did not intend to
support this case.

Cheers

Rich.

>
> Hope that clears it up,
> Johannes
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/892522
>
> Title:
> pyOpenSSL has no support for verifying a certificate
>
> Status in pyOpenSSL:
> New
>
> Bug description:
> Currently, whilst pyopenssl /can/ verify a certificate when connecting
> to an SSL server, it does not have a facility to verify a certificate
> chain on its own. The functionality required is provided in openssl
> (eg. the openssl verify command allows this to be done on the command
> line). Adding the required functionality is relatively
> straightforward, and I've implemented it for the Qt project before.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/pyopenssl/+bug/892522/+subscriptions

Yes, the API that I provided does not check a complete trust chain (like openssl verify with the CApath option does, i.e. it uses a whole directory as a truststore). It is only intended to tell the user: "Is the signature of certificate X by issuer Y correct?". Although it would be rather trivial to extend the API to the latter case (i.e. insert many certificates in the store and require a complete chain of trust until a root CA is hit), that's not what I intended.

If that would be what is wanted, the issuer parameter could be changed into a tuple of X509 objects which would all be inserted in the truststore. Then also add a kwarg "trustchain" in which you can specify a bool that tells if you would only want to check the *immediate* relationship between two certificates (that's what I need) or if you would want to build a complete trustchain (that is what you need, if I understand correctly).

On 7 March 2013 16:08, Johannes Bauer <email address hidden> wrote:
> Yes, the API that I provided does not check a complete trust chain (like
> openssl verify with the CApath option does, i.e. it uses a whole
> directory as a truststore). It is only intended to tell the user: "Is
> the signature of certificate X by issuer Y correct?". Although it would
> be rather trivial to extend the API to the latter case (i.e. insert many
> certificates in the store and require a complete chain of trust until a
> root CA is hit), that's not what I intended.
>
> If that would be what is wanted, the issuer parameter could be changed
> into a tuple of X509 objects which would all be inserted in the
> truststore. Then also add a kwarg "trustchain" in which you can specify
> a bool that tells if you would only want to check the *immediate*
> relationship between two certificates (that's what I need) or if you
> would want to build a complete trustchain (that is what you need, if I
> understand correctly).

I'm not sure I'm following you here, could you give an example? In the
scenario I describe, the intermediates are not trusted and shouldn't
form part of the trust store, they merely allow you to build a chain
from the leaf to one of the roots. I suspect we're probably describing
the same thing, but I'm not quite sure.

I think having the ability to verify a whole chain would be a great
improvement to pyopenssl.

Cheers

Rich.

On 07.03.2013 23:05, Richard Moore wrote:

> I'm not sure I'm following you here, could you give an example? In the
> scenario I describe, the intermediates are not trusted and shouldn't
> form part of the trust store, they merely allow you to build a chain
> from the leaf to one of the roots. I suspect we're probably describing
> the same thing, but I'm not quite sure.
Sure. Let's first define a certificate tree with one self-signed
certificate A:

A -> B -> C -> D
A -> E -> F
B -> G -> H

Or in graphviz notation:

digraph certs {
  A->A;
  A->B;
  B->C;
  C->D;
  A->E;
  E->F;
  B->G;
  G->H;
}

You could go to http://sandbox.kidstrythisathome.com/erdos/index.html
and it will output a nice graph image or view it at http://imgur.com/Nnf8kFl

Now in that graph as the API is currently designed with my patch, it
will yield a successful check for the following issuer/subject pairs and
will throw an exception for all others:

A, A
A, B
B, C
C, D
A, E
E, F
B, G
G, H

I.e. as you can see it clearly models the tree exactly. This is just a
basic building block, however. You would want to extend this to validate
whole chains.

Now there's some confusion because we use different terms for
"truststore". You are talking about the certificates which have ultimate
trust (in this case this would be only A) and I talk about certificates
with *technical* trust (i.e. all root certificates that end up in the
truststore object).

The logical truststore only includes A. The technical truststore also
includes all intermediate CAs, as they're indirectly also trusted if
they have a valid signature from the root.

I suggest to change the API to take a tuple of trusted issuers and a
subject to verify together with a kwarg parameter that indicates whether
the user is only interested in the direct trust relationship (this is
something I'm after in my use case) or wants to build a chain of trust.
Some examples:

This would all pass (direct cert check):
verify_cert([ A ], B)
verify_cert([ A, B, C ], B)
verify_cert([ C, G ], D)

This would all fail (direct cert check):
verify_cert([ ], A)
verify_cert([ B ], A)
verify_cert([ C, G, D, H ], F)

This would all pass (chain building check):
verify_cert([ A, B, G ], H, chain = True)
verify_cert([ A, B, C ], D, chain = True)
verify_cert([ A ], A, chain = True)
verify_cert([ A ], B, chain = True)
verify_cert([ A, E ], F, chain = True)

This would all fail (chain building check)
verify_cert([ ], A, chain = True)
verify_cert([ A ], G, chain = True)
verify_cert([ A ], H, chain = True)
verify_cert([ B, G ], H, chain = True)
verify_cert([ B, C ], D, chain = True)

All in all, the function would not differentiate between ultimate trust
(truststore) and immediate trust (technical truststore). This is
something that can be done with simple user functions (or I could write
a simple wrapper that does this). But ultimately, I'd like to keep the
functionality as simply as possible (reduces the risk of misuse).

Now when I see correctly, what you want is something like this:

verify_chain(truststore, chain, cert)

Which would pass for the following inputs:
verify_chain([ A ], [ A, B, G, H ], H)
verify_chain([ A ], [ A, B, G, H ], G)
verify_cha...

Edit: Obviously in the verify_chain() function definition it needs to relay to verify_cert as:

return verify_cert(tech_truststore, cert, chain = True)

Sorry about that.

On 8 March 2013 12:32, Johannes Bauer <email address hidden> wrote:
> On 07.03.2013 23:05, Richard Moore wrote:
>
>> I'm not sure I'm following you here, could you give an example? In the
>> scenario I describe, the intermediates are not trusted and shouldn't
>> form part of the trust store, they merely allow you to build a chain
>> from the leaf to one of the roots. I suspect we're probably describing
>> the same thing, but I'm not quite sure.
> Sure. Let's first define a certificate tree with one self-signed
> certificate A:
>
> A -> B -> C -> D
> A -> E -> F
> B -> G -> H
>
> Or in graphviz notation:
>
> digraph certs {
> A->A;
> A->B;
> B->C;
> C->D;
> A->E;
> E->F;
> B->G;
> G->H;
> }
>
> You could go to http://sandbox.kidstrythisathome.com/erdos/index.html
> and it will output a nice graph image or view it at http://imgur.com/Nnf8kFl
>
> Now in that graph as the API is currently designed with my patch, it
> will yield a successful check for the following issuer/subject pairs and
> will throw an exception for all others:
>
> A, A
> A, B
> B, C
> C, D
> A, E
> E, F
> B, G
> G, H
>
> I.e. as you can see it clearly models the tree exactly. This is just a
> basic building block, however. You would want to extend this to validate
> whole chains.
>
> Now there's some confusion because we use different terms for
> "truststore". You are talking about the certificates which have ultimate
> trust (in this case this would be only A) and I talk about certificates
> with *technical* trust (i.e. all root certificates that end up in the
> truststore object).
>
> The logical truststore only includes A. The technical truststore also
> includes all intermediate CAs, as they're indirectly also trusted if
> they have a valid signature from the root.
>
> I suggest to change the API to take a tuple of trusted issuers and a
> subject to verify together with a kwarg parameter that indicates whether
> the user is only interested in the direct trust relationship (this is
> something I'm after in my use case) or wants to build a chain of trust.
> Some examples:
>
> This would all pass (direct cert check):
> verify_cert([ A ], B)
> verify_cert([ A, B, C ], B)
> verify_cert([ C, G ], D)
>
> This would all fail (direct cert check):
> verify_cert([ ], A)
> verify_cert([ B ], A)
> verify_cert([ C, G, D, H ], F)
>
> This would all pass (chain building check):
> verify_cert([ A, B, G ], H, chain = True)
> verify_cert([ A, B, C ], D, chain = True)
> verify_cert([ A ], A, chain = True)
> verify_cert([ A ], B, chain = True)
> verify_cert([ A, E ], F, chain = True)
>
> This would all fail (chain building check)
> verify_cert([ ], A, chain = True)
> verify_cert([ A ], G, chain = True)
> verify_cert([ A ], H, chain = True)
> verify_cert([ B, G ], H, chain = True)
> verify_cert([ B, C ], D, chain = True)
>
> All in all, the function would not differentiate between ultimate trust
> (truststore) and immediate trust (technical truststore). This is
> something that can be done with simple user functions (or I could write
> a simple wrapper that does this). But ultimately, I'd like to keep the
> functionality as simply as possible (reduces the risk of misuse).
...

On 09.03.2013 15:54, Richard Moore wrote:
> My concept would also fail for
> verify_chain([A], [B,C,G], F)
>
> but would pass for
>
> verify_chain([A], [B,C,G], E)

Precisely.

> Okay, I see what you're proposing now. Unfortunately, I don't think
> this minimal API is enough to perform the full verification.

But it is :-)

> I agree
> it can check that the leaf validates up to the trust store, but it
> would need a lot of additional logic to be put in place to ensure that
> the leaf is actually valid. Among other things, you'd need to handle a
> large number of additional constraints such as key usage, basic
> constraints (is it a CA, path length), name constraints etc.

This is not true. Since all functions are ultimately relayed to OpenSSLs X509_verify_cert (see patch), all these constraints are checked by OpenSSL already as of now. All length contraints, basic constraints, validity, etc. The thing that it is not checked are CRLs or OCSP.

> Whilst it
> would be possible to do this building on the ability to verify that
> one cert has been signed by another

Not what I'm proposing at all. I just need the API do perform that check, therefore I've exposed the ability to do this. For chain verification, if it's done like I explained, OpenSSL does that work comepletely for free.

>, combined with the extension
> support in pyopenssl it would be a lot of work. Since openssl already
> does these checks, it seems to me that an easier approach is to simply
> make use of the code that's already there.

Precisely.

Has this patch ever made it to pyOpenSSL?

I'm having a similar problem right now, where I need to verify a certificate with CA chain and need to look at every single certificate in the chain. The described verify_cert function would help me a lot with that.

I'm sorry, but I do not think it has. As you can see from the discussion three years back, there was first a LOT of explanation necessary, a lot of misunderstandings and eventually just dead silence. Back then I dropped pyOpenSSL support completely and switched to my own solution.

In hindsight, I thought Richard was the maintainer of pyOpenSSL and thus this bugtracker would have been the correct place. However, I think that Hynek Schlawack is responsible for pyOpenSSL. You might have some luck creating a PR at his GitHub project and/or open a ticket there: https://github.com/hynek/pyopenssl

I personally have zero interest in contributing to pyOpenSSL anymore. It was tedious and annyoing three years back and I'm not making the same mistake twice. I wish you luck, however, Guido, should you attempt to get this functionality in. It is definitely very useful in certain cases. So if you do file a bug or PR against Hynek's GitHub repo, please also link it here. I'm interested to follow what happens.

(random passer by comment)

After reading this and following the trail from https://github.com/pyca/pyopenssl/issues/154 this is allegedly solved via https://github.com/pyca/pyopenssl/pull/155 .