pycurl transport causes tracebacks if the server's SSL cert cannot be verified.

A simple fix is to just create an "InvalidCertificate" error, with 'internal_error=False', and a fairly nice error message.

The page you mentioned is for the "curl" commandline front end. For the library, usually this is more relevant:
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html

And specifically:
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#SSL

This can easily go along with the earlier discussion with vila, about how to enable self-certified hosts. (env var, or config entry). We just allow for a way to not require verification for specific hosts.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John A Meinel пишет:
> A simple fix is to just create an "InvalidCertificate" error, with
> 'internal_error=False', and a fairly nice error message.
>
> The page you mentioned is for the "curl" commandline front end. For the library, usually this is more relevant:
> http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
>
> And specifically:
> http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#SSL
>
> This can easily go along with the earlier discussion with vila, about
> how to enable self-certified hosts. (env var, or config entry). We just
> allow for a way to not require verification for specific hosts.

Can we automatically fallback from pycurl to urllib if certificate
check failed? At least with warning to user?

Alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFvh2OzYr338mxwCURAoyvAJ0Q7k+/N8XtfmV9MEwzEhuY3Mg9qgCghkAH
VqNqpQPzcuBR+tmLtIITWyk=
=njOA
-----END PGP SIGNATURE-----

>>>>> "bialix" == Alexander Belchenko <email address hidden> writes:

    bialix> John A Meinel пишет:
    >> A simple fix is to just create an "InvalidCertificate" error, with
    >> 'internal_error=False', and a fairly nice error message.
    >>
    >> The page you mentioned is for the "curl" commandline front
    >> end. For the library, usually this is more relevant:
    >> http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
    >>
    >> And specifically:
    >> http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#SSL
    >>
    >> This can easily go along with the earlier discussion with
    >> vila, about how to enable self-certified hosts. (env var,
    >> or config entry). We just allow for a way to not require
    >> verification for specific hosts.

I thought about that too, but telling people to disable
certificate verification for *launchpad* ... seems... not the
right thing to do.

    bialix> Can we automatically fallback from pycurl to urllib
    bialix> if certificate check failed? At least with warning to
    bialix> user?

Ouchy, not simply I'm afraid.

Embedding a correct crt file (curl, not libcurl, include such a
file in its distribution*) in the windows distribution may be
more appropriate. And I agree with Andrew's suggestion to talk
about that with pycurl upstream to include the CA for launchpad
(a quick check seems to reveal that starfield is not included so
far).

All in all, I think we should:
1 - issue a nicer error,
2 - allow users to disable certificate verification,
3 - distribute a valid crt file for windows

I can work on 1 and 2, but would appreciate feedback on 3.

    Vincent

*: it's named 'curl-ca-bundle.crt', but it seems it does not
 include the CA for launchpad. On the ubuntu side,
 ca-certificates is a dedicated package.

If ca-certificates is exist in Ubuntu package, may be I could use existing bundle to add to windows installer?

OK. Source package of curl library contains bundle with certificates.
I end up with this testing script that actually works without error:

t = Test()
c = pycurl.Curl()
c.setopt(c.URL, 'https://launchpad.net')
c.setopt(c.WRITEFUNCTION, t.body_callback)
c.setopt(c.CAINFO, 'ca-bundle.crt')
c.perform()
c.close()

The key point is to use
c.setopt(c.CAINFO, 'ca-bundle.crt')

Without this statement pycurl don't want to find valid certificate bundle.

I also try recommendation from this page: http://curl.haxx.se/docs/sslcerts.html
Section 4 explain how curl search bundle on Windows. Unfortunately pycurl
ignores these rules.

Patch for 3:

> 3 - distribute a valid crt file for windows

was landed today. But other things also require to fix. So I don't mark this bug as closed.

Our work SVN server has a self-signed certification, and this error means I can't use bzr-svn to work on private branches.

Am Sonntag, den 16.09.2007, 14:47 +0000 schrieb Ross Burton:
> Our work SVN server has a self-signed certification, and this error
> means I can't use bzr-svn to work on private branches.
You should be able to skip the attempt of bzr to find a bzr-native repository at a https URL by prefixing the URL with
"svn+":

bzr branch svn+https://svn.openchange.org/openchange/trunk

Cheers,

Jelmer
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

$ bzr branch svn+https://svn.o-hand.com/repos/tasks/trunk
bzr: ERROR: libsvn._core.SubversionException: ("Undefined tunnel scheme 'https'", 125002)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 817, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 779, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 477, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 857, in run
    br_from = Branch.open(from_location)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 132, in open
    return control.open_branch(_unsupported)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/format.py", line 177, in open_branch
    repos = self.find_repository()
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/format.py", line 107, in find_repository
    transport = transport.clone_root()
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/transport.py", line 416, in clone_root
    return SvnRaTransport(self.get_repos_root(), self.base)
  File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/errors.py", line 70, in convert
    raise convert_error(e)
SubversionException: ("Undefined tunnel scheme 'https'", 125002)

I presume this is a different bzr-svn bug.

Am Sonntag, den 16.09.2007, 17:30 +0000 schrieb Ross Burton:
> $ bzr branch svn+https://svn.o-hand.com/repos/tasks/trunk
> bzr: ERROR: libsvn._core.SubversionException: ("Undefined tunnel scheme 'https'", 125002)
>
> Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 817, in run_bzr_catch_errors
> return run_bzr(argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 779, in run_bzr
> ret = run(*run_argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 477, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 857, in run
> br_from = Branch.open(from_location)
> File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 132, in open
> return control.open_branch(_unsupported)
> File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/format.py", line 177, in open_branch
> repos = self.find_repository()
> File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/format.py", line 107, in find_repository
> transport = transport.clone_root()
> File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/transport.py", line 416, in clone_root
> return SvnRaTransport(self.get_repos_root(), self.base)
> File "/usr/lib/python2.5/site-packages/bzrlib/plugins/svn/errors.py", line 70, in convert
> raise convert_error(e)
> SubversionException: ("Undefined tunnel scheme 'https'", 125002)
>
> I presume this is a different bzr-svn bug.
yes, this bug was a regression in bzr-svn 0.4.1. Should be ok in bzr-svn
0.4.3.

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

This bug is quite common (at least when using bzr-svn). Can we perhaps upgrade the importance to high?

We can but that will not really change the way to address it.

Summary:
- pycurl *can* be parameterized to accept self-signed certificates but this is frowned upon unless there is a proper way to apply it selectively on a per host basis,
- urllib makes no certificate verification at all (which is worse) because we lack python support,
- there is no easy way to obtain python support *but* python2.6 will include it (the corresponding patch have landed some weeks ago),

Now, if someone feels like plugging some holes, they are welcome to assign the bug to themselves :)

But, python2.6 support means the bug can be cured in the long term. A backport to python2.5 *may* be possible (if not in the core, may be as a plugin). The guy who did the patch for python2.6 said that he may build a package for python2.3, once I see that, I'll look into using it for bzr.

In parallel I'm working on a solution to allow the user to specify certification handling on a per-host basis, once available, it will be possible to tune pycurl.

All in all, I don't forget that bug, it's just that I don't have an acceptable solution for it *right now*.

I forgot to mention, that this bug is AFAIK, the last one needed to be fixed before pycurl support can be dropped.

At the moment this is also failing for launchpad.net, for some reason. Note a user's paste also shows some versions of pycurl can't cope: http://pastebin.ca/744429

kiko wrote:
> At the moment this is also failing for launchpad.net, for some reason. Note a user's paste also shows some versions of pycurl can't cope:
> http://pastebin.ca/744429

With recent bzr.dev, the error reporting is more informative, see bug #141105

Some update:
- the ssl package coming with python2.6 is now available separately for python 2.3 and up,
- I will work on using it with the urllib implementation with python 2.6 and from there with python 2.5 and 2.4,

It looks like using urllib should work then.

i.e. instead of :
 bzr clone https://code.launchpad.net/~amduser29/tortoisebzr/trunk tortoisebzr
use:
 bzr clone https+urllib://code.launchpad.net/~amduser29/tortoisebzr/trunk tortoisebzr

Ran into this one myself after installing pycurl. The transport code now hunts the path for curl-ca-bundle.crt

The libcurl distribution includes a perl script to convert the CA list from Mozilla, and they also conveniently post autogenerated output online

I fixed this by downloading

http://curl.haxx.se/ca/cacert.pem

And putting it on my path as curl-ca-bundle.crt

Our development server uses a self-signed certificate with bzr+https using HTTP Digest authentication to limit access. I can't find a way around this bug to get things to work. The only thing I can think of would be to grab our self-signed cert and distribute it to all of the developers, that's much more of a pain than an option like --no_check_certificate that wget uses.

Is there any plan to add such an option?

On Thu, 2008-07-10 at 20:36 +0000, weswinham wrote:
> Our development server uses a self-signed certificate with bzr+https
> using HTTP Digest authentication to limit access. I can't find a way
> around this bug to get things to work. The only thing I can think of
> would be to grab our self-signed cert and distribute it to all of the
> developers, that's much more of a pain than an option like
> --no_check_certificate that wget uses.

One possible workaround is to install pycurl, or to use urllib+ with
the URLs to avoid pycurl's check.

Thanks,

James

Thanks for the quick response.

I actually installed pycurl on the inkling (from googling around) that it might fix the problem I was having with the error:
bzr branch bzr+https://my.url.tld/code/project/branch
"bzr: ERROR: Invalid http response for https://my.url.tld/code/project/branch/.bzr/smart: Unable to handle http code 401: Authorization Required"

I thought it might be because I was using auth_digest instead of simple auth, and I saw a comment recommending pycurl to get around that. But with pycurl, I'm having the ca cert problem.

Trying urllib+ gives me:
bzr branch urllib+bzr+https://my.url.tld/code/project/branch
bzr: ERROR: Unsupported protocol for url "urllib+bzr+https://dev.policystat.com/code/policystat/trunk
Trying just urllib+https is similar.

I'm now experimenting with added our self-signed cert to my ca-certificates file in ubuntu. I'm hoping that will clear it up, but I'm afraid of trying to port the instructions for installing the cert from ubuntu to mac osx to windows xp/vista for other developers. It seems like there must be an easier way that I'm not seeing. If there were some way to set pycurl options, we could set SSL_VERIFYPEER and/or SSL_VERIFYHOST to 0 and move right on through it.

Well, it seems my real problems lay elsewhere. I installed the certificate and I'm back to:
bzr: ERROR: Invalid http response for https://dev.policystat.com/code/policystat/trunk/.bzr/smart: Unknown response code 401

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

James Westby wrote:
| On Thu, 2008-07-10 at 20:36 +0000, weswinham wrote:
|> Our development server uses a self-signed certificate with bzr+https
|> using HTTP Digest authentication to limit access. I can't find a way
|> around this bug to get things to work. The only thing I can think of
|> would be to grab our self-signed cert and distribute it to all of the
|> developers, that's much more of a pain than an option like
|> --no_check_certificate that wget uses.
|
| One possible workaround is to install pycurl, or to use urllib+ with
| the URLs to avoid pycurl's check.
|
| Thanks,
|
| James
|

Except I think urllib doesn't handle HTTP Digest, only Plain. I could be
wrong, though.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkh2fq4ACgkQJdeBCYSNAAO3ygCghzzTcNu9DVOsHM3OkUJ7dhqQ
fAEAn3JZsDmDDEOaRVSxR+2z4//z/+3x
=ySuV
-----END PGP SIGNATURE-----

>>>>> "john" == John A Meinel <email address hidden> writes:

    john> James Westby wrote:
    >> On Thu, 2008-07-10 at 20:36 +0000, weswinham wrote:
    >>> Our development server uses a self-signed certificate with bzr+https
    >>> using HTTP Digest authentication to limit access. I can't find a way
    >>> around this bug to get things to work. The only thing I can think of
    >>> would be to grab our self-signed cert and distribute it to all of the
    >>> developers, that's much more of a pain than an option like
    >>> --no_check_certificate that wget uses.
    >>
    >> One possible workaround is to install pycurl, or to use urllib+ with
    >> the URLs to avoid pycurl's check.
    >>
    >> Thanks,
    >>
    >> James
    >>

    john> Except I think urllib doesn't handle HTTP Digest, only Plain. I could be
    john> wrong, though.

urllib handles digest.

use http[s]+urllib, not the other way around (bzr don't understand urllib+http).

There *are* plans to implement an option disabling certificate checking.

Currently either you use pycurl and certificates are checked or
you use urllib and they aren't.

Regarding the 401 errors, you must specify at least the user in
the url (or configure your authentication.conf file to provide it
for you).

>>>>> "wes" == weswinham <email address hidden> writes:

    wes> Well, it seems my real problems lay elsewhere. I installed the
    wes> certificate and I'm back to:
    wes> bzr: ERROR: Invalid http response for
    wes> https://dev.policystat.com/code/policystat/trunk/.bzr/smart:
    wes> Unknown response code 401

401 means authentication failed, either user or the password was
wrong. Judging from the url you didn't provide a user, try:

https://<email address hidden>/code/policystat/trunk/

bzr should then prompt you for a password

This is a bug we should fix by at least translating the 401 error
into a clearer message for the user.

Also, digest authentication scheme has a lot of options, urllib doesn't implement them all, for two main reasons:
- lack of time
- digest aimed at providing better security than basic scheme, but can't, by design, compare with https (and if you use https, digest offers no additional security anymore so using basic + https is the preferred way).

That being said, using -Dhttp option in your commands will provide enough information to confirm to diagnose such missing options in the digest implementation.

Is there a simple workaround for this yet? I'm not familiar with the inner workings of pycurl and I'm hitting this too with Bazaar on Windows.

I'm not working on this and don't think I will soon, so better un-assign me.