libwww perl in ubuntu always enforces HTTPS server certificate

The option verify_hostname is like the name suggests only responsible for verifying the host name against the certificate. It does not control the verification of the certificate chain or any other certificate validations, even if it can be used like this in some versions of LWP::Protocol::https. But this is actually a bug, see https://github.com/libwww-perl/lwp-protocol-https/pull/14 (very long discussion).

The only documentation of the option verify_hostname in LWP::UserAgent says:

           "verify_hostname" => $bool
            When TRUE LWP will for secure protocol schemes ensure it connects to servers that have a valid certificate
            matching the expected hostname.

Which confirms that this option cares about verifying the host name only.

To disable any kind of certificate validation you have to use ssl_opts to set SSL_verify_mode to 0 (i.e. SSL_VERIFY_NONE).

So I changed LWP::Porotocol::https back and changed my program thus:

# $ua->ssl_opts( verify_hostname => 0 );
    $ua->ssl_opts( SSL_verify_mode => 0 );

This causes the same error:

Error getting PE routers via REST to blc.serv.as2116.net: 500 Can't connect to blc.serv.as2116.net:443 (certificate verify failed)(Can't connect to blc.serv.as2116.net:443 (certificate verify failed)

So it's still not possible to disable the certificate check. Not to mention that the documentation in the package hasn't been updated to reflect this.

# $ua->ssl_opts( verify_hostname => 0 );
    $ua->ssl_opts( SSL_verify_mode => 0 );
    $ua->ssl_opts( SSL_verifycn_scheme => 'none' );

Additionally it seems to break the principle of least surprise, if someone has working code tested with CPAN LWP and then tries to run it on Ubuntu it will fail without explanation (or any documentation).

I forgot to write that the SSL_vrifycn_scheme => none setting was tested on a whim and it also did not help. That's why the code fragment is there.

Status changed to 'Confirmed' because the bug affects multiple users.

The problem is fixed by LWP::Protocol::https version 6.06. You can either get it from there, or at least patch the bad method. To make it easier, here is the fix, below. The issue seems to be the first else:

sub _extra_sock_opts
{
    my $self = shift;
    my %ssl_opts = %{$self->{ua}{ssl_opts} || {}};
    if (delete $ssl_opts{verify_hostname}) {
 $ssl_opts{SSL_verify_mode} ||= 1;
 $ssl_opts{SSL_verifycn_scheme} = 'www';
    }
    else {
 $ssl_opts{SSL_verify_mode} = 0;
    }
    if ($ssl_opts{SSL_verify_mode}) {
 unless (exists $ssl_opts{SSL_ca_file} || exists $ssl_opts{SSL_ca_path}) {
     eval {
  require Mozilla::CA;
     };
     if ($@) {
  if ($@ =! /^Can't locate Mozilla\/CA\.pm/) {
      $@ = <<'EOT';
Can't verify SSL peers without knowing which Certificate Authorities to trust

This problem can be fixed by either setting the PERL_LWP_SSL_CA_FILE
environment variable or by installing the Mozilla::CA module.

To disable verification of SSL peers set the PERL_LWP_SSL_VERIFY_HOSTNAME
environment variable to 0. If you do this you can't be sure that you
communicate with the expected peer.
EOT
  }
  die $@;
     }
     $ssl_opts{SSL_ca_file} = Mozilla::CA::SSL_ca_file();
 }
    }
    $self->{ssl_opts} = \%ssl_opts;
    return (%ssl_opts, $self->SUPER::_extra_sock_opts);
}

er, by "there" I meant from CPAN.