cannot deploy bundle, cannot resolve URL, TLS handshake timeout

juju issues a GET http request to the charm store

https://api.jujucharms.com/charmstore/v5/~openstack-charmers-next/ceph-mon/meta/any?include=id&include=supported-series&include=published

to fetch metadata about the charm. This HTTP request is timing out.

It sure seems like there's underlying connectivity issues external to Juju here.

This should get fixed when we move to the new shim api.

Hi,

We're consistently seeing this problem when running func tests on an OpenStack Charm. Please have a look to the attached pdf. In all but one of the tests, the issue was related to this bug. The charmstore times out after 10s.

Is there a "juju deploy" option to increase the timeout (eg. 15 or 20s)?

Bumping to high. It sounds like we might be able to increase a timeout value to fix this, and it feels like something that might be affecting production deployments, in addition to the test environment here.

Per conversation in sync, it is probably better to fix this by being smarter about re-using TLS connection, rather than setting a higher timeout.

(If we bump up the timeout, we're probably going to run into other issues talking to the store down the line.)

The OpenStack team also suggested a similar approach to tenacity.retry, where connection timeouts may be retried once (or more if an arg like "--connection-retry" could be passed).

Is there a temp workaround about this?

Sub'd to field-high, this is affecting solutions-qa release testing.

Here's what we believe is another manifestation of this.
We run a Kubernetes test suite that's failing to pull image.

---

containerd_2/var/log/syslog:Feb 9 11:20:23 juju-074d0d-7 containerd[41538]: time="2021-02-09T11:20:23.753972008Z" level=error msg="PullImage
"rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8"
failed" error="failed to pull and unpack image
"rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8": failed to resolve reference "rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8": failed to do request: Head https://rocks.canonical.com/v2/cdk/jujusolutions/jujud-operator/manifests/2.8.8: net/http: TLS handshake timeout"

---

Example run:
https://solutions.qa.canonical.com/testruns/testRun/c9df119d-7bc6-4ebc-8f7c-7240897c6f85

Juju status at the bottom of this page:
https://oil-jenkins.canonical.com/job/fce_build/9620/console

Juju model config and crashdump:
https://oil-jenkins.canonical.com/artifacts/c9df119d-7bc6-4ebc-8f7c-7240897c6f85/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-02-09-11.24.47.tar.gz

All artifacts:
https://oil-jenkins.canonical.com/artifacts/c9df119d-7bc6-4ebc-8f7c-7240897c6f85/index.html

So this feels like something on the order of "your VMs are not getting
enough entropy in order to generate private keys for TLS connections".
I don't know that it is the case, but I worry that just doing retries on
Juju's behalf wont make things better (as it just consumes more of whatever
limited resource is causing TLS handshakes to fail).

On Tue, Feb 9, 2021 at 1:51 PM Joshua Genet <email address hidden>
wrote:

> Here's what we believe is another manifestation of this.
> We run a Kubernetes test suite that's failing to pull image.
>
> ---
>
> containerd_2/var/log/syslog:Feb 9 11:20:23 juju-074d0d-7
> containerd[41538]: time="2021-02-09T11:20:23.753972008Z" level=error
> msg="PullImage
> "rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8"
> failed" error="failed to pull and unpack image
> "rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8": failed to
> resolve reference "
> rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8": failed to do
> request: Head
> https://rocks.canonical.com/v2/cdk/jujusolutions/jujud-operator/manifests/2.8.8:
> net/http: TLS handshake timeout"
>
> ---
>
> Example run:
>
> https://solutions.qa.canonical.com/testruns/testRun/c9df119d-7bc6-4ebc-8f7c-7240897c6f85
>
> Juju status at the bottom of this page:
> https://oil-jenkins.canonical.com/job/fce_build/9620/console
>
> Juju model config and crashdump:
>
> https://oil-jenkins.canonical.com/artifacts/c9df119d-7bc6-4ebc-8f7c-7240897c6f85/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-02-09-11.24.47.tar.gz
>
> All artifacts:
>
> https://oil-jenkins.canonical.com/artifacts/c9df119d-7bc6-4ebc-8f7c-7240897c6f85/index.html
>
> --
> You received this bug notification because you are a member of Canonical
> Field High, which is subscribed to the bug report.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1906372
>
> Title:
> cannot deploy bundle, cannot resolve URL, TLS handshake timeout
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1906372/+subscriptions
>

What's interesting is that there's now 2 external services affected, called by 2 separate clients:

1. charm store (used by the juju client)
2. rocks (used by k8s itself, ie containerd)

Given containerd is also affected, ie it connects to rocks to pull an image totally outside of juju, this doesn't look like a Juju issue per se, and really does appear to be an artifact of the deployment environment.

Ian,

I'm sorry but I don't buy the "it's your environment" line here. The 2.8.7 client was blessed on Dec 11th, 2020 and we shut our CI off the following Wednesday for the holiday break. We started seeing this sporadically starting On Jan 6th [0], basically a day after turning our CI back on. Assuming you didn't release on a Friday that was only a few days for this issue to present. Between the 2.8.7 release tests and today nothing has changed within our test lab that could have caused this.

This has also been confirmed by the OpenStack engineering team [1] and at least one community member.

So either the Juju client has a defect, or the charm store is working poorly. Either way it's a Juju issue.

0. https://solutions.qa.canonical.com/bugs/bugs/bug/1906372
1. https://bugs.launchpad.net/juju/+bug/1906372/comments/3

It's also containerd, independent of Juju.

Below, the containerd service is trying to pull an image from rocks.canonical.com and gets the TLS handshake error. Juju is not involved here.

containerd_2/var/log/syslog:Feb 9 11:20:23 juju-074d0d-7 containerd[41538]: time="2021-02-09T11:20:23.753972008Z" level=error msg="PullImage
"rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8"
failed" error="failed to pull and unpack image
"rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8": failed to resolve reference "rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.8.8": failed to do request: Head https://rocks.canonical.com/v2/cdk/jujusolutions/jujud-operator/manifests/2.8.8: net/http: TLS handshake timeout"

K8s/containred retries pulling images, doesn't it?
https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff

I'm not saying Juju is doing something wrong here, but having retries and backoffs in Juju on pulling resources from charmhub and such make our life much easier.