Juju k8s controller is not getting configuration parameters correctly

In the logs you'll see I was using 2.7-beta1, but I'm getting the same result with juju 2.6/stable

Can we get some more information about this please.

What do you expect to happen with the proxy env vars. Should they be applied to just the controller or do you want them injected into every container on every pod managed by Juju?

Also in relation to the k8s cluster, is it just microk8s or is it also intended to work on CDK/GKE/AKS/EKS etc.

Where would the proxy server be running? Inside the cluster or outside? If inside is it running as a service?

Is the cluster using Istio or a CNI like Calico or Flannel?

I think what we have here is a misunderstanding between --config and
--model-defaults

Configuration that is passed in with --config applies JUST to the
controller model.

If you are wanting particular configuration to apply to new models as
well, then you need to specify --model-defaults.

Any config that is passed in to model-defaults will also apply to the
controller model.

Tim

On 8/10/19 1:23 PM, Harry Pidcock wrote:
> Can we get some more information about this please.
>
> What do you expect to happen with the proxy env vars. Should they be
> applied to just the controller or do you want them injected into every
> container on every pod managed by Juju?
>
> Also in relation to the k8s cluster, is it just microk8s or is it also
> intended to work on CDK/GKE/AKS/EKS etc.
>
> Where would the proxy server be running? Inside the cluster or outside?
> If inside is it running as a service?
>
> Is the cluster using Istio or a CNI like Calico or Flannel?
>
> ** Changed in: juju
> Status: Triaged => Incomplete
>

Hello Harry,

The proxy env vars should be applied to the controller and also injected into every container on every pod managed by Juju.

It is intended to work on every k8s cluster, but it seems to be a k8s independent issue.

The proxy server is running outside the cluster

The cluster is not using Istio or a CNI like Calico or Flannel.

For more information, the issue we're having in Field is that when we do juju deploy osm into that model, the charms of the bundle are resolved correctly, but when the first one is going to be uploaded, we're getting this output:

Get https://api.jujucharms.com/charmstore/v5/~charmed-osm/grafana-k8s-21/meta/any?include=id&include=supported-series&include=published: dial tcp 162.213.33.122:443: i/o timeout

Hi @David,

In the past I may have agreed with you, but Juju did change for a reason.

Originally Juju just supported the http_proxy type model config, and those values were written to the machine's /etc/environment.d directory to be included in the standard system environment, but there were too many weird edge cases to support around that behaviour.

New model config variables were introduced prefixed with juju_ (juju_http_proxy and its ilk). These configuration values were exported as part of the charm environment (including the JUJU_ prefix). It was then up to the charm to decide whether to use the proxy or not. Sometimes the charm knew that for internal communication, no proxies were used, and the proxy was only needed for access outside the model.

For this reason, Juju does not automatically set the proxy variables for any workload. If the charm knows that it needs to use the proxies if they are there, then it should be the charm's responsibility to set the proxy variables in the podspec for the container.

Tim's answer above is relevant to charm deployed workloads, but one thing he didn't realise is that we don't currently run the proxy update worker in a k8s controller. So the controller internals, including the bit that connects to the charm store, do not use a configured proxy.

Whether proxy env vars are the best approach in general for k8s is a point that needs considering, as opposed to possibly more idiomatic solutions. As a workaround for now, you could try:

1. bootstrap the controller
2. edit the controller statefulset to amend the podspec to include the proxy env vars
3. let k8s restart the controller pod
4. check that the controller pod has the proxy env vars set and try a deploy again

@david, did the suggested workaround solve the issue? If so we can look to do a fix in juju

Hello Ian! Sorry for the delay. I was able to test it today.

The JUJU_HTTPS_PROXY did not work. What it worked was putting the HTTPS_PROXY.

So variables with JUJU_ were ignored.

@davidgar15

This is intended. The idea is that the user sets what HTTP proxy is available for use, but Juju only populates the JUJU_HTTPS_PROXY env var. Then the charm can decide if it needs to use the proxy or not. e.g. if the communication is for one unit to hit an HTTP API on another unit it doesn't need the proxy. However, if the unit it trying to reach the public internet it probably does use it. This is why we have the split in different proxy variables.

When Juju was only setting the HTTPS_PROXY env var it became a mess because then there was a ton of NO_PROXY setup that needed working so that other aspects would operate correctly.

@rick, JUJU_HTTPS_PROXY is the preferred way to do it, but k8s controllers do not have the proxy update worker wired up yet. Using non-Juju vars is legacy and deprecated.

Any update on this? Do you guys need anything more from me?

k8s controllers now honour the "juju-" prefixed proxy model config