Strange internal IP is reported when a worker node has multiple NICs

Can you run `juju run --unit kubernetes-worker/2 -- network-get kube-control`? This should show you the ingress addresses that the charm sees. The charm chooses the first ingress address and passes that to kubelet via the `--node-ip` config option. I don't think this is the issue, but it would help if you can rule it out.

I suspect you're right that openstack-cloud-controller-manager is responsible for adding the other addresses. It looks like this behavior can be controlled in openstack-cloud-controller-manager by setting the internal-network-name option[1]. Sadly, our charms don't support this in the generated config[2], so there's no way for you to set it. You won't be able to manually edit the cloud config; cdk-addons will revert your changes every 5 minutes.

I can't think of a good workaround to recommend, but I'm guessing it could be fixed by adding support for the internal-network-name option to our charms. It would probably need to be a config option on the openstack-integrator charm that gets passed through to kubernetes-control-plane.

[1]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#networking
[2]: https://github.com/charmed-kubernetes/layer-kubernetes-common/blob/1df6fc7fd08b14324d993e64f9b6459020229df0/lib/charms/layer/kubernetes_common.py#L550-L606

Hi George, thank you for looking in to this.

I reproduced the issue by adding relation between worker charm and the openstack-integrator charm.

        $ kubectl get nodes -o wide
        NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
        master-1 Ready <none> 27h v1.24.1 192.168.150.50 <none> Ubuntu 20.04.4 LTS 5.4.0-109-generic containerd://1.5.9
        worker-1 Ready <none> 27h v1.24.1 192.168.150.51 <none> Ubuntu 20.04.4 LTS 5.4.0-109-generic containerd://1.5.9
        worker-2 Ready <none> 27h v1.24.1 192.168.150.52 <none> Ubuntu 20.04.4 LTS 5.4.0-109-generic containerd://1.5.9
        worker-3 Ready <none> 27h v1.24.1 192.168.141.41 <none> Ubuntu 20.04.4 LTS 5.4.0-109-generic containerd://1.5.9

Here is the requested output:

        $ juju run -m k8s --unit kubernetes-worker/2 -- network-get kube-control
        bind-addresses:
        - mac-address: fa:16:3e:e2:d7:70
          interface-name: ens3
          addresses:
          - hostname: ""
            address: 192.168.150.53
            cidr: 192.168.150.0/24
          macaddress: fa:16:3e:e2:d7:70
          interfacename: ens3
        - mac-address: 7e:54:d9:66:e6:20
          interface-name: fan-252
          addresses:
          - hostname: ""
            address: 252.53.0.1
            cidr: 252.0.0.0/8
          macaddress: 7e:54:d9:66:e6:20
          interfacename: fan-252
        egress-subnets:
        - 192.168.150.53/32
        ingress-addresses:
        - 192.168.150.53
        - 252.53.0.1

So it seems that the charm doesn't see the other Openstack networks. By the way, --node-ip in the kubelet's cmdline arguments is also correct:

        $ juju ssh -m k8s kubernetes-worker/2 'ps aux | grep kubelet | grep node-ip'
        root 554863 1.3 0.3 3038536 102496 ? Ssl 01:28 0:06 /snap/kubelet/2423/kubelet --kubeconfig=/root/cdk/kubeconfig --v=0 --logtostderr=true --node-ip=192.168.150.53 --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --cloud-provider=external --config=/root/cdk/kubelet/config.yaml --pod-infra-container-image=rocks.canonical.com:443/cdk/pause:3.6
        ubuntu 560091 0.0 0.0 8616 3176 pts/0 Ss+ 01:36 0:00 bash -c ps aux | grep kubelet | grep node-ip
        Connection to 192.168.150.53 closed

If this is causing `kubectl exec` and `kubectl logs` commands to fail, you might be able to work around that by setting the kubelet-preferred-address-types config of kube-apiserver[1] to prefer Hostname first:

juju config kubernetes-control-plane api-extra-args="kubelet-preferred-address-types=Hostname,InternalIP,InternalDNS,ExternalDNS,ExternalIP"

But that will only work if hostnames like worker-3 are resolveable to good IPs.

[1]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

Thanks George, I checked that and the suggested workaround helped:

ubuntu@jumphost:~$ kubectl exec -it -n kube-system openstack-cloud-controller-manager-rg2l8 cat /etc/config/cloud.conf
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[Global]
auth-url = https://keystone.pc1.canonical.com:5000/v3
region = PartnerCloud1
username = sriov-in-k8s-user
password = quodAjAs2
tenant-name = sriov-in-k8s
domain-name = admin_domain
tenant-domain-name = admin_domain
ca-file = /etc/config/endpoint-ca.cert

[LoadBalancer]
use-octavia = true
lb-method = ROUND_ROBIN

Without the workaround I couldn't do kubectl exec, because it hangs trying to connect to incorrect node IP address:

ubuntu@jumphost:~$ kubectl exec -it -n kube-system openstack-cloud-controller-manager-rg2l8 cat /etc/config/cloud.conf
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Error from server: error dialing backend: dial tcp 192.168.190.31:10250: i/o timeout

Also from what read in Openstack Cloud Controller Manager docs [1] and [2], a few other options are worth checking: public-network-name and internal-network-name. Currently they're not exposed as charm configuration options, and not propagated to openstack cloud controller manager as it has been mentioned in the previous comments.

We know from the initial description of this bug that the problem is in multiple internal IPs assigned to nodes, and based on [3] internal-network-name option directly affects which IP address is selected as the internal node IP. So internal-network-name seems to be the fix for the issue if we need nodes to be accessible via their Internal IPs, not their names.

[1] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#networking
[2] https://github.com/charmed-kubernetes/cdk-addons/blob/main/get-addon-templates#L86
[3] https://github.com/kubernetes/cloud-provider-openstack/blob/master/pkg/openstack/instances.go#L551