We are using Kubernetes 1.14 deployed via CDK charms (master: cs:~containers/kubernetes-master-700, worker: cs:~containers/kubernetes-worker-552; openstack integrator: cs:~containers/openstack-integrator-8).
Our openstack integrator is configured with credentials (i.e. not using juju trust). Importantly, the project-name config value is not the OpenStack admin project but a management project (and in other scenarios will be a customer project).
OpenStack is Rocky with Octavia - latest charms.
We can stand up load balancers in OpenStack via CLI fine however, we get failures when standing these up from K8S. This is the error:
Every 2.0s: kubectl describe service ohc-mgmt-prometheus-grafana -n ohc-monitoringjuju-97d08c-0-lxd-1: Wed Aug 14 13:28:53 2019
Name:ohc-mgmt-prometheus-grafana
Namespace:ohc-monitoring
Labels:app=grafana
chart=grafana-3.7.3
heritage=Tiller
release=ohc-mgmt-prometheus
Annotations:<none>
Selector:app=grafana,release=ohc-mgmt-prometheus
Type:LoadBalancer
IP:10.152.183.187
Port:service 80/TCP
TargetPort:3000/TCP
NodePort:service 32470/TCP
Endpoints:10.1.44.5:3000
Session Affinity:None
External Traffic Policy: Cluster
Events:
Type ReasonAgeFromMessage
---- ---------------------
Normal EnsuringLoadBalancer2m5s (x20 over 101m) service-controller Ensuring load balancer
Warning CreatingLoadBalancerFailed 38s (x11 over 64m) service-controller (combined from similar events): Error creating load balancer (will retry): failed to ensure load balancer for service ohc-monitoring/ohc-mgmt-prometheus-graf
ana: Error occurred updating port 10db85cd-811d-40fa-93ed-753db98d29a1 for loadbalancer service ohc-monitoring/ohc-mgmt-prometheus-grafana: Resource not found
We have hacked some of the neutron code to spit out better log messages, and we get the following when this happens:
2019-08-13 22:24:37.633 1863379 INFO neutron.pecan_wsgi.hooks.translation [req-140d3d7b-5a3e-438c-b5d8-b403e5af7c1f 3801cc85c85549a0ad21d5acc490a0fd 08d811a3068248acb1ce950d851def88 - 172176bb0ba34beca92b17508ce29803 172176bb0ba34beca92b1
7508ce29803] Security group ccb5f7b0-9eaf-4038-b804-54149d8800bf does not exist
2019-08-13 22:24:37.634 1863379 INFO neutron.pecan_wsgi.hooks.translation [req-140d3d7b-5a3e-438c-b5d8-b403e5af7c1f 3801cc85c85549a0ad21d5acc490a0fd 08d811a3068248acb1ce950d851def88 - 172176bb0ba34beca92b17508ce29803 172176bb0ba34beca92b1
7508ce29803] PUT failed (client error): The resource could not be found.
This security group refers to the Octavia created security group for the FIP which is created in the services_domain/services project. This security group is already added to the port by Octavia. We believe the issue is that the old embedded openstack cloud provider code is being used by the charms. The k8s OS cloud provider (after creating the LB) creates its own security group and tries to add it to the existing Octavia generated security group. This PUT therefore contains both security groups (the Octavia created one from the services project and the new one from our management project) and fails because Neutron is validating the security groups and the Octavia created one cannot be accessed by the management project scoped token that K8S is using.
Kubernetes therefore aborts the load balancer, issues a delete and tears it back down again. See the code in the OpenStack Cloud Provider here - https://github.com/kubernetes/cloud-provider-openstack/blob/release-1.13/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go#L1231
This looks very similar to this bug https://bugs.launchpad.net/octavia/+bug/1627780.
We have confirmed that the management project user cannot see the octavia generated security group as an "openstack security group show xxxx" of the lb-* security group Octavia created fails with the similar error that K8s Gets ("Error while executing command: No SecurityGroup found for 9bede94a-2121-4d76-8bf4-8014751eaf39") whilst the same user scoped into the admin project can see it.
This *may* be fixed by the external cloud provider code as we note the code is different and does not try to create a security group on the VIP if Octavia is detected, however we have not tested this.
From looking at the code, the edge charms set a flag in the cloud-provider config files that says that Octavia is used as the lbaas. The logic is then different for Octavia vs non-Octavia Lbaas and skips the security group issue we are having .... or at least that's the theory as i can't get them to work!
See: https:/
and how the logic in the openstack cloud provider is linked to this:
https:/