openstack integrator ignore-volume-az not working

Hi, extra comment
that charm pulled
openstack-cloud-controller-manager:v1.22.0

I just saw upstream has v1.23 and v1.23.1, thus release notes do not mention anything related to AZ.

> it looks like something already reported in https://github.com/kubernetes/cloud-provider-openstack/issues/1300

Looks like this issue has been fixed since since the cloud-provider-openstack 1.20 release[1] which is included in cdk-addons 1.21 and later[2].

One thing I noticed in the linked issue is that their StorageClass has the "availability" parameter[3] set to their volume AZ. The default cdk-cinder StorageClass does not include this and there's no way to set it currently. I suspect that's the missing piece.

You might be able to work around this by creating a new StorageClass that has the availability parameter set to nova:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cdk-cinder-nova
  annotations:
    juju.io/workload-storage: "true"
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org
parameters:
  availability: nova

[1]: https://github.com/kubernetes/cloud-provider-openstack/releases/tag/v1.20.0
[2]: https://github.com/charmed-kubernetes/cdk-addons/pull/202/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R14
[3]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md#supported-parameters

Can you please also check that ignore-volume-az is set correctly in the cloud-config secret?

kubectl get secrets -n kube-system cloud-config -o jsonpath='{.data.cloud\.conf}' | base64 -d

You can also try restarting cinder-csi to ensure that it is using the latest cloud-config data:

kubectl rollout restart -n kube-system statefulset/csi-cinder-controllerplugin
kubectl rollout restart -n kube-system daemonset/csi-cinder-nodeplugin

Many thanks for this guidance, @cynerva. I'm not the original bug submitter, but creating a new StorageClass as you've suggested yielded a major improvement and allowed my environment (three compute AZs, single 'nova' volume AZ, ignore-volume-az = true) to properly bind a Cinder PVC and then schedule a pod using that PVC successfully for the first time.

Do you happen to know why explicitly setting parameters.availability = 'nova' has this effect when the linked plugin documentation[3] states that 'nova' should be the default value?

Also, the plugin documentation for the topology feature[4] talks about allowedTopologies replacing parameters.availability -- is it expected that both approaches would work equally well? I was previously experimenting with the former option (based in part on the comments I found in a very similar OpenShift bug[5]) and found that this allowed my PVCs to bind but left associated pods unschedulable:

$ cat test.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    juju.io/workload-storage: "true"
  labels:
    cdk-addons: "true"
  name: cdk-cinder
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowedTopologies:
- matchLabelExpressions:
  - key: topology.cinder.csi.openstack.org/zone
    values:
    - nova
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: cdk-cinder
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  volumes:
  - name: test-pvc
    persistentVolumeClaim:
      claimName: test-pvc
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
      name: "http-server"
    volumeMounts:
    - mountPath: "/usr/share/nginx/html"
      name: test-pvc
$ kubectl apply -f test.yaml
storageclass.storage.k8s.io/cdk-cinder configured
persistentvolumeclaim/test-pvc created
pod/test-pod created
$ kubectl get pvc/test-pvc pod/test-pod -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/test-pvc Bound pvc-128e7fde-e488-4faf-88f4-591028c1eb36 1Gi RWO cdk-cinder 2m56s Filesystem

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/test-pod 0/1 Pending 0 2m47s <none> <none> <none> <none>
$ kubectl describe pod/test-pod
Name: test-pod
Namespace: default
Priority: 0
Node: <none>
Labels: <none>
Annotations: kubernetes.io/psp: privileged
Status: Pending
IP:
IPs: <none>
Containers:
  nginx:
    Image: nginx
    Port: 80/TCP
    Host Port: 0/TCP
    Environment: <none>
    Mounts:
      /usr/share/nginx/html from test-pvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-szgrp (ro)
Conditions:
  Type Status
  PodScheduled False
Volumes:
  test-pvc:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeC...

> Do you happen to know why explicitly setting parameters.availability = 'nova' has this effect when the linked plugin documentation states that 'nova' should be the default value?

I think it's just poorly documented. In the CSI cinder code that handles this[1] I can't find any reference to the "nova" default. It seems like the default behavior is automatic selection based on topology requirements.

> Also, the plugin documentation for the topology feature talks about allowedTopologies replacing parameters.availability -- is it expected that both approaches would work equally well?

I'm not really familiar with the topology feature but it looks like this would require the nodes to be labeled with topology.cinder.csi.openstack.org/zone=nova. This label comes from csi-cinder-nodeplugin[2] but the value is always equal to the node's AZ. I don't see any config options to change this behavior.

So no, as it is now, I don't think the allowedTopologies approach will work for this case. But I might consider it a bug in csi-cinder that it doesn't give you a way to set topology.cinder.csi.openstack.org/zone correctly on the nodes.

[1]: https://github.com/kubernetes/cloud-provider-openstack/blob/6a574baca1dd2d2eee1b69084f245f5bcea51e31/pkg/csi/cinder/controllerserver.go#L75-L84
[2]: https://github.com/kubernetes/cloud-provider-openstack/blob/6a574baca1dd2d2eee1b69084f245f5bcea51e31/pkg/csi/cinder/nodeserver.go#L474-L478

Hi all,
i can confirm allowedTopologies doesn't work, while, creating a new SC as @cynerva suggested in comment #2 works without having to deal with workers labels.

As the cloud controller keeps on recreating the cdk-cinder SC i wanted to patch it and keep it as default class in order to have a single, clean, sc for users.

unfortunately i cannot

kubectl patch storageclass cdk-cinder -p '{"parameters": {"availability":"nova"}}'
The StorageClass "cdk-cinder" is invalid: parameters: Forbidden: updates to parameters are forbidden.

is there a wait the inject such a config (or stop the re-creation?)

Unfortunately, there's currently no way to to inject configuration into the cdk-cinder StorageClass, and no way to stop cdk-addons from recreating it. That would require code changes to add new config options to cdk-addons, and changes to kubernetes-control-plane to expose those options to users.

We're aiming to deprecate cdk-addons, so I suspect the long-term solution will be for us to break csi-cinder out into a dedicated charm and add config options for the StorageClass there.

Hi any news here?

i saw some changes in the cdk-addons repos but i still fail to set the parameters after the cloud-control created the sc

I'm working on adding a config option to kubernetes-control-plane and cdk-addons to make this configurable.

PRs:
https://github.com/charmed-kubernetes/cdk-addons/pull/225
https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/289

These add a config option to kubernetes-control-plane to make the Cinder AZ configurable. For example:

juju config kubernetes-control-plane cinder-availability-zone=nova

Cherry-picked to release branches:

cdk-addons release-1.27: https://github.com/charmed-kubernetes/cdk-addons/commit/b6548165911e094d0db8f92a89fd522158e3694c
cdk-addons release-1.26: https://github.com/charmed-kubernetes/cdk-addons/commit/ab82ab3cdbc0d859dacda5d350867b9e26aece90
cdk-addons release-1.25: https://github.com/charmed-kubernetes/cdk-addons/commit/741e03d754d004a5a285d366e85a22841b7818d1
kubernetes-control-plane release_1.27: https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/commit/54adcca0dd8e34a7b108d9cd1aaa44e5d606684a

any updates on this? has this been fixed in 1.28?