[1.22] Charmed k8s fails to attach PV from csi-cinder - attachdetach-controller: Attach timeout for volume

The underlying Cinder volume gets stuck at "available" and never gets into "attaching" or "in-use", thus the PV is not gonna be available for a pod. For the record, the following step works with v1.21.5 so it's a regression in a broader context.

$ kubectl version --short
...
Server Version: v1.22.2

How to reproduce:

1. Deploy Charmed k8s with openstack-integrator on top of OpenStack

juju add-model k8s-on-openstack "openstack/${OS_REGION_NAME}"

wget -O ~ubuntu/k8s_bundle.yaml https://api.jujucharms.com/charmstore/v5/bundle/kubernetes-core/archive/bundle.yaml

# LP: #1936842
sed -i.bak -e 's/lxd:0/0/' ~ubuntu/k8s_bundle.yaml

# https://github.com/charmed-kubernetes/bundle/blob/master/overlays/openstack-lb-overlay.yaml
cat > ~ubuntu/openstack-lb-overlay.yaml <<EOF
applications:
  openstack-integrator:
    annotations:
      gui-x: "600"
      gui-y: "300"
    charm: cs:~containers/openstack-integrator
    num_units: 1
    trust: true
    to:
    - '0'
    options:
      lb-floating-network: ext_net
relations:
  - ['openstack-integrator:loadbalancer', 'kubernetes-master:loadbalancer']
  - ['openstack-integrator:clients', 'kubernetes-master:openstack']
  - ['openstack-integrator:clients', 'kubernetes-worker:openstack']
EOF

juju deploy --trust ~ubuntu/k8s_bundle.yaml \
    --overlay ~ubuntu/openstack-lb-overlay.yaml

2. Confirm the storage class of "cdk-cinder" is set up by the charm

$ kubectl describe sc
Name: cdk-cinder
IsDefaultClass: No
Annotations: juju.io/workload-storage=true,kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"juju.io/workload-storage":"true"},"labels":{"cdk-addons":"true"},"name":"cdk-cinder"},"provisioner":"cinder.csi.openstack.org"}

Provisioner: cinder.csi.openstack.org
Parameters: <none>
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>

3. Create a PVC and create a pod with the attachment

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc-1
spec:
  storageClassName: cdk-cinder
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: my-pod-1
spec:
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: my-pvc-1
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - name: my-volume
      mountPath: /opt/volumes/my-volume
EOF

[Actual result]

attachdetach-controller AttachVolume.Attach failed for volume "pvc-e77ec6f7-75b3-4d4e-b9c8-7d48e7d8ca46" : Attach timeout for volume bfcab68e-ea62-4328-8f65-368175d668a2

$ kubectl describe pod

...
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 2m18s default-scheduler Successfully assigned default/my-pod-1 to juju-0998b2-k8s-on-openstack-1
  Warning FailedAttachVolume 18s attachdetach-controller AttachVolume.Attach failed for volume "pvc-e77ec6f7-75b3-4d4e-b9c8-7d48e7d8ca46" : Attach timeout for volume bfcab68e-ea62-4328-8f65-368175d668a2
  Warning FailedMount 15s kubelet Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[kube-api-access-zxvhf my-volume]: timed out waiting for the condition

$ openstack volume list --format yaml
- Attached to: []
  ID: bfcab68e-ea62-4328-8f65-368175d668a2
  Name: pvc-e77ec6f7-75b3-4d4e-b9c8-7d48e7d8ca46
  Size: 1
  Status: available

[Expected result - v1.21.5]

AttachVolume.Attach succeeded for volume

$ kubectl describe pod

...
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 55s default-scheduler Successfully assigned default/my-pod-1 to juju-d1d7f0-k8s-on-openstack-1
  Normal SuccessfulAttachVolume 47s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ead5e7e8-11ac-490a-b126-423960457cac"
  Normal Pulling 38s kubelet Pulling image "nginx"
  Normal Pulled 30s kubelet Successfully pulled image "nginx" in 7.910156447s
  Normal Created 29s kubelet Created container nginx
  Normal Started 29s kubelet Started container nginx

$ openstack volume list --format yaml
- Attached to:
  - attached_at: '2021-09-28T02:02:45.000000'
    attachment_id: 9f44a7ea-bfaf-433a-b927-4ad176f9e108
    device: /dev/vdb
    host_name: vast-goose.maas
    id: cba11417-b18a-4d01-9f02-6f13004724c6
    server_id: f7c17617-6340-41ab-8c80-a508a5f3bfab
    volume_id: cba11417-b18a-4d01-9f02-6f13004724c6
  ID: cba11417-b18a-4d01-9f02-6f13004724c6
  Name: pvc-ead5e7e8-11ac-490a-b126-423960457cac
  Size: 1
  Status: in-use