Azure provider: storage stuck in 'pending'

The likely cause is that the volume is mounted at as a device where juju cannot reconcile the mount path with the pending volume. Matching is done based on a variety of factors (eg WWN, device id etc) and each substrate is slightly different. Often, a substrate might introduce a new storage class that mounts slightly differently and Juju needs updating to account for it. we've sen this on AWS with nvme volumes.

It looks like this is also with the OpenStack provider with Juju 2.7.6 as well as 2.8.0.

Separate to Azure, I've tried using the shared Juju 2.x controllers in PS4.5 and that also fails:

| Model Controller Cloud/Region Version SLA Timestamp
| stg-is-content-cache prodstack-is prodstack-45/bootstack-ps45 2.7.6 unsupported 05:22:56Z
| ...
| ubuntu-repository-cache-new/0* waiting allocating 4 10.2xx.xxx.xxx agent initializing

| $ juju list-storage
| ubuntu-repository-cache-new/0 ubuntu-repository-cache/3 filesystem pending

As well as some random CI environment, different controller, 2.8.0:

| Model Controller Cloud/Region Version SLA Timestamp
| XXX jenkins-ci-controller prodstack4.5/bootstack-ps45 2.8.0 unsupported 05:24:00Z
| ubuntu-repository-cache-new/0* waiting allocating 0 10.2x.xxx.xx agent initializing

| $ juju list-storage
| Unit Storage id Type Size Status Message
| ubuntu-repository-cache-new/0 ubuntu-repository-cache/0 filesystem pending

In both cases storage-default-block-source is 'cinder'. In Azure's case, it's 'azure'.

FWIW, before upgrading the shared 2.x controllers to 2.7.6, it was running 2.6.10 and was able to deploy units with attached storage.

For the openstack case (on prodstack 4.5), juju storage --format yaml shows the error

'invalid status (400): {"badRequest": {"message": "Invalid input received: Availability zone ''prodstack-zone-1'' is invalid", "code": 400}}'

In 2.7, we introduced code to include the AZ of the host machine to which the storage would be attached to the CreateVolume() API call. In 2.6 this parameter was empty.

$ nova availability-zone-list
+------------------+---------------+
| Name | Status |
+------------------+---------------+
| prodstack-zone-2 | available |
| prodstack-zone-1 | available |
| nova | not available |
+------------------+---------------+

The compute node is in prodstack-zone-1 and that's what Juju is using for CreateVolume().

Yet showing an existing volumes created outside of this test Juju deployment,

$ nova volume-show 663443e6-a558-485d-83a6-b880745b05f5

availability_zone | nova

So it looks like the allowable AZ for volumes do not match those for compute nodes. Juju doesn't know how to handle this.

It looks like it's two separate bugs here, one for the OpenStack provider and the other for Azure (unless it's also the bit of code to include the AZ that's causing an issue with Azure too?).

Should we split this into two bugs or keep it as one?

For OpenStack, in our set up there's only a single AZ for cinder:

$ nova availability-zone-list
+------------------+---------------+
| Name | Status |
+------------------+---------------+
| prodstack-zone-2 | available |
| prodstack-zone-1 | available |
| nova | not available |
+------------------+---------------+

$ cinder availability-zone-list
+------+-----------+
| Name | Status |
+------+-----------+
| nova | available |
+------+-----------+

Unlike with compute, where there are two and the default one 'nova' disabled.

Yes, there's 2 separate issues here. Different bugs are warranted.

Split out OpenStack issues to LP:1885639. So keeping this one as Azure specific.

https://github.com/juju/juju/pull/11771