Juju doesn't mount storage after lxd container restart

One more detail that may be helpful:

I noticed that at the times the issue happens (the storage is not mounted after a restart) the ctx.filesystems map is empty (the filesystem of the storage is missing there) and then https://github.com/juju/juju/blob/juju-2.9.37/worker/storageprovisioner/storageprovisioner.go#L339 is called (which doesn't mount the storage as the filesystem is missing). The other times, ctx.filesystems is populated with the storage that is needed, so the storage is mounted.

Observed same behavior with lxd 5.0.1 on jammy and juju 2.9.37

Digging into this, it seems like there is sequencing problem when enqueuing onto the scheduler. The code expects that a filesystem change always precedes a filesystem attachment change[1]. In normally day operations this is the case, but for a hard restart, this isn't always the case.

Unfortunately, case statements in a select are shuffled upon every entry. When we receive all the information at once, when the restart has completed there is no guarantee of the correct order[2]. For example in most cases the ordering is valid:

    filesystems changed: []string{"0/0"}
    filesystem attachments changed: []watcher.MachineStorageId{watcher.MachineStorageId{MachineTag:"machine-0", AttachmentTag:"filesystem-0-0"}}

For times when a attachment is not successful, the attachment comes before the change:

    filesystem attachments changed: []watcher.MachineStorageId{watcher.MachineStorageId{MachineTag:"machine-0", AttachmentTag:"filesystem-0-0"}}
    filesystems changed: []string{"0/0"}

The solution is non-obvious, as you can't always guarantee that the proceeding change follows an attachment.

 1. https://github.com/juju/juju/blob/2.9/worker/storageprovisioner/filesystem_events.go#L165
 2. https://github.com/juju/juju/blob/2.9/worker/storageprovisioner/storageprovisioner.go#L304-L358