Charm does not surface errors when supplied block devices cannot be used

Bug #1842663 reported by Liam Young
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Swift Storage Charm
Triaged
Wishlist
Unassigned

Bug Description

In my case juju storage provided the charm with a disk that was already formatted. This caused mkfs.xfs to fail and the deployment to be one zone down. The only visible symptom to the end user was the "Not enough storage zones for minimum replicas" message displayed by the swift-proxy charm. This is confusing to the user as the error was actually in the storage charm. The storage charm should clearly state in juju status that it failed to initialise the required disk(s).

$ juju ssh swift-storage-z3/0 "sudo grep mkfs /var/log/juju/unit*"
2019-09-03 10:06:14 DEBUG config-changed mkfs.xfs: /dev/vdc appears to contain an existing filesystem (ext4).
2019-09-03 10:06:14 DEBUG config-changed mkfs.xfs: Use the -f option to force overwrite.
2019-09-03 10:06:14 WARNING juju-log Format device '/dev/vdc' failed (Command '

$ juju status swift-proxy swift-storage-z1 swift-storage-z2 swift-storage-z3
Model Controller Cloud/Region Version SLA Timestamp
mojo-b9a2fccb gnuoy-serverstack serverstack/serverstack 2.5.4 unsupported 13:23:35Z

App Version Status Scale Charm Store Rev OS Notes
swift-hacluster active 3 hacluster local 12 ubuntu
swift-proxy 2.21.0 blocked 3 swift-proxy local 147 ubuntu
swift-storage-z1 2.21.0 waiting 1 swift-storage local 90 ubuntu
swift-storage-z2 2.21.0 waiting 1 swift-storage local 91 ubuntu
swift-storage-z3 2.21.0 waiting 1 swift-storage local 92 ubuntu

Unit Workload Agent Machine Public address Ports Message
swift-proxy/0 blocked executing 52 10.5.0.72 8080/tcp (config-changed) Not enough storage zones for minimum replicas
  swift-hacluster/2 active idle 10.5.0.72 Unit is ready and clustered
swift-proxy/1 blocked idle 53 10.5.0.58 8080/tcp Not enough storage zones for minimum replicas
  swift-hacluster/1 active idle 10.5.0.58 Unit is ready and clustered
swift-proxy/2* blocked idle 54 10.5.0.74 8080/tcp Not enough storage zones for minimum replicas
  swift-hacluster/0* active idle 10.5.0.74 Unit is ready and clustered
swift-storage-z1/0* waiting idle 55 10.5.0.81 Incomplete relations: proxy
swift-storage-z2/0* waiting idle 56 10.5.0.73 Incomplete relations: proxy
swift-storage-z3/0* waiting idle 57 10.5.0.77 Incomplete relations: proxy

Machine State DNS Inst id Series AZ Message
52 started 10.5.0.72 8ff6e455-a605-4d9d-8597-640ad1bc87fa disco nova ACTIVE
53 started 10.5.0.58 e6fc75cc-b94b-4761-a397-b726896d8af3 disco nova ACTIVE
54 started 10.5.0.74 41e94d83-5e9a-4a2d-8dd5-7a2fdc70c284 disco nova ACTIVE
55 started 10.5.0.81 46a13c50-3cb2-446c-89bf-e040d673e6ab disco nova ACTIVE
56 started 10.5.0.73 df197d5e-30e3-4228-852f-97cc18d06835 disco nova ACTIVE

Revision history for this message
Camille Rodriguez (camille.rodriguez) wrote :

This issue was hit in this review as well https://review.opendev.org/#/c/678945/2 , ci report here : https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-swift-storage/678945/2/3795/index.html.

The swift proxy also complains of not enough storage zones for minimum replicas.

DEBUG:runner: swift-proxy:
DEBUG:runner: charm: cs:~openstack-charmers-next/swift-proxy-170
DEBUG:runner: series: disco
DEBUG:runner: os: ubuntu
DEBUG:runner: charm-origin: jujucharms
DEBUG:runner: charm-name: swift-proxy
DEBUG:runner: charm-rev: 170
DEBUG:runner: exposed: false
DEBUG:runner: application-status:
DEBUG:runner: current: blocked
DEBUG:runner: message: Not enough storage zones for minimum replicas
DEBUG:runner: since: 03 Sep 2019 16:45:41Z
DEBUG:runner: relations:
DEBUG:runner: cluster:
DEBUG:runner: - swift-proxy
DEBUG:runner: identity-service:
DEBUG:runner: - keystone
DEBUG:runner: object-store:
DEBUG:runner: - glance
DEBUG:runner: swift-storage:
DEBUG:runner: - swift-storage
DEBUG:runner: units:
DEBUG:runner: swift-proxy/3:
DEBUG:runner: workload-status:
DEBUG:runner: current: blocked
DEBUG:runner: message: Not enough storage zones for minimum replicas
DEBUG:runner: since: 03 Sep 2019 16:45:41Z
DEBUG:runner: juju-status:
DEBUG:runner: current: idle
DEBUG:runner: since: 03 Sep 2019 16:48:13Z
DEBUG:runner: version: 2.6.8
DEBUG:runner: leader: true
DEBUG:runner: machine: "18"
DEBUG:runner: open-ports:
DEBUG:runner: - 8080/tcp
DEBUG:runner: public-address: 172.17.110.17
DEBUG:runner: version: 2.21.0
DEBUG:runner: endpoint-bindings:
DEBUG:runner: admin: ""
DEBUG:runner: amqp: ""
DEBUG:runner: certificates: ""
DEBUG:runner: cluster: ""
DEBUG:runner: ha: ""
DEBUG:runner: identity-service: ""
DEBUG:runner: internal: ""
DEBUG:runner: nrpe-external-master: ""
DEBUG:runner: object-store: ""
DEBUG:runner: public: ""
DEBUG:runner: swift-storage: ""

Changed in charm-swift-storage:
status: New → Triaged
importance: Undecided → Wishlist
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.