Comment 0 for bug 1557158

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

cs:~openstack-charmers-next/trusty/ceph-osd-81

I wonder if there is a race between partitioning a disk and expecting the corresponding devices to be created. Case in point:
unit-ceph-osd-4[27573]: 2016-03-14 11:36:10 INFO unit.ceph-osd/4.juju-log server.go:268 mon:72: osdize cmd: ['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']
unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Creating new GPT entries.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 GPT data structures destroyed! You may now partition the disk using fdisk or
unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 other utilities.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:16 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Creating new GPT entries.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:16 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:18 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 mkfs.xfs: cannot open /dev/sdb1: No such device or address
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 ceph-disk: Error: Command '['/sbin/mkfs', '-t', 'xfs', '-f', '-i', 'size=2048', '--', '/dev/sdb1']' returned non-zero exit status 1
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 ERROR unit.ceph-osd/4.juju-log server.go:268 mon:72: Unable to initialize device: /dev/sdb
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 312, in <module>
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 hooks.execute(sys.argv)
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 self._hooks[hook_name]()
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 254, in mon_relation
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 prepare_disks_and_activate()
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 185, in prepare_disks_and_activate
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 config('osd-encrypt'))
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/ceph.py", line 435, in osdize
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 reformat_osd, ignore_errors, encrypt)
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/ceph.py", line 490, in osdize_dev
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 raise e
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 subprocess.CalledProcessError: Command '['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']' returned non-zero exit status 1

/dev/sdb is a normal rotary disk on this node. Partitioning went fine at :14 and :16, but at timestamp :20 mkfs complained that /dev/sdb1 does not exist. Should we wait there perhaps?