I wonder if there is a race between partitioning a disk and expecting the corresponding devices to be created. Case in point:
unit-ceph-osd-4[27573]: 2016-03-14 11:36:10 INFO unit.ceph-osd/4.juju-log server.go:268 mon:72: osdize cmd: ['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']
unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Creating new GPT entries.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 GPT data structures destroyed! You may now partition the disk using fdisk or
unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 other utilities.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:16 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Creating new GPT entries.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:16 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:18 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 mkfs.xfs: cannot open /dev/sdb1: No such device or address
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 ceph-disk: Error: Command '['/sbin/mkfs', '-t', 'xfs', '-f', '-i', 'size=2048', '--', '/dev/sdb1']' returned non-zero exit status 1
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 ERROR unit.ceph-osd/4.juju-log server.go:268 mon:72: Unable to initialize device: /dev/sdb
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 312, in <module>
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 hooks.execute(sys.argv)
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 self._hooks[hook_name]()
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 254, in mon_relation
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 prepare_disks_and_activate()
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 185, in prepare_disks_and_activate
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 config('osd-encrypt'))
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/ceph.py", line 435, in osdize
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 reformat_osd, ignore_errors, encrypt)
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/ceph.py", line 490, in osdize_dev
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 raise e
unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 subprocess.CalledProcessError: Command '['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']' returned non-zero exit status 1
/dev/sdb is a normal rotary disk on this node. Partitioning went fine at :14 and :16, but at timestamp :20 mkfs complained that /dev/sdb1 does not exist. Should we wait there perhaps?
cs:~openstack- charmers- next/trusty/ ceph-osd- 81
I wonder if there is a race between partitioning a disk and expecting the corresponding devices to be created. Case in point: osd-4[27573] : 2016-03-14 11:36:10 INFO unit.ceph- osd/4.juju- log server.go:268 mon:72: osdize cmd: ['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb'] osd-4[27573] : 2016-03-14 11:36:14 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 Creating new GPT entries. osd-4[27573] : 2016-03-14 11:36:14 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 GPT data structures destroyed! You may now partition the disk using fdisk or osd-4[27573] : 2016-03-14 11:36:14 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 other utilities. osd-4[27573] : 2016-03-14 11:36:16 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 Creating new GPT entries. osd-4[27573] : 2016-03-14 11:36:16 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 The operation has completed successfully. osd-4[27573] : 2016-03-14 11:36:18 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 The operation has completed successfully. osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 The operation has completed successfully. osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 mkfs.xfs: cannot open /dev/sdb1: No such device or address osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 ceph-disk: Error: Command '['/sbin/mkfs', '-t', 'xfs', '-f', '-i', 'size=2048', '--', '/dev/sdb1']' returned non-zero exit status 1 osd-4[27573] : 2016-03-14 11:36:20 ERROR unit.ceph- osd/4.juju- log server.go:268 mon:72: Unable to initialize device: /dev/sdb osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 Traceback (most recent call last): osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- osd-4/charm/ hooks/mon- relation- changed" , line 312, in <module> osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 hooks.execute( sys.argv) osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- osd-4/charm/ hooks/charmhelp ers/core/ hookenv. py", line 717, in execute osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 self._hooks[ hook_name] () osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- osd-4/charm/ hooks/mon- relation- changed" , line 254, in mon_relation osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 prepare_ disks_and_ activate( ) osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- osd-4/charm/ hooks/mon- relation- changed" , line 185, in prepare_ disks_and_ activate osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 config( 'osd-encrypt' )) osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- osd-4/charm/ hooks/ceph. py", line 435, in osdize osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 reformat_osd, ignore_errors, encrypt) osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- osd-4/charm/ hooks/ceph. py", line 490, in osdize_dev osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 raise e osd-4[27573] : 2016-03-14 11:36:20 INFO unit.ceph- osd/4.mon- relation- changed logger.go:40 subprocess. CalledProcessEr ror: Command '['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']' returned non-zero exit status 1
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
unit-ceph-
/dev/sdb is a normal rotary disk on this node. Partitioning went fine at :14 and :16, but at timestamp :20 mkfs complained that /dev/sdb1 does not exist. Should we wait there perhaps?