Comment 18 for bug 1415954

 SCALE LAB (50 ceph-osd) results:
patch fixes the problem and showed good itself even when was warn due to clock skew

root@node-101:~# ceph -s
    cluster e9df6ff0-ccb8-4a51-9d56-a6f92c85c921
     health HEALTH_WARN clock skew detected on mon.node-102, mon.node-103
     monmap e3: 3 mons at {node-101=192.168.0.4:6789/0,node-102=192.168.0.5:6789/0,node-103=192.168.0.6:6789/0}, election epoch 6, quorum 0,1,2 node-101,node-102,node-103
     osdmap e47: 47 osds: 47 up, 47 in
      pgmap v979: 16576 pgs, 11 pools, 13696 kB data, 51 objects
            98513 MB used, 43401 GB / 43498 GB avail
               16576 active+clean

task logs:

2015-03-13T18:23:26 info: [424] Run hook ---
priority: 300
fail_on_error: true
type: shell
uids:
- '101'
parameters:
  cmd: ruby /etc/puppet/modules/osnailyfacter/modular/astute/ceph_ready_check.rb
  timeout: 1800

2015-03-13T18:26:18 debug: [424] 954bd275-f135-44bf-8cb8-d8f5224d69eb: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"
101", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"try 0\nThere are PGs which are not in active state!\ntry 1\nThere are PGs which are not in active state!\ntry 2\nThere are PGs which are not in active state!\ntry 3\nThere are PGs which are not in active state!\ntry 4\nThere are PGs which are not in active state!\ntry 5\nThere are PGs which are not in active state!\ntry 6\nThere are PGs which are not in active state!\ntry 7\nThere are PGs which are not in active state!\ntry 8\nThere are PGs which are not in active state!\ntry 9\nThere are PGs which are not in active state!\ntry 10\nThere are PGs which are not in active state!\ntry 11\nThere are PGs which are not in active state!\ntry 12\nThere are PGs which are not in active state!\ntry 13\nThere are PGs which are not in active state!\ntry 14\nThere are PGs which are not in active state!\ntry 15\n", :stderr=>"ok\nok\nok\nok\nok\nok\nok\nok\nok\nok\nok\nok\nok\nok\nok\n", :exit_code=>0}}
2015-03-13T18:26:18 debug: [424] 954bd275-f135-44bf-8cb8-d8f5224d69eb: cmd: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/ceph_ready_check.rb
cwd: /
stdout: try 0
There are PGs which are not in active state!
try 1
There are PGs which are not in active state!
try 2
There are PGs which are not in active state!
try 3
There are PGs which are not in active state!
try 4
There are PGs which are not in active state!
try 5
There are PGs which are not in active state!
try 6
There are PGs which are not in active state!
try 7
There are PGs which are not in active state!
try 8
There are PGs which are not in active state!
try 9
There are PGs which are not in active state!
try 10
There are PGs which are not in active state!
try 11
There are PGs which are not in active state!
try 12
There are PGs which are not in active state!
try 13
There are PGs which are not in active state!
try 14
There are PGs which are not in active state!
try 15

stderr: ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok

exit code: 0
2015-03-13T18:26:18 info: [424] Run hook ---
priority: 400