I've observed that a service which is OKAY can flap to CRITICAL sometimes. Depending on when the request is made, the "group by" returns 'null' and grafana panel interpret it as CRITICAL.
request/response:
==============
SELECT last("value") FROM "haproxy_backend_status" WHERE "hostname" =~ /node-110/ AND time > now() - 5m GROUP BY time(1m)
I've observed that a service which is OKAY can flap to CRITICAL sometimes. Depending on when the request is made, the "group by" returns 'null' and grafana panel interpret it as CRITICAL.
request/response: backend_ status" WHERE "hostname" =~ /node-110/ AND time > now() - 5m GROUP BY time(1m)
==============
SELECT last("value") FROM "haproxy_
{"results" :[{"series" :[{"name" :"haproxy_ backend_ status" ,"columns" :["time" ,"last" ],"values" :[[145719360000 0,1],[145719366 0000,1] ,[1457193720000 ,1],[1457193780 000,1], [1457193840000, 1],[14571939000 00,null] ]}]}]}
Solution
=======
add fill(previous) to avoid null results