Thanks Simon!  Great work on this!
 
> Date: Fri, 28 Aug 2015 08:56:26 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1488717] Re: Controller looses connection to elasticserch/kibana
> 
> This is due to a bug [1] in Heka affecting the management of the output
> queue. The issue is fixed in the 0.10 branch but we'll have to wait for
> the next beta release of 0.10 (or the official 0.10 release). IIUC it
> should be a matter of days.
> 
> [1] https://github.com/mozilla-services/heka/issues/1627
> 
> -- 
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1488717
> 
> Title:
>   Controller looses connection to elasticserch/kibana
> 
> Status in Fuel Plugins:
>   Triaged
> 
> Bug description:
>   When deploying cluster with elasticsearch/kibana node - notice after
>   about 24-36 hours the primary controller just falls off the Kibana
>   Dashboard and no messages from the node are revieved at all, then it
>   only shows the last 2 controllers.
> 
>   
>   elasticsearch/kibana node has 32GB of memory, using 13GB for JVM or roughly 40% of total system memory for JVM
> 
>   Primary Controller 
>   #/var/log/lma_collector.log
> 
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:47 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:48 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:48 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:48 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:48 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:48 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:48 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
>   2015/08/25 23:27:48 Plugin 'elasticsearch_output' error: can't deliver matched message: Queue is full
> 
>   Primary Controller has 40GB free available memory,
> 
>   Thanks, Ryan
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel-plugins/+bug/1488717/+subscriptions