I faced with same issue and did small investigation
Looks like at least in my case for some reason firewall on elasticsearch node was not configured in correct way.
I was not able to connect to elastic search --> heka was not able to get messages --> collectd was not able to send messages
I re-applyed firewall.pp manually:
cd /etc/fuel/plugins/elasticsearch_kibana-0.8/ && puppet apply -d -v --modulepath=puppet/modules:/etc/puppet/modules puppet/manifests/firewall.pp
Diff between iptables config before and after:
diff /root/iptables-save /root/iptables-save-1
1c1
< # Generated by iptables-save v1.4.21 on Mon Feb 29 17:32:47 2016
---
> # Generated by iptables-save v1.4.21 on Mon Feb 29 17:34:53 2016
4,5c4,5
< :FORWARD ACCEPT [19:3138]
< :OUTPUT ACCEPT [36633:56370787]
---
> :FORWARD ACCEPT [1:68]
> :OUTPUT ACCEPT [256:179808]
10a11,12
> -A INPUT -p tcp -m multiport --ports 9200 -m comment --comment "100 elasticsearch" -j ACCEPT
> -A INPUT -p tcp -m multiport --ports 80 -m comment --comment "101 kibana" -j ACCEPT
16c18
< # Completed on Mon Feb 29 17:32:47 2016
---
> # Completed on Mon Feb 29 17:34:53 2016
After that I restarted collectd (kill -9 to stop it) and now it looks OK.
So for the first view it looks like puppet-related issue, not heka or collectd
I faced with same issue and did small investigation
Looks like at least in my case for some reason firewall on elasticsearch node was not configured in correct way.
I was not able to connect to elastic search --> heka was not able to get messages --> collectd was not able to send messages
I re-applyed firewall.pp manually:
cd /etc/fuel/ plugins/ elasticsearch_ kibana- 0.8/ && puppet apply -d -v --modulepath= puppet/ modules: /etc/puppet/ modules puppet/ manifests/ firewall. pp
Diff between iptables config before and after:
diff /root/iptables-save /root/iptables- save-1
1c1
< # Generated by iptables-save v1.4.21 on Mon Feb 29 17:32:47 2016
---
> # Generated by iptables-save v1.4.21 on Mon Feb 29 17:34:53 2016
4,5c4,5
< :FORWARD ACCEPT [19:3138]
< :OUTPUT ACCEPT [36633:56370787]
---
> :FORWARD ACCEPT [1:68]
> :OUTPUT ACCEPT [256:179808]
10a11,12
> -A INPUT -p tcp -m multiport --ports 9200 -m comment --comment "100 elasticsearch" -j ACCEPT
> -A INPUT -p tcp -m multiport --ports 80 -m comment --comment "101 kibana" -j ACCEPT
16c18
< # Completed on Mon Feb 29 17:32:47 2016
---
> # Completed on Mon Feb 29 17:34:53 2016
After that I restarted collectd (kill -9 to stop it) and now it looks OK.
So for the first view it looks like puppet-related issue, not heka or collectd