some files are not getting added to logstash.openstack.org

http://logs.openstack.org/77/109177/2/gate/gate-tempest-dsvm-full/97f2cde/logs/

I think part of the issue here is the * isn't working as expected in tags? build_short_uuid:"97f2cde" AND NOT filename:"console.html" AND NOT filename:"logs/syslog.txt" AND NOT filename:"logs/screen-n-cond.txt" AND NOT filename:"logs/screen-g-reg.txt" AND NOT filename:"logs/screen-h-api-cw.txt" is a query that returns 0 results and the files NOTed out there do return data.

From logstash-worker04 C worker:

2014-08-18 16:46:55,412 Unable to get log data.

I think logs.o.o's apache process was unable to serve those files at that time for some reason. We should see if we can catch this in action.

After looking on the logs.o.o apache side there is nothing logged in the access log for these requests. This is potentially network disruption between logs.o.o and our logstack worker nodes.

Please update this bug if we can find more incidences of it.