OpenStack Core Infrastructure

logstash results don't always match up with actual logs

Bug #1225108 reported by Joe Gordon on 2013-09-13

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Core Infrastructure	Fix Released	Medium	Clark Boylan	OpenStack Core Infrastructure havana

Bug Description

the following logstash query (@message:"ConnectionFailed: Connection to neutron failed: Maximum attempts reached" AND @fields.filename:"console.html" AND @fields.build_change:"45424" AND @fields.build_patchset:"8")

produces a hit on 2013-09-13T10:24:46.000 but when looking at the original log file (http://logs.openstack.org/24/45424/8/check/gate-tempest-devstack-vm-neutron/5f4e9d1/console.html) the message string is not there.

Clark Boylan (cboylan) on 2013-09-13

Changed in openstack-ci:
assignee:	nobody → Clark Boylan (cboylan)

Revision history for this message

Clark Boylan (cboylan) wrote on 2013-09-13:

Bad logstash event Edit (550 bytes, text/csv)

Attached is the bad logstash event in csv as exported by Kibana.

Revision history for this message

Clark Boylan (cboylan) wrote on 2013-09-13:

The log file linked to above was for a different test run on the same change and patchset. The test run that produced this failure is http://logs.openstack.org/24/45424/8/check/gate-tempest-devstack-vm-neutron/0150828/console.html. The bug here is that logstash/elasticsearch are not keeping track of the build log path or the build UUID so making a mapping from logstash to log file on the log server is difficult.

To fix this we should carry the build UUID in our log events and/or just include the entire path to the log file in the logstash log events.

Revision history for this message

Clark Boylan (cboylan) wrote on 2013-10-22:

This was fixed by including build UUIDs in the logstash events. Now each log event is uniquely traceable back to its log file and job run.