Fuel for OpenStack

second rsyslog server definition breaks logs to master

Bug #1339659 reported by Matthew Mosesohn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Matthew Mosesohn
Fuel for OpenStack 5.1
5.0.x
Fix Committed
High
Matthew Mosesohn
Fuel for OpenStack 5.0.2

Bug Description

When a second rsyslog server is configured, no logs are sent to the first server. This is causing issues in our CI tests.

From http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html :
# start forwarding rule 1
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd1 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server1:port
# end forwarding rule 1

# start forwarding rule 2
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd2 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server2
# end forwarding rule 2

This should fix deployed nodes logging correctly to both specified rsyslog servers.

Tags: docs library
Matthew Mosesohn (raytrac3r)
Changed in fuel:
assignee: nobody → Matthew Mosesohn (raytrac3r)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/105719

Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Fix in 105719 works to eventually discard messages sent via TCP to a nonresponsive rsyslog server. The best option is to use UDP to avoid this scenario. After a couple of MB of logs get generated, all old messages get collected and sent to Fuel Master. This will repeat indefinitely, though, distributing logs on and off in spurts. It is still a better option than completely broken logging when 2nd server is broken.

tags: added: docs
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/105719
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=aef7924742d0b10b5f5cb944e917a66a9325d0f6
Submitter: Jenkins
Branch: master

commit aef7924742d0b10b5f5cb944e917a66a9325d0f6
Author: Matthew Mosesohn <email address hidden>
Date: Wed Jul 9 16:10:41 2014 +0400

    Fix logging to multiple remote rsyslog servers

    Change-Id: Ie248d795a1cf0e476bde8b6c2dad1fac0b660f79
    Closes-Bug: #1339659

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114270

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/114458

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/5.0)

Related fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114459

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/114458
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=df3ee5b1c3ca29ad8bbf4f570217b0dbe3c81dd7
Submitter: Jenkins
Branch: master

commit df3ee5b1c3ca29ad8bbf4f570217b0dbe3c81dd7
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Aug 15 10:40:41 2014 +0300

    Align ActionQueue rsyslog params in templates

    The changes accepted in fix should be aligned
    with other templates we use for client.conf and
    server.conf then installing and configuring rsyslog.

    Related-bug: #1339659

    Change-Id: I9108b0fb992f3dd5a68f3b9d39d78705e243d9fb
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/114270
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=fd457f904b890e8e5891c3f24fd16cf82a5dd350
Submitter: Jenkins
Branch: stable/5.0

commit fd457f904b890e8e5891c3f24fd16cf82a5dd350
Author: Matthew Mosesohn <email address hidden>
Date: Wed Jul 9 16:10:41 2014 +0400

    Fix logging to multiple remote rsyslog servers

    Change-Id: Ie248d795a1cf0e476bde8b6c2dad1fac0b660f79
    Closes-Bug: #1339659

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/114459
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d35ded526370de29b620b4de53331f846c057b64
Submitter: Jenkins
Branch: stable/5.0

commit d35ded526370de29b620b4de53331f846c057b64
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Aug 15 10:40:41 2014 +0300

    Align ActionQueue rsyslog params in templates

    The changes accepted in fix should be aligned
    with other templates we use for client.conf and
    server.conf then installing and configuring rsyslog.

    Related-bug: #1339659

    Change-Id: I9108b0fb992f3dd5a68f3b9d39d78705e243d9fb
    Signed-off-by: Bogdan Dobrelya <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.