Remote syslog logging paralyzes system

Bug #26986 reported by Peter Cherriman
14
Affects Status Importance Assigned to Milestone
sysklogd (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

I tried to get ubuntu (breezy&hoary) to do remote syslog logging from my ADSL router.
I found that I needed to modify the line:

SYSLOGD="-u syslog"
in /etc/init.d/sysklogd
However I found if I changed it to:
SYSLOGD="-r -u syslog"
then logins toke forever, and reboots or shutdown never finish.
if I change it to
SYSLOGD="-u syslog -r"
everything works perfectly.

I've no idea why the "-r" before the "-u" causes the slow logins and shutdown.

Revision history for this message
Matt Zimmerman (mdz) wrote :

Very weird...

Changed in sysklogd:
assignee: nobody → ogra
Revision history for this message
Nikolaus Rath (nikratio) wrote :

I can confirm that login etc takes forever with -r, but it does not depend on the order of the parameters.

Changed in sysklogd:
status: Unconfirmed → Confirmed
Revision history for this message
Peter Cherriman (pjcherriman) wrote : Re: [Bug 26986] Re: Remote syslog logging (parameter order crucial)

--- Nikolaus Rath <email address hidden> wrote:
> I can confirm that login etc takes forever with -r,
> but it does not
> depend on the order of the parameters.
>
> ** Changed in: sysklogd (Ubuntu)
> Status: Unconfirmed => Confirmed

Well the order was crucial for me.

In /etc/init.d/sysklogd
SYSLOGD="-u syslog -r"
works for me.

I'm able to use the -r option without the login
problem, if I use the above configuration.

I've recently upgraded to dapper, with the same config
file and remote logging is still working. I can check
to see if the order still makes a different for me on
dapper if you want. Just let me know.

Peter Cherriman <email address hidden>

___________________________________________________________
Win tickets to the 2006 FIFA World Cup Germany with Yahoo! Messenger. http://advision.webevents.yahoo.com/fifaworldcup_uk/

Revision history for this message
Nikolaus Rath (nikratio) wrote : Re: Remote syslog logging (parameter order crucial)

My version is

[0] nokile:~$ dpkg -s sysklogd |grep Version
Version: 1.4.1-17ubuntu7

and remote logging doesn't work.

Revision history for this message
Peter Cherriman (pjcherriman) wrote :

Same version as my ubuntu pc (dapper)

pjc@ubuntu:~$ dpkg -s sysklogd |grep Version
Version: 1.4.1-17ubuntu7

pjc@ubuntu:~$ diff /etc/init.d/sysklogd /etc/init.d/sysklogd.dpkg-dist
15c15
< SYSLOGD="-u syslog -r"
---
> SYSLOGD="-u syslog"

Extract from /etc/syslog.conf:
# Logging ADSL router
local0.* /var/log/adslrouter-firewall.log
local1.* -/var/log/adslrouter-vpn.log
local2.* -/var/log/adslrouter-user.log
local3.* -/var/log/adslrouter-call.log
local4.* -/var/log/adslrouter-wan.log
local5.* -/var/log/adslrouter-adsl.log
local0,local1,local2,local3,local4,local5.* -/var/log/adslrouter.log

I've justed checked and while I was seeing syslog entries (for the current day) from a remote device (my ADSL router) but data seems to be missing after the firewall started.

When I checked further, and found that the firestarter firewall was now blocking the syslog udp packets, when the breezy version didn't.

When I enable a firewall rule for the syslog packets from the ADSL router, my syslog started working fully again.

Revision history for this message
Mathias Gug (mathiaz) wrote :

@Nikolaus: Do you still have the problem ? Did you have a firewall configured on the syslogd machine ?

Changed in sysklogd:
status: Confirmed → Incomplete
Revision history for this message
Nikolaus Rath (nikratio) wrote :

I have no second machine available anymore to test the remote logging. The problem of delayed logins seems resolved though.

Revision history for this message
Holger Krause (holger-krause) wrote :

With kubuntu 6.10 I experienced similiar problems when directing the syslog-stream of my DSL-router to the PC: Booting took several minutes and also logging in and off, if I remember correctly. I didn't bother with parameter ordering, but used the facility in /etc/defaults/syslogd to add th '-r' option:

#
# For remote UDP logging use SYSLOGD="-r"
#
SYSLOGD="-r"

This works without any delays with kubuntu 7.04.

Revision history for this message
Nikolaus Rath (nikratio) wrote :

I am now seeing the problem again on a different machine. Adding "-r" to the syslog options in /etc/default/syslogd causes logins to take forever. I am not using a firewall. I receive remote syslog messages from an ADSL router.

Changed in sysklogd:
status: Incomplete → Confirmed
Nikolaus Rath (nikratio)
description: updated
Revision history for this message
Nikolaus Rath (nikratio) wrote :

Addendum: I'm using Ubuntu hardy and the parameter order is *not* crucial. It doesn't matter whether I add -r before or after -u in /etc/init.d/sysklogd. As soon as I add -r, logins take forever.

Revision history for this message
Oliver Grawert (ogra) wrote :

try adding a fake entry for the IP of the remote machine to your syslog servers /etc/hosts file and check if that speeds up things, i suspect since some time that syslog forcefully performs a DNS lookup *per line* while logging ...

btw: afaik the server team plans to abandon syslog in intrepid

Revision history for this message
Nikolaus Rath (nikratio) wrote :

Why should syslog perform a dns lookup when I'm logging in on the *local* machine? I don't see any connection, but I'm going to try your suggestion anyway.

Revision history for this message
Nikolaus Rath (nikratio) wrote :

I can't really believe it, but adding the fake entry to /etc/hosts really fixed the problem. So there seems to be something wrong with the way syslogd does the DNS lookups.

Revision history for this message
Carl Gilbert (dnoyeb) wrote :

Exactly how did the fake entry look in etc/hosts? just an ip to name match?
I believe I am having this same problem. This is my first time with Ubuntu. Since I installed in December, the system eventually locks up. Basically the logins lock up. I have remote logging enabled.

I leave a terminal logged as root so when the system locks up I go to the terminal and restart sysklogd. It starts working again. I guess if I look in my logs I can figure out which of my remote logging items is causing the freezup.

Revision history for this message
Nikolaus Rath (nikratio) wrote :

Yeah, it's just an ordinary entry of the form

[ip of remote machine that logs] [some random name]

Revision history for this message
Carl Gilbert (dnoyeb) wrote :

I added a hosts entry and that fixed the slow login part. Now my ssh logins are ultra fast like they used to be with Mandriva. Also, aren't we talking about reverse-DNS lookups here? The hosts I have that are logging in and that are using remote logging, are in my local domain. So reverse lookups will fail.

Anyway, it appears that at some point something happens that makes the lookup stall, and that stalls sysklogd which means anyone that blocks on a write to syslog can't continue. Or so it would appear.

I can't be sure this fixes the total stall yet.

Revision history for this message
Loïc Minier (lool) wrote :

Updating assignee; this bug seems important but might disappear in the default Ubuntu install if we switch to rsyslog

Changed in sysklogd (Ubuntu):
assignee: Oliver Grawert (ogra) → nobody
Revision history for this message
David Olivari (david-olivari) wrote :

I had the same experience of very slow remote syslog. I switch to rsyslog with no improvement.
I found that if rsyslog try to dns resolv ip so I put a static entry for my log sender ip in /etc/hosts and it solve the problem. Then I switch back to the classic ksyslogd with success.
There might an option to deactivate resolving but I didn't found a simple way

Revision history for this message
Denilson Sá (denilsonsa) wrote : Re: [Bug 26986] Re: Remote syslog logging paralyzes system

On Tue, 04 Aug 2009 05:26:34 -0300, David Olivari
<email address hidden> wrote:
> There might an option to deactivate resolving but I didn't found a
> simple way

/etc/default/rsyslog

# Options to rsyslogd
# -x disables DNS lookups on messages recieved with -r

--
Denilson Figueiredo de Sá
Vialink
Rio de Janeiro - Brasil

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.