Messages not being sent to system logs

Thank you for taking the time to report this bug and helping to make Ubuntu better. Judging from your description, I am going to assign this bug to the rsyslog package in hopes that it will have a faster response. If you believe that I have chosen the wrong package, please change your bug to the correct one. Good luck on getting it fixed!

The previous syslog was from a desktop.
I'm attaching another one from a VirtualBox VM.

I'm seeing this as well. When the rsyslog is getting the new 'lightweight' restart command the log files are not being reopened properly and nothing gets logged from the kernel.

It makes debugging a right pain.

I tried to reproduce this with a 'sudo pkill -1 rsyslog'. I got the message in /var/log/syslog about a lightweight restart, but it kept logging messages. So I couldn't reproduce.

Does a pkill -1 break rsyslog for you, Neil or Vajra? Do either of you know what triggered the HUP?

No idea what's causing it. Mostly I see the restart message on boot
but sometimes I don't.

I feel this is going to be one of those irritating bugs that's
difficult to track down...

2009/8/10 Michael Terry <email address hidden>:
> I tried to reproduce this with a 'sudo pkill -1 rsyslog'.  I got the
> message in /var/log/syslog about a lightweight restart, but it kept
> logging messages.  So I couldn't reproduce.
>
> Does a pkill -1 break rsyslog for you, Neil or Vajra?  Do either of you
> know what triggered the HUP?
>
> --
> [karmic] Messages not being sent to system logs
> https://bugs.launchpad.net/bugs/407862
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Neil Wilson

I tried the command 'sudo pkill -1 rsyslog' in my VirtualBox installation but couldn't see any effect. I have no idea about what triggered the HUP. It just happens to be the last syslog message most of the time, but not always. Sometimes the syslog stops before that message.

I did a check this morning and the pkill -HUP does stop the logging -
if it is actually still working at that point in time.

I also noticed that the logging will just stop *without* seeing the
HUP message. dmesg continues merrily, but nothing appears in
/var/log/syslog or /var/log/messages.

All very strange.

2009/8/10 Neil Wilson <email address hidden>:
> No idea what's causing it. Mostly I see the restart message on boot
> but sometimes I don't.
>
> I feel this is going to be one of those irritating bugs that's
> difficult to track down...
>
> 2009/8/10 Michael Terry <email address hidden>:
>> I tried to reproduce this with a 'sudo pkill -1 rsyslog'.  I got the
>> message in /var/log/syslog about a lightweight restart, but it kept
>> logging messages.  So I couldn't reproduce.
>>
>> Does a pkill -1 break rsyslog for you, Neil or Vajra?  Do either of you
>> know what triggered the HUP?
>>
>> --
>> [karmic] Messages not being sent to system logs
>> https://bugs.launchpad.net/bugs/407862
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>>
>
>
>
> --
> Neil Wilson
>

--
Neil Wilson

Ok problem#1 sorted. The new rsyslog system is starting up dd without the correct blocksize indicator and that is causing everything to hang up.

The new rsyslog init.d file uses

  # shovel /proc/kmsg to pipe readable by syslog user
  start-stop-daemon --start --pidfile $KMSG_PIDFILE --exec /bin/dd -b -m -- if=/proc/kmsg of=$KMSG_PIPE

The old klogd init.d file uses

  # shovel /proc/kmsg to pipe readable by klogd user
  start-stop-daemon --start --pidfile $kmsgpidfile --exec /bin/dd -b -m -- bs=1 if=/proc/kmsg of=$kmsgpipe

Adding 'bs=1' to the rsyslog init.d file cures the lack of kernel logging. Patch attached which also implements HUP based reload for this daemon.

This bug was fixed in the package rsyslog - 4.2.0-2ubuntu1

---------------
rsyslog (4.2.0-2ubuntu1) karmic; urgency=low

  [ Michael Terry ]
  * Merge from debian unstable (LP: #413023), remaining changes:
    - Run as rsyslog:rsyslog
    - Allow reading /proc/kmsg when non-root
    - Cleanly upgrade from sysklogd
  * debian/patches/deroot.patch: Don't allow using the klogctl function to
    read klog messages. Rather, allow /proc/kmsg or nothing, since we have
    special support for reading /proc/kmsg while unprivileged.

  [ Neil Wilson ]
  * debian/rsyslog.init: Set blocksize for dd (LP: #407862) and restore
    reload init argument to original lightweight reload

 -- Michael Terry <email address hidden> Thu, 13 Aug 2009 15:43:29 -0400

Sorry, but it still isn't fixed for me.

$ apt-cache policy rsyslog
rsyslog:
  Instalado: 4.2.0-2ubuntu1
  Candidato: 4.2.0-2ubuntu1
  Tabela de Versão:
 *** 4.2.0-2ubuntu1 0
        500 http://ports.ubuntu.com karmic/main Packages
        100 /var/lib/dpkg/status

$ lsb_release -rd
Description: Ubuntu karmic (development branch)
Release: 9.10

Aug 14 14:17:38 jbs-lpia kernel: [ 15.636268] groups: 0-1 (__cpu_power = 2048)
Aug 14 14:17:38 jbs-lpia rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="2362" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

I have to restart rsyslogd before messages start showing up in system logs again.

Same here, but at least the kern log works now and you can get the
logging working via a restart.

Now to find out what is causing that HUP and why the files aren't
opening properly.

2009/8/14 Jose Bernardo <email address hidden>:
> Sorry, but it still isn't fixed for me.

--
Neil Wilson

Doh, this is so obvious it's embarrassing.

rsyslog is dropping privileges to 'rsyslog:rsyslog'

/var/log is set as follows:

drwxr-xr-x 14 root root 4096 2009-08-16 07:20 /var/log

The files appear to be created before the privileges are dropped - hence why it seems to work for a bit and then stops.

-rw-r----- 1 root adm 88 2009-08-16 08:39 kern.log
-rw-r----- 1 root adm 354 2009-08-16 08:39 syslog
-rw-r----- 1 root adm 354 2009-08-16 08:39 messages

Plus the console has the wrong permissions.

prw-r----- 1 root adm 0 2009-08-16 08:47 /dev/xconsole

And of course when you HUP the syslog daemon it reopens the files and gets permission denied.

So we need to get the permissions sorted out - and figure out why rsyslog is able to open the log files are root.

Which package does the initial create on /var/log?

Of course if the design is that files are opened and then permissions are dropped (as seems to be the case on further reading: http://wiki.rsyslog.com/index.php/Security#Dropping_Privileges), then HUP needs to be ignored completely somehow and the reload command in the init scripts needs reverting back to a simple alias for restart.

Guh, I thought it was safe to revert to how debian did it (sending a HUP) in the latest upload because I noticed that rsyslog forces a lightweight HUP restart when privileges are dropped. It seemed to work for me, but I didn't do extensive testing.

OK, so if the realization is that lightweight restarting is just broken with privilege dropping, then that's good to know. We just go back to the previous init reloading and your original report would just be 'WONT FIX' since we don't support manual HUPing?

However, it surprises me that lightweight restart would be broken. As I said, the code specifically chooses lightweight restarts when privileges are dropped. I would think that means it would be safe.

So Neil, with your original patch for 4.2.0-1ubuntu2, things weren't working? I had tested them, but I think I only tested kernel messages, which wouldn't be affected by the 'open as root, drop priv' issue since the kernel messages are readable by the rsyslog user.

The 'bs=1' kernel message fix helps, since otherwise kernel messages
don't get through at all. With bs=1 you get kernel messages when you
restart the daemon, but it stops again when you issue a 'HUP'.

I put some debug into the code over the weekend and lightweight
restart appears to close all the log files. They are then reopened 'on
demand' - but fail with 'permission denied.'

The lightweight restart option needs to be disabled somehow -
internally and via HUP. Either that or stop closing files in
lightweight mode. The file access code is a bit messy and it looks
hard to stop the file closes on HUP without breaking somehting else.
Perhaps upstream has a better suggestion?

I don't think the report is a 'wont fix' - we kinda need system log
messages to get through. :)

2009/8/17 Michael Terry <email address hidden>:
> Guh, I thought it was safe to revert to how debian did it (sending a
> HUP) in the latest upload because I noticed that rsyslog forces a
> lightweight HUP restart when privileges are dropped.  It seemed to work
> for me, but I didn't do extensive testing.
>
> OK, so if the realization is that lightweight restarting is just broken
> with privilege dropping, then that's good to know.  We just go back to
> the previous init reloading and your original report would just be 'WONT
> FIX' since we don't support manual HUPing?
>
> However, it surprises me that lightweight restart would be broken.  As I
> said, the code specifically chooses lightweight restarts when privileges
> are dropped.  I would think that means it would be safe.
>
> So Neil, with your original patch for 4.2.0-1ubuntu2, things weren't
> working?  I had tested them, but I think I only tested kernel messages,
> which wouldn't be affected by the 'open as root, drop priv' issue since
> the kernel messages are readable by the rsyslog user.
>
> --
> [karmic] Messages not being sent to system logs
> https://bugs.launchpad.net/bugs/407862
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Neil Wilson

I meant 'wont fix' in the sense of after we fix bs=1 and we set the init script to always use hard restarts. Then, we just say 'wont fix' to users of manual HUPs. Is there a use case for manual HUPs vs init script restarts? (serious question, are there 3rd party apps that depend on being able to HUP rsyslog for example?)

As for disabling lightweight restarts via HUPs, I guess that's the same question -- if we just ignore HUPs, will we break anything?

Yes. I think the reload needs to stay as a hard restart until upstream
refactors the code to separate the privileged and non-privileged code
base slightly more effectively.

How would the third party app know that the syslog server that is
running is rsyslog rather than syslog-ng or sysklogd? Don't they all
use different pidfile, init script names and process identifiers? Is
anything tied that closely to rsyslog?

Anything that does depend upon HUPing rsyslog should depend upon it or
its virtual package shouldn't it? Or at the very least
suggest/recommend it.

2009/8/17 Michael Terry <email address hidden>:
> I meant 'wont fix' in the sense of after we fix bs=1 and we set the init
> script to always use hard restarts.  Then, we just say 'wont fix' to
> users of manual HUPs.  Is there a use case for manual HUPs vs init
> script restarts?  (serious question, are there 3rd party apps that
> depend on being able to HUP rsyslog for example?)
>
> As for disabling lightweight restarts via HUPs, I guess that's the same
> question -- if we just ignore HUPs, will we break anything?
>
> --
> [karmic] Messages not being sent to system logs
> https://bugs.launchpad.net/bugs/407862
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Neil Wilson

OK, so I finally got time to sit down and look at this, and I can't reproduce the problem (files that rsyslog can log stop being logged after reload).

I tested normal logs by:
 * tail -f /var/log/syslog
 * sudo chown root:root /var/log/lpr.log # (-rw-r----- 1 root root 776 2009-08-31 10:40 /var/log/lpr.log)
 * sudo /etc/init.d/rsyslog restart
 * sudo /etc/init.d/cups restart
 * sudo /etc/init.d/rsyslog reload
 * sudo /etc/init.d/cups reload

And in the tail output, I saw (cupsd.conf problem is something unrelated, just good output to test rsyslog with):

Aug 31 10:40:11 bongo rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="9805" x-info="http://www.rsyslog.com"] (re)start
Aug 31 10:40:11 bongo rsyslogd: rsyslogd's groupid changed to 103
Aug 31 10:40:11 bongo rsyslogd: rsyslogd's userid changed to 102
Aug 31 10:41:32 bongo cupsd: Unable to read configuration file '/etc/cups/cupsd.conf' - exiting!
Aug 31 10:41:37 bongo rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="9805" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Aug 31 10:41:39 bongo cupsd: Unable to read configuration file '/etc/cups/cupsd.conf' - exiting!

So that appears to be working! It could read it from the start, and it could still read it after a HUP.

Then I tested kern.log:
 * tail -f /var/log/syslog
 * sudo /etc/init.d/rsyslog restart
 * plug in a usb stick
 * sudo /etc/init.d/rsyslog reload
 * pull usb stick out

And I saw:
Aug 31 10:50:32 bongo rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="10300" x-info="http://www.rsyslog.com"] (re)start
Aug 31 10:50:32 bongo rsyslogd: rsyslogd's groupid changed to 103
Aug 31 10:50:32 bongo rsyslogd: rsyslogd's userid changed to 102
Aug 31 10:50:36 bongo kernel: [249915.148097] usb 2-2: new high speed USB device using ehci_hcd and address 40
...
Aug 31 10:50:49 bongo rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="10300" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Aug 31 10:50:53 bongo kernel: [249932.212258] usb 2-2: USB disconnect, address 40

So it was also able to continue reading from kern.log after a reload.

Can you give easy reproduction steps?

For me it is busted as soon as I start up the machine unless I change
the ownership on the /var/log/* files to syslog:syslog and then it
will work fine.

I've just done the same as you after a fresh install of rsyslog to
make sure I'm on the latest repository version.

 * tail /var/log/syslog
 * sudo /etc/init.d/rsyslog restart
 * plug in usb stick
 * sudo /etc/init.d/rsyslog reload
 * pull usb stick out

I get the kernel messages after the restart but not the reload.

Is your /var/log the same permissions as default? Mine is owned
root:root mode 755 and the /var/log/syslog is owned root:adm with mode
640.

Rsyslog is supposed to close all the files on reload, so if a
syslog:syslog owned process can reopen a 640 mode file with root:adm
ownership then the kernel probably has a security hole :-)

This may be a thread sync problem. I'm on AMD 64 with 2 cores. Is
your architecture similar?

2009/8/31 Michael Terry <email address hidden>:
> OK, so I finally got time to sit down and look at this, and I can't
> reproduce the problem (files that rsyslog can log stop being logged
> after reload).

--
Neil Wilson

OK, I see the difference. My /var/log is the same as yours, but my /var/log/syslog is syslog:adm. If I chown it to root:adm, I get the same behavior as you... Now to see why/when it is root:adm (and if it should be). I suspect my syslog:adm is because rsyslog created it for me at some point. But yours may have been from sysklogd days?

I'll note that sysklogd did a rather forceful resetting of log ownership as a result of bug 120085. Before each kickoff in its init.d script, it would reset permissions of log files. Maybe we need something similar? Still digging.

My machine is a fresh install of Karmic Alpha 3 on the bare metal - no
upgrades. So the permissions are created by the current package system
and/or logrotate.

2009/8/31 Michael Terry <email address hidden>:
> OK, I see the difference.  My /var/log is the same as yours, but my
> /var/log/syslog is syslog:adm.  If I chown it to root:adm, I get the
> same behavior as you...  Now to see why/when it is root:adm (and if it
> should be).  I suspect my syslog:adm is because rsyslog created it for
> me at some point.  But yours may have been from sysklogd days?
>
> --
> [karmic] Messages not being sent to system logs
> https://bugs.launchpad.net/bugs/407862
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Neil Wilson

Looks like we want to set $FileOwner in /etc/rsyslog.conf to syslog (duh, my fault for not noticing that was still set to root) and add code to chown files if they aren't $FileOwner and $FileGroup. That should let rsyslog do its thing.

Here's a debdiff that both sets the correct $FileOwner value of syslog and makes sure that we chown all output files as we open them.

Looks fine, thanks - sponsoring.

This bug was fixed in the package rsyslog - 4.2.0-2ubuntu2

---------------
rsyslog (4.2.0-2ubuntu2) karmic; urgency=low

  * Fix log file ownership issues when HUPing an unprivileged rsyslog
    LP: #407862
    - debian/rsyslog.conf: Set $FileOwner to syslog
    - debian/patches/deroot.patch: Always chown output files, since we may
      not be able to read them on a HUP otherwise.

 -- Michael Terry <email address hidden> Mon, 31 Aug 2009 14:58:50 -0400

I have integrated the patch (actually the idea as the code has evolved in the mean time). Thanks for the suggestions. Please note that I think this patch tries to fix something outside of the scope of rsyslogd. I have posted full details in the rsyslog's bug tracker [1] as well as in the doc for the new directive [2]. Consequently, this has been changed to a "feature request" from rsyslog's POV. As such, the "patch" goes into the recent devel (4.7.0). It is rsyslog policy never to add new features to stable or beta versions.

I would appreciate discussion on my view that the root cause is outside of rsyslog. Discussion is especially important as I expect that the work-around we now created will no longer work once rsyslog has new, better privilege drop code (probably in the v5 or v6 timeframe).

Thanks,
Rainer

[1] http://bugzilla.adiscon.com/show_bug.cgi?id=150
[2] http://www.rsyslog.com/doc-rsconf1_omfileforcechown.html

I think we get into the realms of Unix philosophy at this point.

What actually needs to be fixed is the ability to set permissions on
network ports and other 'root only' system facilities. Then rsyslog
could be started as a non-privileged user and still get the facilities
it requires.

This 'obtain what you require and then drop privileges' is a nasty workaround.

To be honest I prefer the Debian workaround where a root only process
grabs the relevant data and resurfaces it via a named pipe - with the
appropriate access permissions. Perhaps rsyslog should look at
adopting that approach for access to 'root-only' facilities. It
certainly prevents security holes.

2009/9/11 Rainer Gerhards <email address hidden>:
> I have integrated the patch (actually the idea as the code has evolved
> in the mean time). Thanks for the suggestions. Please note that I think
> this patch tries to fix something outside of the scope of rsyslogd. I
> have posted full details in the rsyslog's bug tracker [1] as well as in
> the doc for the new directive [2]. Consequently, this has been changed
> to a "feature request" from rsyslog's POV. As such, the "patch" goes
> into the recent devel (4.7.0). It is rsyslog policy never to add new
> features to stable or beta versions.
>
> I would appreciate discussion on my view that the root cause is outside
> of rsyslog. Discussion is especially important as I expect that the
> work-around we now created will no longer work once rsyslog has new,
> better privilege drop code (probably in the v5 or v6 timeframe).
>
> Thanks,
> Rainer
>
> [1] http://bugzilla.adiscon.com/show_bug.cgi?id=150
> [2] http://www.rsyslog.com/doc-rsconf1_omfileforcechown.html
>
> --
> [karmic] Messages not being sent to system logs
> https://bugs.launchpad.net/bugs/407862
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Neil Wilson

I couldn't agree more, and that is why I say that this work-around will be broken once rsyslog's privilege drop code has been rewritten. As stated in the wiki, the current solution is a quick and dirty one, provided only because there seems to be some value in providing it over not providing it.

However, as far as this problem is concerned, this is not over root access or non-root access. The issue is that rsyslogd should run as non-root. Let's assume it finally has decent code to do that. Then it will run, from the start on, as non-root. But then rsyslog.conf specifies that rsyslogd shall write to files where it has no permissions. My point is that either rsyslog.conf is invalid OR the files have been created with wrong permissions. In any case, it is not something that rsyslog can/should fix, because it is outside the scope of its configuration and capabilities. I would consider it the wrong approach to create a root child process just to write to some files, which apparently are set not to be accessible for the daemon users. IMHO this is an inconsistent system setup, and *that* root cause needs to be fixed.

Rainer

Rsyslogd can't run initially as non-root if it is going to listen on
port 512 or directly access the kernel logs. It needs to open that
first as root and then keep them open while dropping privileges. So
we're still going to have to root drop problem regrettably. (I'm sure
you already know that. I'm just pointing it out for the rest of the
audience here).

Once you sort the code so that it goes to non-root early, then it will
fail to open any file for which it doesn't have permissions and will
create files with the correct ownership. I believe that is the right
approach: rsyslog shouldn't be changing permissions on files.

The issue we have at the moment is two fold:

- firstly we either create root owned files, or open root owned files
and then write to them for a bit until the privileges drop and the
files are closed
- secondly we don't get any decent error out of rsyslog reporting the
'Permission denied' issue on opening/reopening of the files.

As you probably already know the file access code could do with a bit
of TLC. It doesn't report errors as well as it should.

If you drop root privileges early enough before any files are opened
or created and report errors properly in the File code then everything
else will follow.

Note that we don't kick off root processes to write files; we kick
them off to read privileged files for which there is no other
alternative - the kernel dmesg output doesn't appear to have a
permission entry that works properly for example, and of course
network ports under 1024 need root permission. It might be possible to
engineer rsyslogd so that it runs as two processes that talk to each
other. One as root to read the network ports and the kernel with the
other running as a normal user to do the normal syslog processing.

2009/9/11 Rainer Gerhards <email address hidden>:
> I couldn't agree more, and that is why I say that this work-around will
> be broken once rsyslog's privilege drop code has been rewritten. As
> stated in the wiki, the current solution is a quick and dirty one,
> provided only because there seems to be some value in providing it over
> not providing it.
>
> However, as far as this problem is concerned, this is not over root
> access or non-root access. The issue is that rsyslogd should run as non-
> root. Let's assume it finally has decent code to do that. Then it will
> run, from the start on, as non-root. But then rsyslog.conf specifies
> that rsyslogd shall write to files where it has no permissions. My point
> is that either rsyslog.conf is invalid OR the files have been created
> with wrong permissions. In any case, it is not something that rsyslog
> can/should fix, because it is outside the scope of its configuration and
> capabilities. I would consider it the wrong approach to create a root
> child process just to write to some files, which apparently are set not
> to be accessible for the daemon users. IMHO this is an inconsistent
> system setup, and *that* root cause needs to be fixed.
>
> Rainer
>
> --
> [karmic] Messages not being sent to system logs
> https://bugs.launchpad.net/bugs/407862
> You received this bug notification because you are a direct subscriber
> of the bug.
>

...

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
> Neil Wilson
> Sent: Friday, September 11, 2009 2:05 PM
> To: Rainer Gerhards
> Subject: Re: [Bug 407862] Re: [karmic] Messages not being sent to
> system logs
>
> Rsyslogd can't run initially as non-root if it is going to listen on
> port 512 or directly access the kernel logs. It needs to open that
> first as root and then keep them open while dropping privileges. So
> we're still going to have to root drop problem regrettably. (I'm sure
> you already know that. I'm just pointing it out for the rest of the
> audience here).

Michael Biebl, the Debian Maintainer, suggested using capabilities to reduce
this need. I will look into this, but other than that I agree.

> Once you sort the code so that it goes to non-root early, then it will
> fail to open any file for which it doesn't have permissions and will
> create files with the correct ownership. I believe that is the right
> approach: rsyslog shouldn't be changing permissions on files.

full ack

>
> The issue we have at the moment is two fold:
>
> - firstly we either create root owned files, or open root owned files
> and then write to them for a bit until the privileges drop and the
> files are closed
> - secondly we don't get any decent error out of rsyslog reporting the
> 'Permission denied' issue on opening/reopening of the files.

I have seen some issue when I worked on the patch this morning. But this
issue was so far unreported. It looks like a regression, and I am not sure if
4.2.0 already has this problem.

>
> As you probably already know the file access code could do with a bit
> of TLC. It doesn't report errors as well as it should.

large parts of the code have been rewritten in the current beta/devel
versions.

> If you drop root privileges early enough before any files are opened
> or created and report errors properly in the File code then everything
> else will follow.

Ah, OK, I see where you are coming from. So the files were initially created
by rsyslogd? I thought they were created by some other process. Mhhh... could
be. But in any case, the very simple cure with the current code base is to
specify the correct file owner right at the top of rsyslog.conf, so the files
will be created with the correct owner right from the beginning. Note that my
patch (and the patch posted here), assumes that the file owner was properly
configured. If there is no proper owner config, neither of the patches will
work. Thus I assumed this were no issue...

> Note that we don't kick off root processes to write files; we kick
> them off to read privileged files for which there is no other
> alternative - the kernel dmesg output doesn't appear to have a
> permission entry that works properly for example, and of course
> network ports under 1024 need root permission. It might be possible to
> engineer rsyslogd so that it runs as two processes that talk to each
> other. One as root to read the network ports and the kernel with the
> other running as a normal user to do the normal syslog processing.

you can already do...

I think I missed to express me clearly enough in my previous reply:

> - firstly we either create root owned files, or open root owned files
> and ...

Why do you create these files as root-owned in the first place? Why not
create them with the right user?That is my primary point.

Rainer

> Why do you create these files as root-owned in the first place? Why not
> create them with the right user? That is my primary point.

I agree. The logrotate.d file that rsyslog uses in Debian/Ubuntu should use the 'create' directive which says which user/group to create files as.

> Michael Biebl, the Debian Maintainer, suggested using capabilities to reduce
> this need. I will look into this, but other than that I agree.

I looked into this a bit. You'd need to use the CAP_SYS_ADMIN capability. Which is sort of a catch-all. It allows the program to do many, many root-y things [1]. Honestly, I'd prefer to have a root dd process (which is contained and pretty safe) feeding an unprivileged rsyslog than have an rsyslog with CAP_SYS_ADMIN.

[1] http://www.lids.org/lids-howto/node57.html

> I agree. The logrotate.d file that rsyslog uses in Debian/Ubuntu should use the 'create' directive which says
> which user/group to create files as.

Hmm, I guess that's actually not needed. Without the create directive, it leaves the creating up to rsyslog, which is fine for our purposes.

> > I agree. The logrotate.d file that rsyslog uses in Debian/Ubuntu
> should use the 'create' directive which says
> > which user/group to create files as.
>
> Hmm, I guess that's actually not needed. Without the create directive,
> it leaves the creating up to rsyslog, which is fine for our purposes.

... but in that case, I wonder why there is an issue in the first place.
rsyslogd, correctly configured, creates files with the correct user. If the
files have the correct user, and the right permissions, there is no issue in
re-opening files after a HUP.

The original bug report claimed that the files were root-owned. So I conclude
the problem is either of this:

a) rsyslog.conf is incorrect and does not specify the correct file owner at
the correct location
OR
b) some other process creates the files with the wrong user

In theory, there is a possibility c) that there is a bug in rsyslogd that
prevents creation with the proper owner.

So far, I outrule c) because my testing indicates this is not the case.

So it is either a) or b).

Now comes the important point: if it is a), the problem will persist, because
the provided fix will also fail because of the configuration.

So it looks like it is b). That, however, means that the current solution
works to some degree, but fails under some circumstances. This, I think, is
pretty dangerous and very hard to solve when a bug report comes in. Please
see my failure case described in the links I posted in one of my earlier
messages.

So while I have implemented a work-around in rsyslog, I still think this is
the wrong cure and careful consideration and review of all system components
is needed. Also keep in mind that the work-around will most likely always
fail with future rsyslg versions that have proper privilege drop code. Thus
the current work-around buys some time to find the component the incorrectly
creates the files - but it does not more.

Of course, distro-specific questions are not really of my concern. But I am
being somewhat persistent on this issue as I believe Ubuntu is a very
important distribution, does great work and as such deserves good
contributions.

Rainer

I agree it's probably (b), with the addendum that certainly part of the problem was previous rsyslog versions, which created the files as root. If that were the only problem, I suppose the packaging scripts could change permissions at install time, but that would require special logic to handle the case where the user set their own user/group.

I don't have any evidence yet that another program is creating/changing permissions on those files (all the current reports are explainable by rsyslog creating them as root before the user/group config change -- there was a period of time where rsyslog was running unprivileged, but the config told it to create files as root, my bad), but I haven't done an audit to make sure it's the case. I agree that such a review would be helpful.

So on the theory that the only real issue is previous rsyslog versions (until such an above review finds anything), the current 'force owner on open' behavior could conceivably be replaced by some install-time chowning, but to do that right (respect any user modification of either user/group or output files), it would involve parsing rsyslog.conf & rsyslog.conf.d/*.

Maybe for this usecase (distro changing default user/group), when rsyslog becomes fully-unprivileged, it could include a utility program called, say, rsyslog-fix-permissions that would run as root and chown all its output files?

I suppose a second alternative is to force a logrotate that doesn't create the files and let rsyslog recreate its files. But that relies on the logrotate config and seems a little forceful. I don't think it would be wise to logrotate when an administrator isn't expecting it.

Thanks very much for your Ubuntu concern. I appreciate it!

> I agree it's probably (b), with the addendum that certainly part of the
> problem was previous rsyslog versions, which created the files as root.
> If that were the only problem, I suppose the packaging scripts could
> change permissions at install time, but that would require special
> logic
> to handle the case where the user set their own user/group.
>
> I don't have any evidence yet that another program is creating/changing
> permissions on those files (all the current reports are explainable by
> rsyslog creating them as root before the user/group config change --
> there was a period of time where rsyslog was running unprivileged, but
> the config told it to create files as root, my bad), but I haven't done
> an audit to make sure it's the case. I agree that such a review would
> be helpful.

OK, if it is only this transient problem, I think the current fix is a good
one to solve the situation. Nevertheless, you correctly point out that some
issue may occur if the user changes the file owner. Of course, one can argue
that then this is a user error, but I agree it would be useful to have some
utility to aid the user in that process. At least some doc should explain the
situation and what to watch for.

> So on the theory that the only real issue is previous rsyslog versions
> (until such an above review finds anything), the current 'force owner
> on
> open' behavior could conceivably be replaced by some install-time
> chowning, but to do that right (respect any user modification of either
> user/group or output files), it would involve parsing rsyslog.conf &
> rsyslog.conf.d/*.
>
> Maybe for this usecase (distro changing default user/group), when
> rsyslog becomes fully-unprivileged, it could include a utility program
> called, say, rsyslog-fix-permissions that would run as root and chown
> all its output files?

This sounds very reasonable. I will keep this on my mind when working on the
new privilege drop code (whenever this may be). I tend to think that this
must be done by an interactive instance of rsyslogd, because only it has the
full knowledge of the configuration. But it may be really tricky for
dynafiles, where rsyslogd only knows at runtime, "as it happens" which files
need to be opened. Anyway, I think we can postpone that discussion until I
finally begin to implement that functionality.

> I suppose a second alternative is to force a logrotate that doesn't
> create the files and let rsyslog recreate its files. But that relies
> on
> the logrotate config and seems a little forceful. I don't think it
> would be wise to logrotate when an administrator isn't expecting it.

full ack

I still have one concern, and that is probably easy to verify: what happens
if logrotate runs? Does it create the files with the proper owner and
permissions? And if so, how does it know?

I understand that in Ubuntu terms, the "rsyslog logrotate script" is
considered part of the rsyslog package, but from the rsyslog project's POV
(mine ;)), this is something totally external, so I neither know exactly what
happens nor do I have any control over it. Actually, logrotate is my primary
c...

> I still have one concern, and that is probably easy to verify: what happens
> if logrotate runs? Does it create the files with the proper owner and
> permissions? And if so, how does it know?

The default logrotate script does not create the files. It will move the existing file out of the way and let rsyslog recreate. It will also HUP (but this becomes a restart with new upstartification changes) rsyslog.

One interesting thing about logrotate is that it also doesn't know about user changes to the list of output files. The config just assumes. So it requires some skill as an admin to add/remove files correctly.

/var/log/syslog
{
 rotate 7
 daily
 missingok
 notifempty
 delaycompress
 compress
 postrotate
  restart rsyslog >/dev/null 2>&1 || true
 endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
 rotate 4
 weekly
 missingok
 notifempty
 compress
 delaycompress
 sharedscripts
 postrotate
  restart rsyslog >/dev/null 2>&1 || true
 endscript
}

First - apologies for the vagueness that follows. I'm only an amateur and while I do record configurations I end up with, I don't record much if anything on the way. Second, I got here via installing rsyslog myself on Jaunty and then upgrading to Karmic which may also have confused things. I currently have rsyslog 4.2.0-2ubuntu5.

Anyway, while trying to get to a setup I like, I've been having all sorts of trouble with file/dir permissions. I think I'm ok now and I can get on with the other stuff I want to do but I have a couple of observations/questions which may arise from either the package or my fingers.

When I started looking at rsyslog after the upgrade, I found that rsyslog.conf did not set $DirOwner or $DirGroup at all. Having set those the same as $FileOwner and $FileGroup rsyslog managed to create the directories I want but I still had problems with dynafiles being created and eventually found this bug report.

Following Rainer's link [1] in comment #28 I read in Michael's description that $PrivDropToUser and $PrivDropToGroup should be set the same as $FileOwner and $FileGroup. I had $PrivDropTo as syslog:syslog but $File as syslog:adm. I don't believe I set any of those myself. I've now set all the groups to syslog and things seem to be working now.

This last issue leaves me questions though:
- _must_ $PrivDropTo match $File? I haven't spotted that as a requirement anywhere else or even mentioned in the docs I've read.
- should I have used syslog:syslog or syslog:adm?
- is the intention behind group adm to give read-only access to various admin files for appropriate users?
- if I use syslog:adm, should/must the syslog user be in group adm?

1) PrivDropTo doesn't need to match $File. Certainly not for the group. But PrivDropToUser should be able to read files created by FileOwner. Else, rsyslog can't read the files it creates.
2) syslog:adm is fine. The group doesn't matter so much. adm is the recommended group for system log files.
3) Yes, adm is for read-only access to system files.
4) No, syslog doesn't need to be in adm.

Thanks for the quick answers Michael, I'm afraid they've thrown up what looks like a bug to me. It's similar to this bug but I'm happy to open a new one if that's more appropriate.

Following your answers I set the following in rsyslog.conf (plus all the other parts obviously):
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirOwner syslog
$DirGroup adm
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

I want dynamic files based on severity (among other things but this is what I tested) using the following template:
$template tLevel, "/var/log/%$MONTH%/%$DAY%/%HOSTNAME%.%SYSLOGSEVERITY-TEXT%"

I sent a log entry with:
logger -p local0.debug 'debug test'

and this resulted in the following syslog entry:
Nov 16 15:37:27 robbie rsyslogd: Could not open dynamic file '/var/log/11/16/robbie.debug' - discarding message

although ls -l /var/log/11/16 showed:
-rw-r----- 1 syslog syslog 0 2009-11-16 15:37 robbie.debug
-rw-r----- 1 syslog adm 156681 2009-11-16 15:35 robbie.info
-rw-r----- 1 syslog adm 622 2009-11-16 15:37 robbie.syslog

Yesterday when the only difference was:
$FileGroup syslog
$DirGroup syslog

robbie.debug was created and written to as I expected. Just in case it's relevant I restarted rsyslog after editing the config with:
service rsyslog restart

robbie.info and robbie.syslog were created overnight owned by syslog:syslog but I did a chown before the test I've described here.

The 2 things that look like bugs to me are:
- creating robbie.debug with the wrong group
- rsyslog being unable to write to it even with the wrong group.

I also have this same issue.

I noticed that my new files were all being created with the ownership of "syslog:syslog". I was able to workaround this problem with a 'chgrp -Rv adm /var/log/foobar', where 'foobar' was the directory containing the new files. This should work, but I'm unsure if it will work when files are recreated during the next log rotation.

I haven't had more time to investigate, but it does appear that there is a mismatch with the file ownership.

@Gigglesworth, I'm still suffering the same problem as stated above at #42. For now I've just set all group configs to syslog. I did hit another problem which was all my own work though which might be relevant to you. As you'll see from the dynamic file template above, rsyslog should create a new directory /var/log/<mm> as the month changes. This didn't work for me but that was simply because rsyslog was running as syslog:syslog but /var/log was root:root 0755. That at least was clear once I thought to look.

To save anyone else doing this other test (with apologies to Michael for not simply accepting your statement) I also tested if it made any difference putting user syslog in supplementary group syslog. I couldn't see any difference for rsyslog although as an interactive user I couldn't chgrp a file to a group I wasn't a member of.

Opening gnome-system-log shows it searches for, and does not find:

/var/log/auth.log.0: Erro ao verificar o ficheiro '/var/log/auth.log.0': Ficheiro ou directoria inexistente
/var/log/daemon.log.0: Erro ao verificar o ficheiro '/var/log/daemon.log.0': Ficheiro ou directoria inexistente
/var/log/debug.0: Erro ao verificar o ficheiro '/var/log/debug.0': Ficheiro ou directoria inexistente
/var/log/kern.log.0: Erro ao verificar o ficheiro '/var/log/kern.log.0': Ficheiro ou directoria inexistente
/var/log/messages.0: Erro ao verificar o ficheiro '/var/log/messages.0': Ficheiro ou directoria inexistente
/var/log/syslog.0: Erro ao verificar o ficheiro '/var/log/syslog.0': Ficheiro ou directoria inexistente
/var/log/user.log.0: Erro ao verificar o ficheiro '/var/log/user.log.0': Ficheiro ou directoria inexistente

Does the count start on zero?

I found this page while searching for a fix for the same problem ("rsyslogd was huped..." and no more logging after that (I think cron might have stopped running my scripts then too. can that be?)
anyway, after reading all the technical stuff here, most of which I don't understand (what is a HUP anyway?), I still haven't figured out what 'm supposed to do. Has this bug been fixed, and me getting it is a different bug? is it only fixed in rsyslogd 4.7? will uninstalling rsyslogd and installing sysklogd solve it?
Thanx

I'm still having the same problem too. Here's a strace of the rsyslog process when it receives the HUP:

[0] spitzer:/var/log# strace -p 7830
Process 7830 attached - interrupt to quit
select(1, NULL, NULL, NULL, {17, 289248}) = ? ERESTARTNOHAND (To be restarted)
--- SIGHUP (Hangup) @ 0 (0) ---
rt_sigaction(SIGHUP, {0x40c630, [], SA_RESTORER, 0x7f0cd0500190}, NULL, 8) = 0
rt_sigreturn(0x1) = -1 EINTR (Interrupted system call)
futex(0x800594, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x800590, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
sched_yield() = 0
select(1, NULL, NULL, NULL, {30, 0}) = 0 (Timeout)
access("/var/log/syslog", F_OK) = 0
open("/var/log/syslog", O_WRONLY|O_CREAT|O_NOCTTY|O_APPEND|O_CLOEXEC, 0640) = -1 EACCES (Permission denied)
select(1, NULL, NULL, NULL, {30, 0}) = 0 (Timeout)
select(1, NULL, NULL, NULL, {30, 0}) = 0 (Timeout)
access("/var/log/syslog", F_OK) = 0
open("/var/log/syslog", O_WRONLY|O_CREAT|O_NOCTTY|O_APPEND|O_CLOEXEC, 0640) = -1 EACCES (Permission denied)
select(1, NULL, NULL, NULL, {30, 0}) = 0 (Timeout)
select(1, NULL, NULL, NULL, {30, 0}) = 0 (Timeout)

/var/log/syslog is owned by root:adm with permissions 0664 and created by logrotate after the rotation.

Having the same problem here. Running 9.10 64bit on EC2, this is from very early this morning:

Apr 8 05:32:05 ubuntu rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="544" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

Additionally the permissions on syslog:

-rw-r----- 1 syslog adm 4670 2010-04-08 11:03 /var/log/syslog

Having the same problem here. Running 10.04 32bit.

Apr 11 12:26:46 home rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="902" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

Running 10.04 64bits, the same message show up BUT it seems normal. I've seen the following behavior:
The message show in /var/log/messages; afterward it's the first one of /var/log/syslog. Log seems to have been rotated.
If I execute something like that:
$ logger -t test "This is a test message"
test message shows up in both syslog and messages.
This is all absolutely normal, isn't?

I'm available if further test is needed…

I have last message in /var/log/syslog:
May 26 07:35:16 kutoid rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="953" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
uname -a:
Linux kutoid 2.6.32-22-generic #33-Ubuntu SMP Wed Apr 28 13:27:30 UTC 2010 i686 GNU/Linux
Kubuntu Linux Lucid Lynx

rsyslogd -v :
rsyslogd 4.2.0, compiled with:
        FEATURE_REGEXP: Yes
        FEATURE_LARGEFILE: Yes
        FEATURE_NETZIP (message compression): Yes
        GSSAPI Kerberos 5 support: Yes
        FEATURE_DEBUG (debug build, slow code): No
        Atomic operations supported: Yes
        Runtime Instrumentation (slow code): No

See http://www.rsyslog.com for more information.

root@integre:/var/log# tail messages
Oct 10 06:45:53 integre rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="807" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 11 06:46:15 integre rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="807" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 12 06:47:48 integre rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="807" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 13 06:39:36 integre rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="807" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 13 06:39:36 integre rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="807" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 14 06:48:07 integre rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="807" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
root@integre:/var/log# rsyslogd -v
rsyslogd 4.2.0, compiled with:
        FEATURE_REGEXP: Yes
        FEATURE_LARGEFILE: Yes
        FEATURE_NETZIP (message compression): Yes
        GSSAPI Kerberos 5 support: Yes
        FEATURE_DEBUG (debug build, slow code): No
        Atomic operations supported: Yes
        Runtime Instrumentation (slow code): No

See http://www.rsyslog.com for more information.
root@integrecomm:/var/log# uname -a
Linux integrecomm 2.6.32-24-server #43-Ubuntu SMP Thu Sep 16 16:05:42 UTC 2010 x86_64 GNU/Linux
root@integrecomm:/var/log#

Didn't use to do this when I first installed the OS. I noticed it recently.

Ubuntu 10.10 32-Bit on Athlon II X4
/var/log/messages :

...

Oct 19 19:18:16 desktop kernel: [ 20.290536] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 19 19:46:02 desktop rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="977" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

Oct 20 19:16:51 desktop kernel: [ 19.653599] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 20 19:17:08 desktop pulseaudio[1716]: ratelimit.c: 3 events suppressed
Oct 20 19:44:16 desktop rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="1068" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 20 19:50:01 desktop pulseaudio[1716]: ratelimit.c: 727 events suppressed

Regards,
P.

Unfortunately, the included fix (force changing ownership on start-up) breaks logging of innd and running news-daily, as the logfiles should be owned by news.news but are now forced to syslog.adm. The init.d script from syslogd mentioned in https://bugs.launchpad.net/bugs/120085 uses "syslogd-listfiles -a" to get all logfiles used by syslogd but without the news logfiles! The now included fix just changes any logfile used by rsyslogd. Any reason to not do it like in the old syslogd init-script?

Continuing problem confirmed for Ubuntu 10.10 64-Bit with latest updates on a Dell OptiPlex 960, Intel Core 2 Duo E8600 (2x3.33GHz):
rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="841" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

This problem is still happening in fresh 11.04 installs. It makes troubleshooting very difficult when I have no logs to consult.

Same problem here with rsyslog 4.2.0-2ubuntu8 on ubuntu-server 10.10 (Dell PowerEdge R300)

is this still broken? any update from the maintainer?

Ubuntu, you're doing just great.

First posted in 2009-08-02.
Today is 2011-09-09

3 years and the problem is still there, should we mark this is as a feature maybe?

From my PoV, this is a problem that stems back to either the config provided and/or other packages.

Ubuntu makes rsyslog drop privileges. That's fine, though has some documented quirks until v6. Most importantly for this bug, files are opened before dropping privs. Nevertheless, files are created with proper owners *iff* the owner is properly configured. Then, privs are dropped. If the file owner is wrong, rsyslog can close files on HUP, but obviously not reopen it.

The described problem happens under the following circumstances:

(a) rsyslog.conf does not corretly contain file owner directives, so files are in the initial phase created with wrong user

(b) a third party package (usually logrotate) re-creates the files with an invalid owner or permissions

For both problems bug reports exists here. In rsyslog v6, I have finally refactored the code so that files are opened after priv drop. With the current state of affairs, this means that rsyslog v6 on Ubuntu will probably never start. So I really think it would be time to clean-up package dependencies or run rsyslog as root if the source of changing file permissions can not be found.

FWIW, I tried various scenarios described here, including HUPing and logrotate'ing in a fresh Oneiric install. I couldn't reproduce any of the problems, namely that if /var/log/syslog has syslog:adm ownership, it does so after the logrotate. Further, logger messages always go to the new /var/log/syslog.

I did some limited tests in Maverick and also couldn't reproduce it.

Those of you who are having a problem, can you confirm that it still exists, and in which version of Ubuntu?

I can confirm that the problem still exists in Ubuntu 0.04 LTS

I can confirm that the problem still exists in Ubuntu 10.04 LTS

Nikolaus, can you provide exact steps to reproduce the problem for you?

I have a fully up-to-date 10.04.3 LTS. /var/log/syslog is owned by syslog:adm. Note that if your file does *not* have these ownerships, you should `sudo chown syslog:adm /var/log/syslog` before you begin.

$ tail -f /var/log/syslog
# In another shell
$ logger testing
# testing shows up in syslog
# In another shell
$ pkill -HUP rsyslogd
# In the tail, I see the 'HUPed... lightweight' message
$ logger testing
# The message shows up in the tail

So that all looks to work perfectly. /var/log/syslog is still owned by syslog:adm. Now I do

$ logrotate --force --verbose /etc/logrotate.conf

All the logs get rotated, and /var/log/syslog is still owned by syslog:adm, and it contains the 'HUPed...lightweight' message. Writing a log message to it continues to work fine.

[0] spitzer:~# logger testing

[0] spitzer:~# tail -2 /var/log/syslog
 10:08:48 beta AptDaemon: INFO: Shutdown was requested
Sep 7 10:09:42 spitzer root: testing

[0] spitzer:~# logrotate --force --verbose /etc/logrotate.conf
reading config file /etc/logrotate.conf
compress_prog is now /bin/bzip2
uncompress_prog is now /bin/bunzip2
compress_ext is now .bz2
reading config info for /var/log/wtmp
reading config info for /var/log/btmp
reading config info for /var/log/syslog

Handling 3 logs

rotating pattern: /var/log/wtmp forced from command line (1 rotations)
empty log files are not rotated, only log files >= 131072 bytes are rotated, old logs are removed
considering log /var/log/wtmp
  log needs rotating
rotating log /var/log/wtmp, log->rotateCount is 1
dateext suffix '-20110907'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/bzip2
renaming /var/log/wtmp.1.bz2 to /var/log/wtmp.2.bz2 (rotatecount 1, logstart 1, i 1),
renaming /var/log/wtmp.0.bz2 to /var/log/wtmp.1.bz2 (rotatecount 1, logstart 1, i 0),
old log /var/log/wtmp.0.bz2 does not exist
renaming /var/log/wtmp to /var/log/wtmp.1
creating new /var/log/wtmp mode = 0664 uid = 0 gid = 43
removing old log /var/log/wtmp.2.bz2

rotating pattern: /var/log/btmp forced from command line (1 rotations)
empty log files are not rotated, only log files >= 131072 bytes are rotated, old logs are removed
considering log /var/log/btmp
  log needs rotating
rotating log /var/log/btmp, log->rotateCount is 1
dateext suffix '-20110907'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/bzip2
renaming /var/log/btmp.1.bz2 to /var/log/btmp.2.bz2 (rotatecount 1, logstart 1, i 1),
renaming /var/log/btmp.0.bz2 to /var/log/btmp.1.bz2 (rotatecount 1, logstart 1, i 0),
old log /var/log/btmp.0.bz2 does not exist
renaming /var/log/btmp to /var/log/btmp.1
creating new /var/log/btmp mode = 0660 uid = 0 gid = 43
removing old log /var/log/btmp.2.bz2

rotating pattern: /var/log/syslog
 forced from command line (4 rotations)
empty log files are not rotated, only log files >= 131072 bytes are rotated, old logs are removed
considering log /var/log/syslog
  log needs rotating
rotating log /var/log/syslog, log->rotateCount is 4
dateext suffix '-20110907'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/bzip2
renaming /var/log/syslog.4.bz2 to /var/log/syslog.5.bz2 (rotatecount 4, logstart 1, i 4),
renaming /var/log/syslog.3.bz2 to /var/log/syslog.4.bz2 (rotatecount 4, logstart 1, i 3),
renaming /var/log/syslog.2.bz2 to /var/log/syslog.3.bz2 (rotatecount 4, logstart 1, i 2),
renaming /var/log/syslog.1.bz2 to /var/log/syslog.2.bz2 (rotatecount 4, logstart 1, i 1),
renaming /var/log/syslog.0.bz2 to /var/log/syslog.1.bz2 (rotatecount 4, logstart 1, i 0),
old log /var/log/syslog.0.bz2 does not exist
renaming /var/log/syslog to /var/log/syslog.1
creating new /var/log/syslog mode = 0664 uid = 0 gid = 4
running postrotate script
removing old log /var/log/syslog.5.bz2

[0] spitzer:~# logger testing
[0] spitzer:~# tail /var/log/syslog
[0] spitzer:~# tail -2 /var/log/syslog.1
Sep 7 10:08:48 beta AptDaemon: INFO: Shutdown was...

Thanks for the details Nikolaus; I want to get just a bit more information.

On Sep 07, 2011, at 02:15 PM, Nikolaus Rath wrote:

>[0] spitzer:~# logger testing
>[0] spitzer:~# tail -2 /var/log/syslog
> 10:08:48 beta AptDaemon: INFO: Shutdown was requested
>Sep 7 10:09:42 spitzer root: testing
>[0] spitzer:~# logrotate --force --verbose /etc/logrotate.conf
[output deleted]
>[0] spitzer:~# logger testing
>[0] spitzer:~# tail /var/log/syslog
>[0] spitzer:~# tail -2 /var/log/syslog.1
>Sep 7 10:08:48 beta AptDaemon: INFO: Shutdown was requested
>Sep 7 10:09:42 spitzer root: testing
>
>
>==> no more messages in syslog.

At this point, what is the ownership and permissions on /var/log/syslog?

After the logrotate command, I can still send messages to syslog with

$ logger -i testing

Note that the -i is necessary to make each subsequent syslog entry different,
because /etc/rsyslogd.conf has

$RepeatedMsgReduction on

by default. Thus if you're testing with just `logger testing` these messages
will get suppressed. I just want to be sure that this isn't tricking you into
thinking that syslog is broken.

I also testing this with a very minimal logrotate.conf file:

-----snip snip-----
weekly
rotate 4
create

/var/log/syslog
{
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
    sharedscripts
}
-----snip snip-----

I still can't reproduce your problem. :(

On 09/07/2011 02:16 PM, Barry Warsaw wrote:
>> [0] spitzer:~# logger testing
>> [0] spitzer:~# tail -2 /var/log/syslog
>> 10:08:48 beta AptDaemon: INFO: Shutdown was requested
>> Sep 7 10:09:42 spitzer root: testing
>> [0] spitzer:~# logrotate --force --verbose /etc/logrotate.conf
> [output deleted]
>> [0] spitzer:~# logger testing
>> [0] spitzer:~# tail /var/log/syslog
>> [0] spitzer:~# tail -2 /var/log/syslog.1
>> Sep 7 10:08:48 beta AptDaemon: INFO: Shutdown was requested
>> Sep 7 10:09:42 spitzer root: testing
>>
>>
>> ==> no more messages in syslog.
>
> At this point, what is the ownership and permissions on /var/log/syslog?

# ls -l /var/log/syslog
-rw-rw-r-- 1 root adm 0 2011-09-07 10:11 /var/log/syslog

> After the logrotate command, I can still send messages to syslog with
>
> $ logger -i testing
>
> Note that the -i is necessary to make each subsequent syslog entry different,
> because /etc/rsyslogd.conf has
>
> $RepeatedMsgReduction on
>
> by default. Thus if you're testing with just `logger testing` these messages
> will get suppressed. I just want to be sure that this isn't tricking you into
> thinking that syslog is broken.

I don't think so:

[0] spitzer:~# logger a new, improved testing message
[0] spitzer:~# tail /var/log/syslog
[0] spitzer:~#

> I also testing this with a very minimal logrotate.conf file:
>
> -----snip snip-----
> weekly
> rotate 4
> create
>
> /var/log/syslog
> {
> postrotate
> reload rsyslog >/dev/null 2>&1 || true
> endscript
> sharedscripts
> }
> -----snip snip-----

I've attached my logrotate.conf in case it matters.

>
> I still can't reproduce your problem. :(

Well, let me know if there's anything I can do to help...

[0] spitzer:~# dpkg-query -l rsyslog
Desired=Unknown/Install/Remove/Purge/Hold
|
Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-====================-====================-========================================================
ii rsyslog 4.2.0-2ubuntu8.1 enhanced multi-threaded
syslogd

Best,

   -Nikolaus

--
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

On Sep 07, 2011, at 09:43 PM, Nikolaus Rath wrote:

>On 09/07/2011 02:16 PM, Barry Warsaw wrote:
>> At this point, what is the ownership and permissions on /var/log/syslog?
>
># ls -l /var/log/syslog
>-rw-rw-r-- 1 root adm 0 2011-09-07 10:11 /var/log/syslog

So the core problem here is that after logrotate, your /var/log/syslog is
owned by root instead of syslog.

>I've attached my logrotate.conf in case it matters.

Actually, it does! I'm sorry, I should have asked for a copy of that a long
time ago. :/ FWIW, I can reproduce the problem with your logrotate.conf
file.

Take a look at line 18:

-----snip snip-----
create 664 root adm
-----snip snip-----

From the manpage:

       create mode owner group
              Immediately after rotation (before the postrotate script is
              run) the log file is created (with the same name as the log
              file just rotated). mode specifies the mode for the log file
              in octal (the same as chmod(2)), owner specifies the user name
              who will own the log file, and group specifies the group the
              log file will belong to. Any of the log file attributes may be
              omitted, in which case those attributes for the new file will
              use the same values as the original log file for the omitted
              attributes. This option can be disabled using the nocreate
              option.

So, you're forcing all the new log files to be owned by root:adm, which breaks
rsyslogd. I'll bet that changing that line to just `create` will fix your
problem.

Now, here's a question: did you modify that line yourself, or did that come
with the package? My Lucid system just has `create` on a line by itself, so
I'm guessing you added that. If not, then is this machine updated from an
earlier version of Ubuntu? I think rsyslog 4.2.0-2ubuntu8.1 (which is what I
also have on my Lucid system) does not have that line in it.

In any case, I'm going to mark this bug as invalid, because I think we've
narrowed it down enough to know it's not a bug in rsyslog. Depending on your
response, we might need to open a new bug on logrotate.

Thanks so much for the excellent feedback!

On 09/08/2011 11:28 AM, Barry Warsaw wrote:
> Now, here's a question: did you modify that line yourself, or did that come
> with the package? My Lucid system just has `create` on a line by itself, so
> I'm guessing you added that. If not, then is this machine updated from an
> earlier version of Ubuntu? I think rsyslog 4.2.0-2ubuntu8.1 (which is what I
> also have on my Lucid system) does not have that line in it.

That system was upgraded several times, so I really can't tell where
this line comes from, sorry.

Best,

   -Nikolaus

--
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

i'm making this discussion come up as i have the same problem as described on 3 diferent machine all running ubuntu 10.04 (2 laptops and 1 server). if this is not so important for the laptop, it counts for my server as i usually read the syslog to see if evrything works fine... without a log file, i'm coming blind...

I didn't change any permisions on any file on these machine. for example I had this bug on my server last week, and reinstalled it 7 days ago. reading the syslog a couple of days after it shows:

jean-louis@x-wing:~$ tail -100 /var/log/syslog
Jan 21 06:40:30 x-wing rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="2173" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Jan 21 07:17:01 x-wing CRON[3990]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jan 21 08:17:01 x-wing CRON[4003]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jan 21 09:17:01 x-wing CRON[4017]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

when googling: rsyslogd was HUPed, type 'lightweight' there are thousands of links coming up, then this is not an isolated problem.

what should be done to solve this issue ?

i am available for any information, and can even provide an access to a machine if i can help solving.

surfingalien, I'm getting the same issue on 3 of my servers as well.

Running 10.04LTS with latest updates. Have you found any fixes?

Still broken in an up-to-date 10.04.4 LTS.

Yesterday afternoon leaving with disconnecting usb drive.
When returned (after 8 'o clock) the system was unresponsive.
System restarted at 8:23

Sep 18 16:24:31 tahars kernel: [115887.494108] usb 1-7: USB disconnect, address 5
Sep 19 07:40:10 tahars kernel: [170826.226381] type=1505 audit(1348033210.421:28): operation="profile_replace" pid=24685 name="/sbin/dhcli
ent3"
Sep 19 07:40:10 tahars kernel: [170826.226614] type=1505 audit(1348033210.421:29): operation="profile_replace" pid=24685 name="/usr/lib/Ne
tworkManager/nm-dhcp-client.action"
Sep 19 07:40:10 tahars kernel: [170826.226759] type=1505 audit(1348033210.421:30): operation="profile_replace" pid=24685 name="/usr/lib/co
nnman/scripts/dhclient-script"
Sep 19 07:40:14 tahars rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="867" x-info="http://www.rsyslog.com"] rsyslogd was HU
Ped, type 'lightweight'.
Sep 19 07:45:36 tahars popularity-contest: unable to submit report to http://popcon.ubuntu.com/popcon-submit.cgi.
Sep 19 07:45:36 tahars popularity-contest: unable to submit report.
Sep 19 07:50:32 tahars popularity-contest: unable to submit report to http://popcon.ubuntu.com/popcon-submit.cgi.
Sep 19 07:50:32 tahars popularity-contest: unable to submit report.
Sep 19 08:23:16 tahars kernel: imklog 4.2.0, log source = /proc/kmsg started.

root@ms:~# tail -n 1 /var/log/messages
Oct 26 06:25:01 atlantic1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="855" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

root@ms:~# uname -a
Linux ms 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux

root@ms:~# cat /etc/issue
Debian GNU/Linux 6.0 \n \l

logging is sent to /var/log/syslog not /var/log/messages

I got this one 1 machine, Precise (32 bit) installed.

For me, it seems, that somebody used the machine as "Guest", did not logged out and Firefox did an update, which lead to a total crash of the system... (but this is juast a guess!)

Last messages in /var/log/syslog.1 where:
May 20 07:30:01 icafe-d anacron[8374]: Anacron 2.3 started on 2013-05-20
May 20 07:30:01 icafe-d anacron[8374]: Will run job `cron.daily' in 5 min.
May 20 07:30:01 icafe-d anacron[8374]: Will run job `cron.weekly' in 10 min.
May 20 07:30:01 icafe-d anacron[8374]: Jobs will be executed sequentially
May 20 07:35:01 icafe-d anacron[8374]: Job `cron.daily' started
May 20 07:35:01 icafe-d anacron[8380]: Updated timestamp for job `cron.daily' to 2013-05-20
May 20 07:47:40 icafe-d rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="365" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

Should i report a new bug cause of different OS/architecture?

Hi,

I got this on my HP Envy 23 as last message in /var/log/syslog before a complete freeze (keyboard, mouse and ping hangs)

May 21 23:28:02 neruda kernel: [ 901.308548] [drm:drm_mode_addfb], [FB:33]
May 21 23:28:02 neruda kernel: [ 901.308555] [drm:radeon_crtc_page_flip], flip-ioctl() cur_fbo = ffff8802114eec00, cur_bbo = ffff88020f206000
May 21 23:28:03 neruda rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="1069" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

I have files a sperate bug:
https://bugs.launchpad.net/ubuntu/+source/xserver-xorg-video-ati/+bug/1178846

Can this bug realy lead to a complete freeze?

I got this one too.

I was unable to SSH into the box, however it would respond to ping. I'd describe this as freezing my box and it's resolution was a manual restart of the VM.

Jun 25 06:36:04 tux rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="733" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'

Ubuntu 10.10

I can confirm this in Debian 6.0.7, default install, default daemon configs.
Logging just stops after couple of days. I gave a shot to an aptitude upgrade but when I saw neither rsyslog/logrotate would not update I aborted it.

ii rsyslog 4.6.4-2 enhanced multi-threaded syslogd
ii logrotate 3.7.8-6 Log rotation utility

This is a LOGROTATE issue.
Here is a quick workaround, edit /etc/logrotate.d/rsyslog

Replace the 2 reload line to restart.
                invoke-rc.d rsyslog restart > /dev/null
                invoke-rc.d rsyslog restart > /dev/null

Since I can't confirm the result right now, I will let you know if this "fix" does not work.

I see this with ubuntu 13.10, all up-to-date.

rsyslogd 5.8.11
logrotate 3.8.3

I see this with Ubuntu 16.04 (amd64), all up-to-date.

rsyslog 8.16.0-1ubuntu3
logrotate 3.8.7-2ubuntu2

Observed on Ubuntu 22.04, it seems syslog and few others weren't updated since I upgraded from 20.04. I did reboot several times since then, but when I discovered this today rsyslog was inactive/dead. I manually started rsyslog, and now log files are updated, but I don't know if it will work after reboot (can't reboot right now). The permissions were/are correct (syslog:adm).

rsyslog 8.2112.0-2ubuntu2.2
logrotate 3.19.0-1ubuntu1.1