Logging/Log rotation does not work for catalina.out

Bug #1964881 reported by Evren Yurtesen
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tomcat10 (Ubuntu)
New
Undecided
Unassigned
tomcat9 (Debian)
New
Unknown
tomcat9 (Ubuntu)
Fix Released
Undecided
Andreas Hasenack
Focal
Fix Released
Undecided
Andreas Hasenack
Jammy
Fix Released
Undecided
Andreas Hasenack

Bug Description

[Impact]

Log handling in tomcat9 is broken in several ways:

a) logrotate fails to rotate the catalina.out log file

b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged)

c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it)

[Test Plan]

Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy:

lxc launch ubuntu:jammy j-tomcat9-logging

lxc shell j-tomcat9-logging

apt update && apt install tomcat9

Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm:
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9
total 12
drwxrws--- 1 tomcat adm 188 Jul 20 18:32 .
drwxrwxr-x 1 root syslog 314 Jul 20 18:32 ..
-rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log
-rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt

But here the problems start, and these are the ones fixed by this SRU:

a) rsyslog is complaining that it can't change the ownership of catalina.out:

root@j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog
Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ]

b) logrotate fails:

root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
error: error opening /var/log/tomcat9/catalina.out: Permission denied

And catalina.out remains unrotated:
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/
total 12
drwxrws--- 1 tomcat adm 188 Jul 20 18:32 .
drwxrwxr-x 1 root syslog 430 Jul 20 18:33 ..
-rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log
-rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt

c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore:

before reinstall:
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out

after reinstall:
root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y
Reading package lists... Done
(...)
Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out

And logging is broken:
root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog
Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]

Now install the tomcat9 package from proposed.

a) rsyslog won't complain anymore about failing to open or chown the file:
root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket
root@j-tomcat9-logging:~# > /var/log/syslog
root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket
root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0]
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ]
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104
Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start

b) This time logrotate works, and the catalina.out file will be rotated:

root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/
total 24
drwxrws--- 1 tomcat adm 216 Jul 20 18:39 .
drwxrwxr-x 1 root syslog 612 Jul 20 18:39 ..
-rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log
-rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out
-rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log
-rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt

c) reinstalling the package won't break logging again:
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out

root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall
Reading package lists... Done
(...)
Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...

root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out

[Where problems could occur]

These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made.

It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand.

[Other Info]
Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881
MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340

For focal, this SRU is also changing the permissions of /var/log/tomcat9 from 02750 to 02770. In jammy and later, it's 02770 already.

[Original Description]

In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out`
This could be fixed in a newer package but was not backported:
https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881

In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out`

Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this.

This causes following error:

rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ]

At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out`

This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu79
Architecture: amd64
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2022-02-27 (18 days ago)
InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121)
Package: tomcat9 9.0.58-1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12
RebootRequiredPkgs: Error: path contained symlinks.
Tags: jammy
Uname: Linux 5.15.0-18-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True

Related branches

description: updated
Revision history for this message
Chris Guiver (guiverc) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1964881

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote : Dependencies.txt

apport information

tags: added: apport-collected
description: updated
Revision history for this message
Evren Yurtesen (eyurtese-g) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote : ProcEnviron.txt

apport information

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

@Chris Guiver (guiverc), I have executed apport.

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

Ideas on possible fixes:

In Ubuntu 20.04

* Perhaps the log directory fix can be backported, in addition this require solution 2 from below. If solution 1 is applied from below, there would be no need to backport the fix from the newer package.
https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881/comments/4

In Ubuntu 21.10/22.04

* Solution possibility 1: As the issue is because of the difference of the user that `rsyslogd` runs as between Debian and Ubuntu. One solution would be running `rsyslogd` as root user same as in Debian. This automatically fix everything for the foreseeable future as the Debian `tomcat` package would work fine without any modifications.

* Solution possibility 2: modify /etc/logrotate.d/tomcat and replace `su tomcat adm` with `su syslog adm`. Update `/etc/syslog.d/tomcat9.conf` and replace `fileOwner="tomcat"` with `fileOwner="syslog"` This fix should be applied to every time `tomcat` is imported from upstream Debian.

* Solution possibility 3: Any ideas?

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

I prepared a patch fixing this bug. Solved the issue by changing rsyslogd and logrotated's tomcat config using syslog:adm permissions.

This should be possible to apply to 22.04 and 21.10. Probably not to focal/20.04 as it is missing the patch which grants write access to `adm` group to `/var/log/tomcat9`
https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881/comments/4
If combined with that, it can be applied to 20.04 also.

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

debdiff for focal including the `/var/log/tomcat` folder permissions fix.

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :
Robie Basak (racb)
tags: added: server-todo
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in tomcat9 (Ubuntu Focal):
status: New → Confirmed
Changed in tomcat9 (Ubuntu Jammy):
status: New → Confirmed
Changed in tomcat9 (Ubuntu):
status: New → Confirmed
Changed in tomcat9 (Ubuntu):
assignee: nobody → Andreas Hasenack (ahasenack)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Taking a look at this together with frr's bug #1958162 which is similar. Maybe I can apply the same fix/pattern to both.

Changed in tomcat9 (Ubuntu Focal):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in tomcat9 (Ubuntu Jammy):
assignee: nobody → Andreas Hasenack (ahasenack)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'm checking kinetic first. Many things to adjust ;)

For one, I see also other log files in /var/log/tomcat9 which are created by the java process itself, via the configuration in /etc/tomcat9/logging.properties. From what I can see, these have their own rotation rules (currently 90 days), and are separate.

So we have:
root@k-tomcat9-logging:~# fuser /var/log/tomcat9/*
/var/log/tomcat9/catalina.2022-06-21.log: 9616
/var/log/tomcat9/catalina.out: 9631
/var/log/tomcat9/localhost.2022-06-21.log: 9616
/var/log/tomcat9/localhost_access_log.2022-06-21.txt: 9616

9616: java
9631: rsyslogd

Let's see...

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

There are 4 things writing to /var/log/tomcat9 and they all need to agree on permissions and ownership:
- rsyslog: catalina.out only, syslog:adm
- logrotate: catalina.out only, tomcat:adm
- java (tomcat9 itself): localhost*, catalina.<date>: tomcat:tomcat (:adm via sgid bit)
- tomcat9's maintainer scripts (postinst): chown -Rh tomcat:adm /var/log/tomcat9

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Evren, I'm not ignoring your patches, ok? :) This is just me getting up to speed with an untainted view of how things are :)

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

Andreas , that is very reasonable. I found a better solution after I created patches.

Would it be an acceptable solution if the fileOwner setting is removed from /etc/rsyslog.d/tomcat9.conf and su setting is removed from /etc/logrotate.d/tomcat9 file which are shipped with tomcat9 package?

This would let rsyslog to create files with syslog:adm permissions in ubuntu and root:adm in Debian (as rsyslog runs as root there). This is what some other packages have if you want to check for a good example. It might be that, this way a delta could be avoided.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Let's see.

From the start, that gives us catalina.out syslog:adm 0640, and all the other log files produced by the java process are tomcat:adm 0640.

If I run logrotate after that, I get:

truncated catalina.out syslog:adm 0640
rotated catalina.out.1 syslog:adm 0640

I'm unsure why catalina.out.1 is owned by syslog:adm and not tomcat:adm, because the config still has "create 640 tomcat adm". Maybe rsyslog won the race and created the file, and maybe we can drop this "create" line from the logrotate config for tomcat9.

So now I have:
- logs produced by the java process: tomcat:adm 0640
- logs rotated by logrotate: syslog:adm 0640
- log produced by rsyslog: syslog:adm 0640

-rw-r----- 1 tomcat adm 21K Jun 23 13:51 catalina.2022-06-23.log
-rw-r----- 1 syslog adm 0 Jun 23 13:52 catalina.out
-rw-r----- 1 syslog adm 7.7K Jun 23 13:52 catalina.out.1
-rw-r----- 1 syslog adm 20 Jun 23 13:43 catalina.out.2.gz
-rw-r----- 1 syslog adm 93 Jun 23 13:42 catalina.out.3.gz
-rw-r----- 1 tomcat adm 0 Jun 23 13:32 localhost.2022-06-23.log
-rw-r----- 1 tomcat adm 0 Jun 23 13:32 localhost_access_log.2022-06-23.txt

Let's reinstall tomcat9. And I get:
-rw-r----- 1 tomcat adm 27K Jun 23 14:03 catalina.2022-06-23.log
-rw-r----- 1 tomcat adm 0 Jun 23 13:52 catalina.out
-rw-r----- 1 tomcat adm 7.7K Jun 23 13:52 catalina.out.1
-rw-r----- 1 tomcat adm 20 Jun 23 13:43 catalina.out.2.gz
-rw-r----- 1 tomcat adm 93 Jun 23 13:42 catalina.out.3.gz
-rw-r----- 1 tomcat adm 0 Jun 23 13:32 localhost.2022-06-23.log
-rw-r----- 1 tomcat adm 0 Jun 23 13:32 localhost_access_log.2022-06-23.txt

That's because of tomcat9's postinst which does chown -Rh tomcat:adm /var/log/tomcat9.

And now rsyslog cannot write to catalina.out anymore:
Jun 23 14:06:03 k-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2204.0 try https://www.rsyslog.com/e/2433 ]

And this time logrotate's "create" kicked in, because an empty catalina.out is created as tomcat:adm.

So we need to exclude /var/log/tomcat9/catalina.out from tomcat9's recursive chown in postinst.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I wonder why logrotate is using "copytruncate" for catalina.out. The manpage also says that "create" is ignored when "copytruncate" is used.

Since catalina.out is written to by rsyslogd, maybe we could skip "copytruncate" and "create" and just HUP rsyslogd via /usr/lib/rsyslog/rsyslog-rotrate

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Changed in tomcat9 (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

> If I run logrotate after that, I get:
> truncated catalina.out syslog:adm 0640

How are you running logrotate? Because the problem was that it could not be truncated by rsyslog running as `tomcat` user.

Anyway, I checked your branch and your fix looks quite similar to mine. The only difference is that I had `fileOwner="syslog"` in `debian/rsyslog/tomcat9.conf` but that is inconsequential. As my setting was redundant, because the default fileowner for rsyslog is `syslog`. (in Ubuntu). So I could say, yes it works :)

That said, it is sad that there is a delta. I think this could be avoided if Debian maintainers would agree on using `root:adm` in their side and Ubuntu defaults to `syslog:adm`. As the default fileOwner is `root` for Debian and it is for Ubuntu in rsyslog. The logrotate also defaults to `root` in Debian and `syslog` in Ubuntu. But maybe it is too difficult to co-operate in this level. I do not know...

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

> if Debian maintainers would agree on using `root:adm` in their side

That would still leave the java-produced logs with a different ownership (tomcat:adm), because they are written to by a process that runs as tomcat:tomcat.

I still don't see a nice way out what will work without any delta at all in ubuntu.

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

> I still don't see a nice way out what will work without any delta at all in ubuntu.

"nice way" is relative. It seems the most common way for Debian to create logs is as root:adm. Is there any rule which says catalina.out must be owned by tomcat?

Consider the "jetty9" package in Debian. Rsyslog creates and writes logs of it to /var/log/jetty9 with root:adm ownership. In logrotated conf, it has "create 640 jetty adm" but it does nothing due to "copytruncate". But more importantly, nobody seems to care that `jetty' user does not own the logs.

Basically fix for tomcat9, is to simply remove 'fileOwner="tomcat"' from Debian and 'fileOwner="syslog"' from Ubuntu logrotate.d/tomcat9.conf Then why would there be a need for Delta? The only thing required is that Debian should accept `tomcat` rsyslog logs to have root:adm ownership.

Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

> I still don't see a nice way out what will work without any delta at all in ubuntu.

I am not sure who owns the logs have any effect in this being nice. Actually, maybe you did not realize, but the logs that are written by rsyslog and tomcat are same log lines in different formatting. One might say, this is also ugly :)

But what is wrong with rsyslog writing logs as root:adm in Debian, and syslog:adm in Ubuntu? This is exactly how it works for most other Debian packages. For example check "jetty9". Nobody complains that 'jetty' does not own those logs.

If tomcat9 rsyslog config would miss fileOwner setting (in Debian also). This could work well for both Ubuntu and Debian.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

> "nice way" is relative. It seems the most common way for Debian to create logs is as root:adm. Is
> there any rule which says catalina.out must be owned by tomcat?

What about the logs produced by the java tomcat9 process? They will be owned by tomcat:adm, and there is no config tweak that I know of to change that: the process just runs as tomcat. In the current debian packaging of tomcat9 all its logs are consistent in their ownership: tomcat:adm. What is the argument we have to ask the debian packager to change that so that one set of logs is root:adm, and the other is tomcat:adm?

> Actually, maybe you did not realize, but the logs that are written by rsyslog and tomcat are same
> log lines in different formatting.

And it's probably again in the systemd journal, no? :)

Let's switch to code. What is our concrete proposal for debian to change its logging for tomcat9? We need to consider the 4 things that write to /var/log/tomcat9:
- rsyslog
- tomcat9 itself
- logrotate
- tomcat9's postinst script

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I uploaded the current fix to ubuntu 22.10, we can still iterate over it during the development cycle.

Changed in tomcat9 (Ubuntu Jammy):
status: Confirmed → In Progress
status: In Progress → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package tomcat9 - 9.0.64-2ubuntu1

---------------
tomcat9 (9.0.64-2ubuntu1) kinetic; urgency=medium

  * Fix logging for unprivileged rsyslogd (LP: #1964881):
    - d/logrotate.template: use syslog:adm for log rotation so that
      rsyslog can write to the file
    - d/rsyslog/tomcat9.conf: drop "fileOwner" as it cannot be set by an
      unprivileged rsyslogd
    - d/tomcat9.postinst: adjust ownership of catalina.out so that
      rsyslogd can write to it. Also change the rotated log files for
      consistency.

 -- Andreas Hasenack <email address hidden> Thu, 23 Jun 2022 18:02:52 -0300

Changed in tomcat9 (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Evren Yurtesen (eyurtese-g) wrote :

> In the current debian packaging of tomcat9 all its logs are consistent in their ownership

Is there consistency? Because some other package like 'jetty9' is writing as 'root:adm' while `tomcat9` package 'tomcat:adm'. So I can say that between different packages, there is no consistency in who owns the logs.

Yes, within the `/var/log/tomcat9` one can claim that the files are owned by same 'user:group'. Therefore consistent. But I do not see what positive effect does it bring?

It is actually not good that the logs are owned by `tomcat` user anyway, at least from a security point. This is a user which executes a web server. Can read own logs. So it would make more sense to go to `root:adm` by default. Yes, there is the elephant in the room as `tomcat` itself is writing logs there and there is no easy way to make those logs unreadable by tomcat as it rotates them also. But that is something we do not have to touch.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

> Yes, there is the elephant in the room as `tomcat` itself is writing logs there and there is no
> easy way to make those logs unreadable by tomcat as it rotates them also.

Maybe the best way forward is to centralize the logs via rsyslog only. Adjust or remove the existing logging.properties. I don't know much about java and how it logs stuff, though.

Changed in tomcat9 (Debian):
status: Unknown → New
Changed in tomcat9 (Ubuntu Jammy):
status: Confirmed → In Progress
Changed in tomcat9 (Ubuntu Focal):
status: Confirmed → In Progress
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Evren, or anyone else affected,

Accepted tomcat9 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tomcat9/9.0.58-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in tomcat9 (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Changed in tomcat9 (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote :

Hello Evren, or anyone else affected,

Accepted tomcat9 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tomcat9/9.0.31-1ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Download full text (4.7 KiB)

Jammy SRU verification

Reproducing the problems first:
ubuntu@j-tomcat9-logging:~$ apt-cache policy tomcat9
tomcat9:
  Installed: 9.0.58-1
  Candidate: 9.0.58-1
  Version table:
 *** 9.0.58-1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

a) rsyslog complaints
root@j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog
Jul 27 13:41:48 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ]

b) logrotate fails
root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf
error: error opening /var/log/tomcat9/catalina.out: Permission denied
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9
total 12
drwxrws--- 1 tomcat adm 188 Jul 27 13:41 .
drwxrwxr-x 1 root syslog 538 Jul 27 13:43 ..
-rw-r----- 1 tomcat adm 5995 Jul 27 13:41 catalina.2022-07-27.log
-rw-r----- 1 syslog adm 3523 Jul 27 13:41 catalina.out
-rw-r----- 1 tomcat adm 0 Jul 27 13:41 localhost.2022-07-27.log
-rw-r----- 1 tomcat adm 0 Jul 27 13:41 localhost_access_log.2022-07-27.txt

c) reinstall breaks logging again
root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 syslog adm 3523 Jul 27 13:41 /var/log/tomcat9/catalina.out

root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y
(...)
Preparing to unpack .../tomcat9_9.0.58-1_all.deb ...
Unpacking tomcat9 (9.0.58-1) over (9.0.58-1) ...
Setting up tomcat9 (9.0.58-1) ...
Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ...
(...)

root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
-rw-r----- 1 tomcat adm 3523 Jul 27 13:41 /var/log/tomcat9/catalina.out

# grep -E "catalina\.out.*Permission denied" /var/log/syslog
Jul 27 13:43:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]
Jul 27 13:44:00 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]

Testing fixed version from proposed
root@j-tomcat9-logging:~# apt-cache policy tomcat9
tomcat9:
  Installed: 9.0.58-1ubuntu0.1
  Candidate: 9.0.58-1ubuntu0.1
  Version table:
 *** 9.0.58-1ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-proposed/universe amd64 Packages

a) rsyslog errors
rsyslog is clean:
root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket
root@j-tomcat9-logging:~# > /var/log/syslog
root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket
root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog
Jul 27 13:49:24 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0]
Jul 27 13:49:24 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
Jul 27 13:49:24 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ]
Jul 27 13:49:24 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111
Jul 27 13:49:24 j-tomcat9-logging rsysl...

Read more...

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Download full text (6.7 KiB)

Focal verification

Reproducing the problems:
root@f-tomcat9-logging:~# apt-cache policy tomcat9
tomcat9:
  Installed: 9.0.31-1ubuntu0.2
  Candidate: 9.0.31-1ubuntu0.2
  Version table:
 *** 9.0.31-1ubuntu0.2 500
        500 http://br.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages

a) rsyslog is complaining that it can't change the ownership of catalina.out:

In the case of focal, it's even worse, as catalina.out cannot even be created because the parent directory doesn't allow the adm group write access:

root@f-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out
ls: cannot access '/var/log/tomcat9/catalina.out': No such file or directory

root@f-tomcat9-logging:~# grep -E 'catalina\.out' /var/log/syslog
Jul 27 14:04:30 f-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2001.0 try https://www.rsyslog.com/e/2433 ]

b) logrotate fails:
Again, focal is different, since the log file doesn't exist, logrotate won't fail yet out of the box:

root@f-tomcat9-logging:~# logrotate -f /etc/logrotate.conf

root@f-tomcat9-logging:~# ls -la /var/log/tomcat9/
total 8
drwxr-s--- 1 tomcat adm 164 Jul 27 14:04 .
drwxrwxr-x 1 root syslog 430 Jul 27 14:07 ..
-rw-r----- 1 tomcat adm 5668 Jul 27 14:04 catalina.2022-07-27.log
-rw-r----- 1 tomcat adm 0 Jul 27 14:04 localhost.2022-07-27.log
-rw-r----- 1 tomcat adm 0 Jul 27 14:04 localhost_access_log.2022-07-27.txt

c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore:

Since catalona.out doesn't exist, reinstalling the package doesn't break it:
root@f-tomcat9-logging:~# ls -la /var/log/tomcat9/
total 8
drwxr-s--- 1 tomcat adm 164 Jul 27 14:04 .
drwxrwxr-x 1 root syslog 430 Jul 27 14:07 ..
-rw-r----- 1 tomcat adm 5668 Jul 27 14:04 catalina.2022-07-27.log
-rw-r----- 1 tomcat adm 0 Jul 27 14:04 localhost.2022-07-27.log
-rw-r----- 1 tomcat adm 0 Jul 27 14:04 localhost_access_log.2022-07-27.txt

root@f-tomcat9-logging:~# apt install --reinstall tomcat9
(...)
Unpacking tomcat9 (9.0.31-1ubuntu0.2) over (9.0.31-1ubuntu0.2) ...
Setting up tomcat9 (9.0.31-1ubuntu0.2) ...
Processing triggers for rsyslog (8.2001.0-1ubuntu1.3) ...

root@f-tomcat9-logging:~# ls -la /var/log/tomcat9/
total 12
drwxr-s--- 1 tomcat adm 164 Jul 27 14:04 .
drwxrwxr-x 1 root syslog 430 Jul 27 14:07 ..
-rw-r----- 1 tomcat adm 11836 Jul 27 14:08 catalina.2022-07-27.log
-rw-r----- 1 tomcat adm 0 Jul 27 14:04 localhost.2022-07-27.log
-rw-r----- 1 tomcat adm 0 Jul 27 14:04 localhost_access_log.2022-07-27.txt

So for focal, the main problem is (a), and (b) and (c) don't get a chance to appear because of that. But once (a) is fixed, (b) and (c) will happen, and need to be fixed as well.

Installing the fixed version from proposed:
root@f-tomcat9-logging:~# apt-cache policy tomcat9
tomcat9:
  Installed: 9.0.31-1ubuntu0.3
  Candidate: 9.0.31-1ubuntu0.3
  Version table:
 *** 9.0.31-1ubuntu0.3 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packa...

Read more...

tags: added: verification-done-focal
removed: verification-needed-focal
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package tomcat9 - 9.0.58-1ubuntu0.1

---------------
tomcat9 (9.0.58-1ubuntu0.1) jammy; urgency=medium

  * Fix logging for unprivileged rsyslogd (LP: #1964881):
    - d/logrotate.template: use syslog:adm for log rotation so that
      rsyslog can write to the file
    - d/rsyslog/tomcat9.conf: drop "fileOwner" as it cannot be set by an
      unprivileged rsyslogd
    - d/tomcat9.postinst: adjust ownership of catalina.out so that
      rsyslogd can write to it. Also change the rotated log files for
      consistency.

 -- Andreas Hasenack <email address hidden> Wed, 20 Jul 2022 16:05:45 -0300

Changed in tomcat9 (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for tomcat9 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package tomcat9 - 9.0.31-1ubuntu0.3

---------------
tomcat9 (9.0.31-1ubuntu0.3) focal; urgency=medium

  * Fix logging for unprivileged rsyslogd (LP: #1964881):
    - d/logrotate.template: use syslog:adm for log rotation so that
      rsyslog can write to the file
    - d/tomcat9.postinst: adjust ownership of catalina.out so that
      rsyslogd can write to it. Also change the rotated log files for
      consistency.
    - d/tomcat9.tmpfile: /var/log/tomcat9 should be 02770 now

 -- Andreas Hasenack <email address hidden> Wed, 20 Jul 2022 15:09:00 -0300

Changed in tomcat9 (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Sami hulkko (sahulkko) wrote :

Hi,
I get error:
error: error opening /var/log/tomcat9/catalina.out: Permission denied

Logrotate fails and I have never had tomcat installed in this machine.
SH

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Sami,
you got this error while doing what exactly?
Running tomcat, running logrotate, .. ?

I read your "Logrotate fails and I have never had tomcat installed in this machine." as "you might just have installed tomcat. So I did try to have a system with:
- logrotate, but no tomcat9 -> no issue
- restarted logrotate -> no issue
- installed tomcat9 -> no issue
- restarted logrotate -> no issue
So I need to ask:
- what exactly you have done, did you set or trigger any deeper logrotate config/action?
- Does "I have never had tomcat installed" mean before now, or still not installed?

no longer affects: tomcat10 (Ubuntu Focal)
no longer affects: tomcat10 (Ubuntu Jammy)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

The same issues affects tomcat10, and I believe the same changes from tomcat9 can be applied to tomcat10.

commit f20fff9cedd4d2db449491ec815b8cda38f84c72
Author: Andreas Hasenack <email address hidden>
Date: Thu Dec 22 14:58:06 2022 -0300

          + d/tomcat9.postinst: adjust ownership of catalina.out so that
            rsyslogd can write to it. Also change the rotated log files for
            consistency.

commit 6565d1422b07b0f2e1d4b23718121692e8887b94
Author: Andreas Hasenack <email address hidden>
Date: Thu Dec 22 14:57:47 2022 -0300

          + d/rsyslog/tomcat9.conf: drop "fileOwner" as it cannot be set by an
            unprivileged rsyslogd

commit 07479b35d889ce117a643fbc45621271241517f9
Author: Andreas Hasenack <email address hidden>
Date: Thu Dec 22 14:57:21 2022 -0300

        - Fix logging for unprivileged rsyslogd (LP #1964881):
          + d/logrotate.template: use syslog:adm for log rotation so that
            rsyslog can write to the file

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "Fixes LP: #1964881" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Robie Basak (racb)
tags: removed: server-todo
Revision history for this message
Julian Andres Klode (juliank) wrote :

There doesn't seem to be anything here to sponsor, all the tasks for tomcat9 are done, there are no open merge requests, and there's nothing since "this also affects tomcat10" kind of comment.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.