rabbitmq logrotate doesn't work properly

Bug #1473405 reported by Leontiy Istomin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Medium
Ivan Suzdal
7.0.x
Won't Fix
Medium
Denis Meltsaykin
8.0.x
Fix Released
Medium
Ivan Suzdal
9.x
Invalid
Medium
Alexey Lebedeff

Bug Description

the logrotate config for rabbitmq removes old logs instead of create a new one. /etc/logrotate.d/rabbitmq-server: http://paste.openstack.org/show/361513/
[root@node-8 ~]# ls -la /var/log/rabbitmq/rabbit\@node-8*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:42 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:42 /<email address hidden>
[root@node-8 ~]# service rabbitmq-server rotate-logs > /dev/null
[root@node-8 ~]# ls -la /var/log/rabbitmq/rabbit\@node-8*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:05 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:05 /<email address hidden>

We need to use the following command:
DATE=-`date +%y_%m_%d_%H_%M_%S`; rabbitmqctl rotate_logs ${DATE}
[root@node-13 ~]# ls -la /var/log/rabbitmq/rabbit\@node-13*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:47 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 10:47 /<email address hidden>
[root@node-13 ~]# DATE=-`date +%y_%m_%d_%H_%M_%S`; rabbitmqctl rotate_logs ${DATE}
Rotating logs to files with suffix "-15_07_10_11_06_05" ...
...done.
[root@node-13 ~]# ls -la /var/log/rabbitmq/rabbit\@node-13*
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:06 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 91 Jul 10 11:06 /<email address hidden>15_07_10_11_06_05
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:06 /<email address hidden>
-rw-r----- 1 rabbitmq rabbitmq 0 Jul 10 11:06 /<email address hidden>15_07_10_11_06_05

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'

Changed in fuel:
milestone: none → 7.0
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → Medium
status: New → Confirmed
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Bartlomiej Piotrowski (bpiotrowski)
Revision history for this message
Florek (sflorczak) wrote :

Because ROTATE_SUFFIX in the configuration file is not set the log is cleared.
However "service rabbitmq-server rotate-logs" shouldn't be used in the first place. The proper way to rotate the logs is using: logrotate -f /etc/logrotate.d/rabbitmq-server

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Leontiy Istomin (listomin) wrote :

I believe the bug should be in "confirmed" state because we lose rabbitmq logs. If the right way to solve this issue is using logrotate instead of native rabbitmq logrotation please implement that.

Changed in fuel:
status: Invalid → Confirmed
Florek (sflorczak)
Changed in fuel:
assignee: Bartlomiej Piotrowski (bpiotrowski) → Florek (sflorczak)
Revision history for this message
Florek (sflorczak) wrote :

The issue can be reproducing in the following way:
in /var/log/rabbitmq there should be at least two files:
 - <email address hidden> with size less than 100MB (value configured in /etc/logrotate.conf), where X is the number of the node
 - any other file (*.log) with size more than 100MB
whenever logrotate will be run (/usr/sbin/logrotate -v /etc/logrotate.conf) it'll rotate the second file - because its larger than the defined threshold, will not rotate <email address hidden> because it's below threshold and after that the postscript will be run (according to /etc/logrotate.d/rabbitmq-server):
/etc/init.d/rabbitmq-server rotate-logs > /dev/null
Because the "ROTATE_SUFFIX" is not set in the /etc/init.d/rabbitmq-server no logs will be rotated, instead of that content of the /<email address hidden> will be flushed.

The simplest way to fix the issue is to change /etc/logrotate.d/rabbitmq-server in the following way:

delete:

 sharedscripts
        postrotate
            /etc/init.d/rabbitmq-server rotate-logs > /dev/null
        endscript

add:
 copytruncate

This way the "clearing script" will not be run. "copytruncate" will copy current log file to the new file and truncate the file currently used by the rabbitmq. Thanks to that the file handle used by rabbitmq doesn't have to be changed and the process itself doesn't have to be reloaded.

Changed in fuel:
assignee: Florek (sflorczak) → MOS Packaging Team (mos-packaging)
Changed in fuel:
assignee: MOS Packaging Team (mos-packaging) → Igor Yozhikov (iyozhikov)
assignee: Igor Yozhikov (iyozhikov) → Aleksander Mogylchenko (amogylchenko)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/trusty/rabbitmq-server (7.0)

Fix proposed to branch: 7.0
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/10372

Igor Shishkin (teran)
Changed in fuel:
milestone: 7.0 → 8.0
Dmitry Pyzhov (dpyzhov)
tags: added: area-linux
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/centos7/rabbitmq-server (8.0)

Fix proposed to branch: 8.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/14927

Changed in fuel:
status: Confirmed → In Progress
Changed in fuel:
assignee: Aleksander Mogylchenko (amogylchenko) → Ivan Suzdal (isuzdal)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to packages/centos7/rabbitmq-server (8.0)

Reviewed: https://review.fuel-infra.org/14927
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: c03a5cc1c14c5a0c1426f88f87b628d86d407fb8
Author: Ivan Suzdal <email address hidden>
Date: Tue Dec 22 10:23:29 2015

Remove postscript, add copytruncate to logrotate.

Change-Id: Ide704954dbdac8be479ea4ae9b94479dd523677c
Closes-Bug: #1473405

Changed in fuel:
status: In Progress → Fix Committed
no longer affects: fuel/7.0.x
no longer affects: fuel
Revision history for this message
Ivan Lozgachev (ilozgachev) wrote :

Verified on Fuel 8.0 build 518, ENV-13

Revision history for this message
Alexey Lebedeff (alebedev-a) wrote :

Fixed for centos7 and ubuntu by packaging rabbit 3.6.1, where `rotate_logs` was completely fixed.

Revision history for this message
Sergey Kulanov (skulanov) wrote :

Folks, please check for master branch, probably you forgot to fix logrotate script. Please check failed test [1]:

1)
[root@nailgun ~]# rpm -q --changelog rabbitmq-server
* Wed Mar 16 2016 Alexey Lebedeff <email address hidden> - 3.6.1-3~mos1
- d9a7cc1 Fix systemd issues
- 383e912 Tune 3.6.1 for RHEL and backport some patches
- c389137 Import upstream package for 3.6.1
- b0a8190 Bump version for rebuild
- 66e63bf Merge "Remove postscript, add copytruncate to logrotate." into 8.0
- d231c4a Suffix update for rabbitmq-server * Modification made according to https://github.com/openstack/fuel-specs/blob/master/specs/8.0/separate-mos-from-centos.rst#rpm-packages-versioning

2)
[root@nailgun ~]# rpm -ql rabbitmq-server-3.6.1-3.el7~mos1.noarch | grep logrotate
/etc/logrotate.d/rabbitmq-server
[root@nailgun ~]# cat /etc/logrotate.d/rabbitmq-server
/var/log/rabbitmq/*.log {
        weekly
        missingok
        rotate 20
        compress
        delaycompress
        notifempty
        sharedscripts
        postrotate
            /sbin/service rabbitmq-server rotate-logs > /dev/null
        endscript
}

3)
[root@nailgun ~]# service rabbitmq-server rotate-logs
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.

should be:
        postrotate
            /usr/sbin/rabbitmqctl -q rotate_logs
        endscript

[1]. https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.logrotate/51/

Revision history for this message
Alexey Lebedeff (alebedev-a) wrote :

I've fixed `rabbitmqctl rotate_logs` in upstream version of rabbit - https://github.com/rabbitmq/rabbitmq-server/pull/543 and so patched logrotate script is no longer needed.

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Won't Fix for 7.0-updates because of Medium importance

tags: added: wontfix-low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.