Fuel for OpenStack

Nailgun api.log consumes 100% of disk space

Bug #1720340 reported by Alexander Rubtsov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Won't Fix
High
MOS Maintenance
Fuel for OpenStack 10.x-updates
Mitaka
Fix Released
High
Denis Meltsaykin
Fuel for OpenStack 9.2-mu-3

Bug Description

MOS: 9.2 (build 562)

This commit disables makes logrotate for Nailgun to be launched once per day rather than every 30 minutes as it was before:
https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0f27fc81a1cfab49647b09e9ff50f5d7d9f006c9

It may lead to situations where Nailgun log will consume all available space if it's growing too fast.
(e.g. DEBUG mode is enabled and the environment consists of many nodes).

Here is a sample from 80 nodes environment:
[root@fuel log]# cd nailgun/
[root@fuel nailgun]# ls -lrth
total 5.4G
-rw-r--r-- 1 root root 288 Sep 25 18:02 oswl_vm_collectord.log
-rw-r--r-- 1 root root 282 Sep 25 18:02 statsenderd.log
-rw-r--r-- 1 root root 297 Sep 25 18:02 oswl_image_collectord.log
-rw-r--r-- 1 root root 8.4K Sep 25 18:02 assassind.log
-rw-r--r-- 1 root root 321 Sep 25 18:02 oswl_keystone_user_collectord.log
-rw-r--r-- 1 root root 276 Sep 25 18:02 receiverd.log
-rw-r--r-- 1 root root 300 Sep 25 18:02 oswl_flavor_collectord.log
-rw-r--r-- 1 root root 300 Sep 25 18:02 oswl_tenant_collectord.log
-rw-r--r-- 1 root root 300 Sep 25 18:02 oswl_volume_collectord.log
-rw-r----- 1 root root 15M Sep 26 07:17 app.log
-rw-r--r-- 1 root root 5.4G Sep 26 07:17 api.log

If so, the Fuel master node may end up in inoperable state faster than the next scheduled rotation will be performed.

Revision history for this message
Alexander Rubtsov (arubtsov) wrote :

sla1 for 9.0-updates

Changed in fuel:
importance: Undecided → High
assignee: nobody → MOS Maintenance (mos-maintenance)
tags: added: customer-found sla1
Denis Meltsaykin (dmeltsaykin)
Changed in fuel:
milestone: none → 10.x-updates
status: New → Won't Fix
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-library (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/36725

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/fuel-library (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/36725
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: ce7dda3af137ff2cd3220d6ab9257224e88c895a
Author: Denis V. Meltsaykin <email address hidden>
Date: Thu Oct 5 16:44:41 2017

Return nailgun logs to fuel-logrotate

Recently fuel's logrotate configuration has been cleaned up from
duplicates and non-relevant docker folders. Also the nailgun's config
has been removed from the fuel's rotation since it has its own dedicated
rotation config. Unfortunately logrotate runs once a day which is not
enough for nailgun which generates gigabytes of logs in debug mode. This
commit returns nailgun's log rotation to fuel's config.

Also remove the native nailgun logrotate, since nailgun is not used
without Fuel in our configurations.

Change-Id: Ibf3339719ada4ef3f34b95a336f90a882c7a940f
Closes-Bug: #1720340

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Fix proposed to branch: 9.0/mitaka
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/#/c/36879/

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Verified on Fuel 9.2 MU-3 (snapshots/proposed-2017-10-16-142324)

[root@nailgun mos_mu]# cat /etc/logrotate.d/fuel.nodaily | grep nailgun
"/var/log/nailgun-agent.log"
"/var/log/nailgun/*.log"

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.