logrotate on master node does not have configuraion for /var/log/docker-logs/remote/...

Bug #1349809 reported by Alexey Abashkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Bogdan Dobrelya
5.0.x
Fix Committed
Critical
Bogdan Dobrelya

Bug Description

all logs from discovered nodes are pleaced under /var/log/docker-logs/remote/<NODEsIP>/bootstrap/
these log files aren't mentioned in logrotate configuration:

# grep "/var/log/docker-" /etc/logrotate.d/*
/etc/logrotate.d/10-fuel-docker.conf:"/var/log/docker-*.log"
/etc/logrotate.d/20-fuel-docker.conf:"/var/log/docker-*.log"

After some idle time (10 nodes are dicovered. deployment is not started) we have full /var partition on master node.

description: updated
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

It is being served by /var/log/remote simlink as well

Changed in fuel:
status: New → Invalid
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

[root@nailgun ~]# less /var/log/remote
lrwxrwxrwx 1 root root 27 Jul 28 09:59 /var/log/remote -> /var/log/docker-logs/remote/
[root@nailgun ~]# find /var/log/remote/ -name "*.log.*"
[root@nailgun ~]# logrotate -f /etc/logrotate.d/10-fuel-docker.conf
[root@nailgun ~]# find /var/log/remote/ -name "*.log.*" | wc -l
355

As you can see forced rotation does rotate remote logs dir

Revision history for this message
Alexey Abashkin (aabashkin) wrote :

This is my /etc/logrotate.d/10-fuel-docker.conf (MOS 5.0, fresh install):
==========================================
# managed by puppet

"/var/log/kern.log"
"/var/log/debug"
"/var/log/syslog"
"/var/log/daemon.log"
"/var/log/auth.log"
"/var/log/audit/audit.log"
"/var/log/user.log"
"/var/log/mail.log"
"/var/log/cron.log"
"/var/log/docker"
"/var/log/docker-*.log"
"/var/log/lxc/*.log"
"/var/log/fuelmenu.log"
"/var/log/nailgun-agent.log"
"/var/log/nailgun/*.log"
"/var/log/astute/*.log"
"/var/log/ostf*.log"
"/var/log/httpd/*"
"/var/log/nginx/*.log"
"/var/log/rabbitmq/*"
"/var/log/cobbler/*.log"
"/var/log/supervisor/*.log"
"/var/log/dump.log"
{
# This file is used for daily log rotations, do not use size options here
  sharedscripts
  weekly
  # rotate only if 30M size or bigger
  minsize 30M
  # truncate file, do not delete & recreate
  copytruncate
  # keep logs for 4 rotations
  rotate 4
  # compression will be postponed to the next rotation, if uncommented
  # delaycompress
  compress
  # ignore missing files
  missingok
  # do not rotate empty files
  notifempty
  postrotate
      # issue HUPing for all rsyslog pids including ones in the docker containers, if any.
      /usr/bin/pkill -HUP rsyslogd 2> /dev/null 2> /dev/null || true
      reload rsyslog >/dev/null 2>&1 || true
  endscript
}

==========================================

this is your steps on my master node:

============================================
[root@fuel dockerctl]# find /var/log/remote/ -name "*.log.*"
[root@fuel dockerctl]# logrotate -f /etc/logrotate.d/10-fuel-docker.conf
[root@fuel dockerctl]# find /var/log/remote/ -name "*.log.*" | wc -l
0
============================================

I do not have logs rotated but "disk full".

Changed in fuel:
status: Invalid → New
Mike Scherbakov (mihgen)
Changed in fuel:
milestone: none → 5.1
assignee: nobody → Fuel Library Team (fuel-library)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Please update the environment info.

As far as I can see, the issue was fixed both in master and stable/5.0 branches for 5.1 and 5.0.1 releases, but 5.0 release is affected, indeed.

at master branch git blame shows
5319075f (Bogdan Dobrelya 2014-06-16 11:35:05 +0300 25) "/var/log/remote/*-*/*.log"
https://github.com/stackforge/fuel-library/commit/5319075fb440b0880122f64af14056e3808f9d54

at stable/5.0:
94cb3af9 (Bogdan Dobrelya 2014-06-16 11:35:05 +0300 26) "/var/log/remote/*-*/*.log"
https://github.com/stackforge/fuel-library/commit/94cb3af924cffd4433d038280164208a41b48256

Closes bug: #1329991
Closes bug: #1316957

Changed in fuel:
status: New → Incomplete
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

looks like the issue is a dup of https://bugs.launchpad.net/fuel/+bug/1329991

Revision history for this message
Alexey Abashkin (aabashkin) wrote :

Not exactly.

I think you need to add one more line like

/var/log/remote/*/bootstrap/*log

for "discovered" nodes.

Changed in fuel:
importance: Undecided → High
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

You are right, currently bootstrap logs aren't rotated.
If I add "/var/log/remote/*-*/*/*.log"
logrotate -d /etc/logrotate.d/10-fuel-docker.conf 2>&1 | grep bootstrap
shows it is ok now.

Changed in fuel:
status: Incomplete → Confirmed
status: Confirmed → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/110586

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/110599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/110586
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c7bb60af409d5bd51e01d18bdbc96ace5541c43a
Submitter: Jenkins
Branch: master

commit c7bb60af409d5bd51e01d18bdbc96ace5541c43a
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 30 13:04:09 2014 +0300

    Add bootstrap remote logs to rotation

    Current rotation templates are missing the 2nd level
    of directories inside of /var/log/remote/*/
    and that is an issue due to bootstrap logs are residing
    there as well.

    Closes-bug: #1349809

    Change-Id: If8f61220eeb897faf3ba403833564de85718b03d
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/110599
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=dec8421a18e3ccfa35978b36b5de3292b53e6b69
Submitter: Jenkins
Branch: stable/5.0

commit dec8421a18e3ccfa35978b36b5de3292b53e6b69
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 30 13:04:09 2014 +0300

    Add bootstrap remote logs to rotation

    Current rotation templates are missing the 2nd level
    of directories inside of /var/log/remote/*/
    and that is an issue due to bootstrap logs are residing
    there as well.

    poke ci2
    Closes-bug: #1349809

    Change-Id: If8f61220eeb897faf3ba403833564de85718b03d
    Signed-off-by: Bogdan Dobrelya <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.