[fuel-library] Incorrect logrotate configuration leads to lack of free disk space on master node

AFAIK, logrotate templates cannot be changed on fly out of box. I could suggest the following options to do it:
1) by CM tools (e.g. puppet agent runs) which will update /etc/logrotate.d/* as appropriate
2) by life cycle management tools, including monitoring + notifications + some triggers
3) by improving pre-deployment planning for logs storage and rotation policy as well ( there is a blueprint to address it https://blueprints.launchpad.net/fuel/+spec/manage-logs-with-free-space-consideration )

Miroslav, please elaborate what is
"At first glance the following configuration files for logrotate on master node contain the incorrect settings:
/etc/cron.hourly/logrotate on root partition - points to not existing /etc/logrotate.d/20-fuel.conf

Same misconfiguration exists inside rsyslog container."

Ok, I can see it - /etc/cron.hourly/logrotate should be fixed, indeed

20-fuel*.conf is actually a hourly scheduled rotation template. If it never called, like now, there is a chance that all free space of /var/log/ would be filled within a one day time frame - then next scheduled daily rotation is awaiting its time for execution.

Fix proposed to branch: master
Review: https://review.openstack.org/126601

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/126602

Just verified - 5.0.1 has the same issue.

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/126828

Reviewed: https://review.openstack.org/126601
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d7497cfad51701717df890ea247a4b69584c14e6
Submitter: Jenkins
Branch: master

commit d7497cfad51701717df890ea247a4b69584c14e6
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 7 17:29:15 2014 +0200

    Fix hourly log rotation schedule

    W/o this patch, new log rotation template for
    Fuel 5.x master node with docker (which is
    /etc/logrotate.d/20-fuel-docker.conf) would
    never be scheduled for execution.
    And that could have caused master node's /var/log/
    free space exhausted until the next planned daily
    log rotation will come.

    DocImpact
    Closes-bug: #1378327

    Change-Id: I21ba3debcfda4d94d5e25786f97996f6431e5ec3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/126602
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=76cdbb798d26f39711d776121f49f42a2a3b068a
Submitter: Jenkins
Branch: stable/5.1

commit 76cdbb798d26f39711d776121f49f42a2a3b068a
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 7 17:29:15 2014 +0200

    Fix hourly log rotation schedule

    W/o this patch, new log rotation template for
    Fuel 5.x master node with docker (which is
    /etc/logrotate.d/20-fuel-docker.conf) would
    never be scheduled for execution.
    And that could have caused master node's /var/log/
    free space exhausted until the next planned daily
    log rotation will come.

    DocImpact
    Closes-bug: #1378327

    Change-Id: I21ba3debcfda4d94d5e25786f97996f6431e5ec3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/126828
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a6737e68030cd92ef13bb74f176b74294139cb50
Submitter: Jenkins
Branch: stable/5.0

commit a6737e68030cd92ef13bb74f176b74294139cb50
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 7 17:29:15 2014 +0200

    Fix hourly log rotation schedule

    W/o this patch, new log rotation template for
    Fuel 5.x master node with docker (which is
    /etc/logrotate.d/20-fuel-docker.conf) would
    never be scheduled for execution.
    And that could have caused master node's /var/log/
    free space exhausted until the next planned daily
    log rotation will come.

    Closes-bug: #1378327

    Change-Id: I21ba3debcfda4d94d5e25786f97996f6431e5ec3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

When running logrotate manually with this command
logrotate -f /etc/logrotate.conf
we can observe this error
error: 20-fuel-docker.conf:3 duplicate log entry for /var/log/kern.log

To avoid this:
add to /etc/logrotate.conf before(this is important) 'include /etc/logrotate.d' directive this line:
tabooext + .ignoredaily

After that
mv /etc/logrotate.d/20-fuel-docker.conf /etc/logrotate.d/20-fuel-docker.conf.ignoredaily

Now logrotate will ignore file with hourly log rotation options /etc/logrotate.d/20-fuel-docker.conf.ignoredaily, and we get rid of "duplicate log entry" message.

I think this modification should be added both to fuel master node and to rsyslog container.

Good point, thank you

Alexander, please make a new bug with your comments in #12. This won't solve any issues with log rotation, but it will suppress some errors in logs. That makes it a low priority bug.

Verified on ISO #19

"build_id": "2014-11-17_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "19", "auth_required": true, "api": "1.0", "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1", "production": "docker", "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-17_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "19", "api": "1.0", "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1", "production": "docker", "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"}}}, "fuellib_sha": "add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"

Verified on ISO #114

"build_id": "2014-11-18_22-00-23", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "114", "auth_required": true, "api": "1.0", "nailgun_sha": "b0add09c4361fee8fc70637c9a6ef42fbe738abe", "production": "docker", "fuelmain_sha": "e556f0e1b00c30ec5c4b374ca2878c047c8686c2", "astute_sha": "65eb911c38afc0e23d187772f9a05f703c685896", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-18_22-00-23", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "114", "api": "1.0", "nailgun_sha": "b0add09c4361fee8fc70637c9a6ef42fbe738abe", "production": "docker", "fuelmain_sha": "e556f0e1b00c30ec5c4b374ca2878c047c8686c2", "astute_sha": "65eb911c38afc0e23d187772f9a05f703c685896", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "5a5275370b33ab3b9a403728a1c7ad173289e4a0"}}}, "fuellib_sha": "5a5275370b33ab3b9a403728a1c7ad173289e4a0"

Should there be a different bug report on /var being full resulting in failed/corrupted docker containers as a result?

It seems to me that log rotation is not sufficient. At least not if there is any chance of /var getting full.

A more permanent solution would be to allocate different space for logging and containers, so that one can not corrupt the other?

File system corruption in containers on master node:
https://bugs.launchpad.net/fuel/+bug/1383741

@Baboune, yes, logs storage (/var) should be planned carefully and it would be nice if Fuel could address it (see related BP https://blueprints.launchpad.net/fuel/+spec/manage-logs-with-free-space-consideration).

And of cause, logs storage should be separated from any statefull / containers' data.

How to apply this fix on a 5.1 running environment?

$ find / -name "logrotate-hourly"

/etc/puppet/2014.1.1-5.0.2/modules/anacron/files/logrotate-hourly
/etc/puppet/2014.1.1-5.1/modules/anacron/files/logrotate-hourly
/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/aaec9d27982a7dacdc6ff6a73aced11ce25105765d30e563f3f834ee2a9144aa/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/d5406e8d52cbb3611788cec6401d8fe23d4e06410be25436be0887fb105b3ed2/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/ff720279a8fcc16643946a29c22e6ea27be5c615835b0c3ed336c549b6da8233/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/56f78ed246849a774867b8fcb12c7de918f7b54e937f5982828c5c64e45eb4c8/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/13308458cdd9f1b572d21ca40f14c467c8f45db620dbaa508f442d02886b318c/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/d701157cd6614bdb122c3e42bde6786609859b537a7d1b62b6b2355598343a0d/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/1f63a20da5fda081c7d0a9a24a01cb0e5f21541b0de9b8441b133460b7e73424/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/dd861b355c2cbc706be9accf0f93749ba7a5574fce286b5d5330e8f70e8bc4a9/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/379298a33f353ba2ea33760aafc25c20ba53f4861d471b10f9043ab19cb75eb4/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/c667991c77f179982e73218121a6053eb760c392915262ed75a3a97cd0c87053/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/009535254f30cc0bb2f2819dc4cfb16d9957c91ab9bf02b1e08974e79f091775/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly

@Baboune, in order to apply it manually, first, update /etc/cron.hourly/logrotate file at the master node as it is suggested in the patch. Next, make sure you have some free space at /var partition and run "/usr/sbin/logrotate -fv /etc/logrotate.d/20-fuel*.conf"