logrotation of /var/log/libvirt/qemu is misconfigured for clouds

The 'minsize 100k' setting may not be the only problem.

I suspect that even when rotated, empty log files of deleted instances would stay behind forever (as they would be rotated forever and are not distinguishable from files of still existing instances that just don't write anything. And 'notifempty' doesn't help as then they would never be deleted as things like 'maxage' are only applied when the files are rotated).

Another issue may be the distinction of deleted instances from shutdown instances. I guess logrotate can't tell these apart, while you probably want to keep the latter and get rid of the former.

In order to

- keep the logs of deleted instances for N days
- not rotate/delete the logs of shutdown instances
- not rotate (and then lose) the last messages logged by running instances

how about:

-->
/var/log/libvirt/qemu/*.log {
        weekly
        missingok
        rotate 4
        compress
        delaycompress
        copytruncate
        notifempty
        lastaction
                keep=`virsh list --all|grep instance|awk '{print "! -name "$2".log*"}'`
                find /var/log/libvirt/qemu/ -type f -mtime 30 $keep -exec rm {} \;
        endscript
}
<--

The main changes to the current default are
- replace 'minsize 100k' by 'notifempty': this is mostly to not rotate empty logs, but to none the less ensure the 'lastaction' script is run whenever there is actually a change (lastaction is *only* run by logrotate when some rotation happened);
- add a lastaction script: this is meant to preserve the logs of all instances that still exist and should expire out the logs of deleted instances after 30 days.

There is at least one issue with this apparoach: logrotate will need 4 weeks before the first logs are deleted, so an initial cleanup may be needed for compute nodes that piled up logs.

See comments #50 & #51 in bug 832507:

"Patches are ready to solve this entirely in the libvirt layer one & for all. It'll be fixed with libvirt 1.3.3

https://www.redhat.com/archives/libvir-list/2016-February/msg01449.html"

The comments in #3 relate to a different problem in nova with regards to console logs, not libvirt logs which this bug relates to.

@jamespage re your comment #4 this LP does still somewhat relate to the bug i mentioned in #3 since nova now has the ability [1] to leverage virtlogd for actions like getConsoleOutput and iiuc libvirt will log by default to /var/log/libvirt/qemu/ yet for this type of log we would never want to do a logrotate (since that is managed by virtlogd itself).

[1] https://blueprints.launchpad.net/nova/+spec/libvirt-virtlogd

Sorry i mean to paste this link since it is the actual spec that was implemented for Ocata:

https://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/libvirt-virtlogd.html

This log information has historically been quite valuable for support. If we do modify this configuration, we should consider keeping at least a reasonable history longer than 7 days going back at least a number of months.