The default configuration of httpd-cc.conf on the cluster controller is causing apache2 to write the /var/log/eucalyptus/cc.log* files incorrectly. Versions 1.62, 2.02 of eucalyptus-cc.

The default configuration of httpd-cc.conf on the cluster controller is causing apache2 to write the /var/log/eucalyptus/cc.log* files incorrectly. The adverse effects from the incorrect setting are shown below.

someuser@somecchost:/var/lib/eucalyptus# lsof | grep cc.log
dhcpd3 14739 dhcpd 9w REG 8,3 20025704 1073742081 /var/log/eucalyptus/cc.log.5 (deleted)
apache2 25342 eucalyptus 9w REG 8,3 9889552 1073742264 /var/log/eucalyptus/cc.log
apache2 25343 eucalyptus 9w REG 8,3 9889552 1073742264 /var/log/eucalyptus/cc.log
apache2 25344 eucalyptus 9w REG 8,3 9889552 1073742264 /var/log/eucalyptus/cc.log
apache2 25345 eucalyptus 9w REG 8,3 9889552 1073742264 /var/log/eucalyptus/cc.log
apache2 25346 eucalyptus 9w REG 8,3 20025704 1073742081 /var/log/eucalyptus/cc.log.5 (deleted)
apache2 25371 eucalyptus 7w REG 8,3 3817703 1073742201 /var/log/eucalyptus/cc.log.3
apache2 25375 eucalyptus 9w REG 8,3 9889552 1073742264 /var/log/eucalyptus/cc.log
apache2 25376 eucalyptus 9w REG 8,3 20025704 1073742081 /var/log/eucalyptus/cc.log.5 (deleted)
apache2 25761 eucalyptus 9w REG 8,3 9889552 1073742264 /var/log/eucalyptus/cc.log
someuser@somecchost:/var/lib/eucalyptus# ps aux|grep apache
root 25338 0.0 0.0 57072 2256 ? Ss Dec06 0:15 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25342 0.1 0.1 1268136 92272 ? Sl Dec06 29:33 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25343 0.1 0.1 1268140 90676 ? Sl Dec06 29:58 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25344 0.1 0.1 1268136 92332 ? Sl Dec06 29:41 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25345 0.1 0.1 1268152 90600 ? Sl Dec06 29:43 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25346 0.1 0.1 1268528 92596 ? Sl Dec06 29:54 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25371 0.1 0.1 1268144 90548 ? Sl Dec06 29:37 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25375 0.1 0.1 1268132 92824 ? Sl Dec06 29:53 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25376 0.1 0.1 1268144 90948 ? Sl Dec06 29:41 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
106 25761 0.3 0.0 1280204 32388 ? S Dec06 81:43 apache2 -f /var/run/eucalyptus/httpd-cc.conf -D FOREGROUND
root 31471 0.0 0.0 7624 920 pts/0 S+ 09:13 0:00 grep apache

When looking at open cc.log* files you will notice that some are deleted because eucalyptus rotates the log files in it's output handler in util/misc.c as logprintfl(). They are being rotated while other threads are holding on to the file, thus rotating them until deleted. Here is the incorrect setting from /var/run/eucalyptus/httpd-cc.conf:

<IfModule mpm_prefork_module>
 StartServers 5
 MinSpareServers 5
 MaxSpareServers 10
 MaxClients 8
 MaxRequestsPerChild 0
</IfModule>

<IfModule mpm_worker_module>
 StartServers 5
 MaxClients 8
 MaxRequestsPerChild 0
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadLimit 64
 ThreadsPerChild 1
</IfModule>

<IfModule mpm_event_module>
    StartServers 5
    MaxClients 8
    MinSpareThreads 25
    MaxSpareThreads 75
    ThreadLimit 64
    ThreadsPerChild 1
    MaxRequestsPerChild 0
</IfModule>

The default configuration found in the 1.6.2 eucalyptus source and 2.0.2 source under tools called httpd.conf is set correctly.

<IfModule mpm_prefork_module>
 StartServers 0
 MinSpareServers 1
 MaxSpareServers 0
 MaxClients 1
 MaxRequestsPerChild 0
</IfModule>

<IfModule mpm_worker_module>
 StartServers 0
 MaxClients 1
 MaxRequestsPerChild 0
 MinSpareThreads 1
 MaxSpareThreads 0
 ThreadLimit 1
 ThreadsPerChild 1
</IfModule>

<IfModule mpm_event_module>
    StartServers 0
    MaxClients 1
    MinSpareThreads 1
    MaxSpareThreads 0
    ThreadLimit 1
    ThreadsPerChild 1
    MaxRequestsPerChild 0
</IfModule>

Here is version information for the eucalyptus-cc package that was cleanly installed to check for the misconfiguration.
someuser@someplace:/var/run/eucalyptus$ dpkg -l |grep eucalyptus
ii eucalyptus-cc 1.6.2-0ubuntu30.4 Elastic Utility Computing Architecture - Clu
ii eucalyptus-common 1.6.2-0ubuntu30.4 Elastic Utility Computing Architecture - Com
ii eucalyptus-gl 1.6.2-0ubuntu30.4 Elastic Utility Computing Architecture - Log
someuser@someplace:/var/run/eucalyptus$ sudo apt-get install eucalyptus-cc

Another way I figured this out is that the documentation for version 1.1 explicitly says to change the configuration file to the above settings: http://open.eucalyptus.com/wiki/eucalyptus-source-code-11

Thanks

Eric Pershey