/ fs is full on controller (big /var/log/mongod.log)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Confirmed
|
High
|
Matthew Mosesohn |
Bug Description
astute_sha: 3c374c9f7bfbdbc
auth_required: true
build_id: 2014-11-12_22-00-23
build_number: '99'
feature_groups:
- mirantis
fuellib_sha: bc3c6298f203af3
fuelmain_sha: 144262798c38c91
nailgun_sha: 563749b01dd26e1
ostf_sha: 720cc1308c3a708
production: docker
Steps to reproduce:
1. Install cluster on 100 nodes (3 controllers in HA, 97 computes + ceph) wih ceilometer, ceph, neutron gre
2. Run tempest and rally tests
after ~8-9 hours uptime login to main controller, fs on / is full
[root@node-64 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 15G 15G 0 100% /
tmpfs 16G 37M 16G 1% /dev/shm
/dev/md0 194M 27M 158M 15% /boot
/dev/mapper/
[root@node-64 ~]# ls -lh /var/log/ | grep mongod.log
-rw------- 1 root adm 3.9G Nov 14 03:01 mongod.log
-rw------- 1 root adm 3.7G Nov 14 03:01 mongod.log.1
-rw------- 1 root adm 13M Nov 14 02:01 mongod.log.4.gz
-rw------- 1 root adm 19M Nov 14 01:01 mongod.log.5.gz
[root@node-64 ~]# uptime
05:53:05 up 8:50, 0 users, load average: 2.54, 2.46, 2.61
Sorry, can't attache snapshot
[root@fuel rally]# du -sh /var/log/
9.7G /var/log/
Matt recently merged this: https:/ /review. openstack. org/#/c/ 133956/ . MongoDB log should be rotated hourly according to configuration. We will investigate this