Fuel for OpenStack

[library] Logs from MongoDB server should be written to a separate file instead of syslog

  1. Series 5.1.x
  2. Bug #1367234
Bug #1367234 reported by Artem Panchenko
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Matthew Mosesohn
Fuel for OpenStack 5.1
5.0.x
Fix Committed
High
Tomasz 'Zen' Napierala
Fuel for OpenStack 5.0.2
5.1.x
Fix Committed
High
Tomasz 'Zen' Napierala
Fuel for OpenStack 5.1
6.0.x
Fix Released
High
Matthew Mosesohn
Fuel for OpenStack 6.0

Bug Description

Recently I ran long-time Rally performance test of OpenStack environment deployed using Fuel and after 10 hours of testing I observed that MySQL on primary controller is down due to lack of free disk space:

<27>Sep 9 09:58:01 node-1 mysqld: 2014-09-09 09:58:01 8287 [Warning] Disk is full writing './mysql-bin.000008' (Errcode: 28 - No space left on device). Waiting for someone to free space...

It also caused a lot of errors in other services such as nova-api and rabbitmq. Configuration of cluster was the following:

1. CentOS, HA, KVM, NovaFlatDHCP, Sahara, Ceilometer, OpenStack debug logging enabled
2. 1 controller+mongo, 2 controller+cinder+mongo, 1 compute and 2 compute+ceph nodes

And I used default disk partitioning configuration, so for 'root' filesystem was allocated 44 GB of disk space. During performance testing all of free disk space was occupied by huge log file:

[root@node-1 ~]# ls -lah /var/log/messages
-rw-------. 1 root root 30G Sep 9 09:43 /var/log/messages

And most of its content belonged to mongodb server:

[root@node-1 ~]# wc -l /var/log/messages
67681931 /var/log/message

[root@node-1 ~]# awk '{print $5}' /var/log/messages | sort | uniq -c | sort -n | tail -5

    760 kernel:
   1000 nailgun-agent:
  53995 rabbitmq
3930722 haproxy[47952]:
63490772 mongod.27017[16088]:

Here you can find the part of MongoDB logs:

http://paste.openstack.org/show/108840/

As you can see 93% of lines in /var/log/messages file were from MongoDB server and it wasn't rotated by logrotate, because that file is rotated weekly. In my opinion to avoid such issues we should configure rsyslog to write MongoDB logs to a separate file and add it to the logrotate's hourly rotation config (/etc/logrotate.d/20-fuel.conf).

Also, probably it's safer to create a separate LVM volume specially for logs and mount it to the /var/log, so then we could increase it's size in case of 'OpenStack debug logging' option is enabled.

Tags: library
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
status: New → Confirmed
Tomasz 'Zen' Napierala (tzn)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Tomasz 'Zen' Napierala (tzn)
Vladimir Kuklin (vkuklin)
Changed in fuel:
milestone: 5.1 → 6.0
milestone: 6.0 → 5.1
Tomasz 'Zen' Napierala (tzn)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/120588

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/120588
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f686e9ea00eaa9ae141e283ff2b8b28bacd89f72
Submitter: Jenkins
Branch: master

commit f686e9ea00eaa9ae141e283ff2b8b28bacd89f72
Author: Tomasz 'Zen' Napierala <email address hidden>
Date: Wed Sep 10 23:25:29 2014 +0200

    Adds "/var/log/messages" to hourly log rotation

    Because still many services, including MongoDB, are logging to standard log
    destination we should rotate the logs more agressively. Moreover MongoDB in
    version 2.4 cannot use LogFacility, so it's easier to fix it in logrotate than
    in MonngoDB module itself.

    Change-Id: Ice1bbf4a3bf2d453a41d7660b8742a074ec29216
    Closes-Bug: 1367234

Changed in fuel:
status: Triaged → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/120820

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/120821

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/120820
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=395fd9d20a003603cc9ad26e16cb13c1c45e24e6
Submitter: Jenkins
Branch: stable/5.1

commit 395fd9d20a003603cc9ad26e16cb13c1c45e24e6
Author: Tomasz 'Zen' Napierala <email address hidden>
Date: Wed Sep 10 23:25:29 2014 +0200

    Adds "/var/log/messages" to hourly log rotation

    Because still many services, including MongoDB, are logging to standard log
    destination we should rotate the logs more agressively. Moreover MongoDB in
    version 2.4 cannot use LogFacility, so it's easier to fix it in logrotate than
    in MonngoDB module itself.

    Change-Id: Ice1bbf4a3bf2d453a41d7660b8742a074ec29216
    Closes-Bug: 1367234
    (cherry picked from commit f686e9ea00eaa9ae141e283ff2b8b28bacd89f72)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/120821
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d3ce50bea9ec91ac932238e676decd4d24bc28e8
Submitter: Jenkins
Branch: stable/5.0

commit d3ce50bea9ec91ac932238e676decd4d24bc28e8
Author: Tomasz 'Zen' Napierala <email address hidden>
Date: Wed Sep 10 23:25:29 2014 +0200

    Adds "/var/log/messages" to hourly log rotation

    Because still many services, including MongoDB, are logging to standard log
    destination we should rotate the logs more agressively. Moreover MongoDB in
    version 2.4 cannot use LogFacility, so it's easier to fix it in logrotate than
    in MonngoDB module itself.

    Change-Id: Ice1bbf4a3bf2d453a41d7660b8742a074ec29216
    Closes-Bug: 1367234
    (cherry picked from commit f686e9ea00eaa9ae141e283ff2b8b28bacd89f72)

Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

This just rotates /var/log/messages. I'll move the file to /var/log/mongod.log in rsyslog both on nodes and on Fuel Master and add rotation policy.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/133956

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/133956
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6d0309ef2fce9b8032185bc01c7122b600844a1b
Submitter: Jenkins
Branch: master

commit 6d0309ef2fce9b8032185bc01c7122b600844a1b
Author: Matthew Mosesohn <email address hidden>
Date: Wed Nov 12 16:49:55 2014 +0400

    Move mongodb logs to mongod.log

    Enables consolidation of MongoDB logs in one location,
    instead of polluting /var/log/messages.

    Added rotation for /var/log/mongod.log on deployed nodes,
    while the current Fuel Master rotation requires no changes.

    Change-Id: I3bbc6b87ed9d125191041c310069136c59d8d1f9
    Closes-Bug: 1367234

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Mykhaylo Slobodyan (mykhaylo-slobodyan) wrote :

Verified in 6.0 ISO 49

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "49"
  build_id: "2014-12-09_22-41-06"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "2c99931072d951301d395ebd5bf45c8d401301bb"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "22bd43b89a17843f9199f92d61fc86cb0f8772f1"
  fuelmain_sha: "3aab16667f47dd8384904e27f70f7a87ba15f4ee"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/153546

Tomasz 'Zen' Napierala (tzn)
Changed in fuel:
status: Fix Released → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/153546
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ea460229777a356008551c0e15e5fce67eff8623
Submitter: Jenkins
Branch: stable/5.1

commit ea460229777a356008551c0e15e5fce67eff8623
Author: Matthew Mosesohn <email address hidden>
Date: Wed Nov 12 16:49:55 2014 +0400

    Move mongodb logs to mongod.log

    Enables consolidation of MongoDB logs in one location,
    instead of polluting /var/log/messages.

    Added rotation for /var/log/mongod.log on deployed nodes,
    while the current Fuel Master rotation requires no changes.

    Change-Id: I3bbc6b87ed9d125191041c310069136c59d8d1f9
    Closes-Bug: 1367234
    (cherry picked from commit 6d0309ef2fce9b8032185bc01c7122b600844a1b)

Revision history for this message
Yury Konov (yukonoff) wrote :

I experience similar behavior on MOS 7.0 + MU1 with LMA plugin.

[root@fuel ~]# df -h /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-varlog
                       40G 40G 0 100% /var/log

[root@fuel ~]# ls -lah $(find /var/log/ -type f | xargs du -h | sort -h | tail -n 1 | cut -f2)
-rw-r----- 1 root adm 16G Фев 8 15:53 /var/log/docker-logs/remote/node-28.domain.tld/mongod.27017.log

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.