control size of neutron logs

Bug #1620864 reported by Armando Migliaccio
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Ihar Hrachyshka

Bug Description

From a recent analysis [1] on a master change, it has been noted that the size of neutron logs is amongst the biggest in an OpenStack deployment. This bug report is tracking the effort to trim down some of the non necessary traces that make the logs bloat, as this may as well affect operability.

[1] http://paste.openstack.org/show/567259/

Tags: logging
Changed in neutron:
status: New → Confirmed
assignee: nobody → Armando Migliaccio (armando-migliaccio)
importance: Undecided → High
tags: added: logging
Changed in neutron:
milestone: none → newton-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/366424

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/366440

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/366478

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/366424
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a5f78b30149b44f858ed98ab8a83608c206d3de9
Submitter: Jenkins
Branch: master

commit a5f78b30149b44f858ed98ab8a83608c206d3de9
Author: Armando Migliaccio <email address hidden>
Date: Tue Sep 6 17:29:17 2016 -0700

    Reduce the number of debug traces emitted by registry callbacks

    The tool has become popular beyond my wildest expectations. A typical
    run in the gate at debug level yields more than 20K traces in the
    neutron-server alone. However, reducing the number of traces is
    beneficial across the board and that's the first attempt at bringing
    the size of Neutron logs closer to other services'.

    Partial-bug: #1620864

    Change-Id: I2576e59961ec8928ad71d778253b2d37d470d909

Changed in neutron:
assignee: Armando Migliaccio (armando-migliaccio) → Ihar Hrachyshka (ihar-hrachyshka)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/366478
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a7c8a825a46bf99015781d1875b926cf9668c92b
Submitter: Jenkins
Branch: master

commit a7c8a825a46bf99015781d1875b926cf9668c92b
Author: Armando Migliaccio <email address hidden>
Date: Tue Sep 6 21:19:11 2016 -0700

    Emit router payload only during failures

    When everything works as expected, no-one hardly pays any attention
    to this log trace, which accounts for an incredible amount of log data.
    This change proposes to emit the router payload only during failures
    (when debugging info is needed the most), and furthermore it relocates
    it to the L3 agent log files, where it is more pertinent.

    Partial-bug: #1620864

    Change-Id: I64281b963ba52c0a100a6194b7cafc5e9b1a8e74

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/366440
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5743ed4679d43c325c5b3a2a4955088e35018c97
Submitter: Jenkins
Branch: master

commit 5743ed4679d43c325c5b3a2a4955088e35018c97
Author: Armando Migliaccio <email address hidden>
Date: Thu Sep 8 13:42:11 2016 -0700

    Reduce log level for ryu in OVS agent log

    In a tempest run, Ryu is logging about ~30K debug traces and it puts the
    OVS agent's log amongst the biggest log files in a single node OpenStack
    deployment in the gate. This patch sets Ryu and its components to log at
    INFO level and above to reduce the number of traces emitted.

    This patch, alongside [1,2], brings down the size of the biggest Neutron
    log files, server and OVS agent respectively. More can surely be done,
    however callbacks and ryu are the most obvious ones, and following up
    with further cuts may be dealt with on a ad-hoc basis.

    Closes-bug: #1620864

    [1] https://review.openstack.org/#/c/366424/
    [2] https://review.openstack.org/#/c/366478/
    Change-Id: I40b7c77601788ae2e7428c7a0206ca2a807d10dc

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/370363

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/371235

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/370363
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d6df96ddf265be103dd5851bbcdba46242230d19
Submitter: Jenkins
Branch: master

commit d6df96ddf265be103dd5851bbcdba46242230d19
Author: Armando Migliaccio <email address hidden>
Date: Wed Sep 14 11:25:43 2016 -0700

    Skip warnings during DHCP port actions if no error is raised

    With multiple concurrent DHCP operations, the server log may be
    filled with warning traces about the inability to complete the
    DHCP action correctly. These should be emitted only when reraising
    the underlying exception, namely when the underlying error is
    not an expected nor a handled one.

    Related-bug: 1621345
    Related-bug: 1620864

    Change-Id: I1f00609f8181e3765238550f45e6e114d2df7e6e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/371235
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9ac3596c6feca21d024240293476046874e51d73
Submitter: Jenkins
Branch: master

commit 9ac3596c6feca21d024240293476046874e51d73
Author: Armando Migliaccio <email address hidden>
Date: Thu Sep 15 18:52:54 2016 -0700

    Fix events misnomer in callback registry debug trace

    events is the module, we want to log the event variable.

    Related-bug: #1620864

    Change-Id: Iec0d403c57938b533dd3b314c100b6e21c3fea2a

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.0.0.0rc1

This issue was fixed in the openstack/neutron 9.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.