Rsyslog Forwarder HA Charm

misconfigured charm can cause /var/log/syslog and messages to grow nearly 1G/s after reboot

Bug #1895184 reported by Zachary Zehring
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Rsyslog Forwarder HA Charm
Won't Fix
Undecided
Unassigned

Bug Description

Rsyslog Forwarder HA charm was deployed as a placeholder, for forward hosts to be configured later. Found that after a reboot, /var/log/syslog and /var/log/messages grew at a high rate, filling the whole root partition in minutes. The logging is as follows:

Sep 10 14:59:34 brtlvmrs0563co * Sep 9 21:45:46 brtlvmrs0563co * Sep 9 21:45:45 brtlvmrs0563co * Sep 9 21:45:44 brtlvmrs0563co * Sep 9 21:45:44 brtlvmrs0563co * Sep 9 21:45:43 brtlvmrs0563co * Sep 9 21:45:42 brtlvmrs0563co * Sep 9 21:45:42 brtlvmrs0563co * Sep 9 21:45:41 brtlvmrs0563co * Sep 9 21:45:40 brtlvmrs0563co * Sep 9 21:45:40 brtlvmrs0563co * Sep 9 21:45:39 brtlvmrs0563co * Sep 9 21:45:38 brtlvmrs0563co * Sep 9 21:45:37 brtlvmrs0563co * Sep 9 21:45:37 brtlvmrs0563co * Sep 9 21:45:36 brtlvmrs0563co * Sep 9 21:45:35 brtlvmrs0563co * Sep 9 21:45:35 brtlvmrs0563co * Sep 9 21:45:34 brtlvmrs0563co * Sep 9 21:45:34 brtlvmrs0563co * Sep 9 21:45:33 brtlvmrs0563co * Sep 9 21:45:32 brtlvmrs0563co * Sep 9 21:45:32 brtlvmrs0563co * Sep 9 21:45:31 brtlvmrs0563co * Sep 9 21:45:30 brtlvmrs0563co * Sep 9 21:45:30 brtlvmrs0563co * Sep 9 21:45:29 brtlvmrs0563co * Sep 9 21:45:28 brtlvmrs0563co * Sep 9 21:45:28 brtlvmrs0563co * Sep 9 21:45:27 brtlvmrs0563co * Sep 9 21:45:26 brtlvmrs0563co * Sep 9 21:45:26 brtlvmrs0563co * Sep 9 21:45:25 brtlvmrs0563co * Sep 9 21:45:25 brtlvmrs0563co * Sep 9 21:45:24 brtlvmrs0563co * Sep 9 21:45:23 brtlvmrs0563co * Sep 9 21:45:23 brtlvmrs0563co * Sep 9 21:45:22 brtlvmrs0563co * Sep 9 21:45:21 brtlvmrs0563co * Sep 9 21:45:21 brtlvmrs0563co * Sep 9 21:45:20 brtlvmrs0563co * Sep 9 21:45:20 brtlvmrs0563co * Sep 9 21:45:19 brtlvmrs0563co * Sep 9 21:45:19 brtlvmrs0563co * Sep 9 21:45:18 brtlvmrs0563co * Sep 9 21:45:18 brtlvmrs0563co * Sep 9 21:45:17 brtlvmrs0563co * Sep 9 21:45:17 brtlvmrs0563co * Sep 9 21:45:16 brtlvmrs0563co * Sep 9 21:45:16 brtlvmrs0563co * Sep 9 21:45:15 brtlvmrs0563co * Sep 9 21:45:15 brtlvmrs0563co * Sep 9 21:45:14 brtlvmrs0563co * Sep 9 21:45:14 brtlvmrs0563co * Sep 9 21:45:13 brtlvmrs0563co * Sep 9 21:45:12 brtlvmrs0563co * Sep 9 21:45:12 brtlvmrs0563co * Sep 9 21:45:11 brtlvmrs0563co * Sep 9 21:45:11 brtlvmrs0563co * Sep 9 21:45:10 brtlvmrs0563co * Sep 9 21:45:10 brtlvmrs0563co * Sep 9 21:45:09 brtlvmrs0563co * Sep 9 21:45:08 brtlvmrs0563co * Sep 9 21:45:08 brtlvmrs0563co * Sep 9 21:45:07 brtlvmrs0563co * Sep 9 21:45:07 brtlvmrs0563co * Sep 9 21:45:06 brtlvmrs0563co * Sep 9 21:45:06 brtlvmrs0563co * Sep 9 21:45:05 brtlvmrs0563co * Sep 9 21:45:05 brtlvmrs0563co * Sep 9 21:45:04 brtlvmrs0563co * Sep 9 21:45:04 brtlvmrs0563co * Sep 9 21:45:04 brtlvmrs0563co * Sep 9 21:45:03 brtlvmrs0563co * Sep 9 21:45:03 brtlvmrs0563co * Sep 9 21:45:03 brtlvmrs0563co * Sep 9 21:45:02 brtlvmrs0563co * Sep 9 21:45:02 brtlvmrs0563co * Sep 9 21:45:02 brtlvmrs0563co * Sep 9 21:45:01 brtlvmrs0563co * Sep 9 21:45:01 brtlvmrs0563co * Sep 9 21:45:01 brtlvmrs0563co * Sep 9 21:45:00 brtlvmrs0563co * Sep 9 21:45:00 brtlvmrs0563co * Sep 9 21:44:59 brtlvmrs0563co * Sep 9 21:44:59 brtlvmrs0563co * Sep 9 21:44:59 brtlvmrs0563co * Sep 9 21:44:58 brtlvmrs0563co * Sep 9 21:44:58 brtlvmrs0563co * Sep 9 21:44:58 brtlvmrs0563co * Sep 9 21:44:57 brtlvmrs0563co * Sep 9 21:44:57 brtlvmrs0563co * Sep 9 21:44:57 brtlvmrs0563co * Sep 9 21:44:57 brtlvmrs0563co * Sep 9 21:44:56 brtlvmrs0563co * Sep 9 21:44:56 brtlvmrs0563co * Sep 9 21:44:56 brtlvmrs0563co * Sep 9 21:44:56 brtlvmrs0563co * Sep 9 21:44:55 brtlvmrs0563co * Sep 9 21:44:55 brtlvmrs0563co * Sep 9 21:44:55 brtlvmrs0563co * Sep 9 21:44:54 brtlvmrs0563co * Sep 9 21:44:54 brtlvmrs0563co * Sep 9 21:44:54 brtlvmrs0563co * Sep 9 21:44:54 brtlvmrs0563co * Sep 9 21:44:53 brtlvmrs0563co * Sep 9 21:44:53 brtlvmrs0563co * Sep 9 21:44:53 brtlvmrs0563co * Sep 9 21:44:53 brtlvmrs0563co * Sep 9 21:44:52 brtlvmrs0563co * Sep 9 21:44:52 brtlvmrs0563co * Sep 9 21:44:52 brtlvmrs0563co * Sep 9 21:44:52 brtlvmrs0563co * Sep 9 21:44:52 brtlvmrs0563co * Sep 9 21:44:51 brtlvmrs0563co * Sep 8 20:10:13 brtlvmrs0563co kernel: [ 0.000000] gran_size: 32M chunk_size: 128M num_reg: 10 lose cover RAM: 65G

Problem was resolved instantly by removing the charm from the model. The config was as follows (using latest revision 2):

 rsyslog-forwarder-ha:
    charm: cs:rsyslog-forwarder-ha
    options:
      forward_hosts: "1.1.1.1"
      watch-files: "/var/log/* /var/log/*/*"
      log-locally: true
      protocol: tcp

Revision history for this message
Jose Guedez (jfguedez) wrote :

Tried to reproduce using version "2", but it's not currently available at least:

$ juju deploy rsyslog-forwarder-ha-2
ERROR cannot resolve URL "cs:rsyslog-forwarder-ha-2": charm or bundle not found

Deployed the current stable instead (20):

$ juju deploy rsyslog-forwarder-ha
Located charm "rsyslog-forwarder-ha" in charm-store, revision 20
Deploying "rsyslog-forwarder-ha" from charm-store charm "rsyslog-forwarder-ha", revision 20 in channel stable

However, I was not able to reproduce the issue, even after setting the same configs, or having a valid target forward_host configuration.

Marking incomplete for the moment. Feel free to re-open if you have more information or encounter the issue again.

Changed in charm-rsyslog-forwarder-ha:
status: New → Incomplete
Revision history for this message
Eric Chen (eric-chen) wrote :

This charm is no longer being actively maintained. Please consider using the new Canonical Observability Stack instead.
(https://charmhub.io/topics/canonical-observability-stack)

Changed in charm-rsyslog-forwarder-ha:
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.