[2.5] No logs stored in /var/log/maas/rsyslog/

Bug #1790735 reported by Andres Rodriguez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse

Bug Description

After the recent changes to syslog, now /var/log/maas/rsyslog is now empty:

ubuntu@autopkgtest:/var/log/maas/rsyslog$ ls -l
total 0

This was obtained after machines had commissioned.

This issue was spotted after this commit [1] and causing the CI to fail [2]

[1]: https://git.launchpad.net/maas/commit/?id=fe00d1b7b6ba82e78cb43c927242a34431c6fb42
[2]: http://10.245.136.4:8080/job/maas-bionic-master-git/32/

Related branches

description: updated
Changed in maas:
importance: Undecided → High
status: New → Triaged
assignee: nobody → Blake Rouse (blake-rouse)
milestone: none → 2.5.0beta1
tags: added: rack-proxy track
Changed in maas:
importance: High → Critical
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Aside from the fact that there's no logging anymore in maas.log and /var/log/maas/rackd.log, I started seeing a lot of these on rackd.log

2018-09-04 23:45:57 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: [Errno 12] Cannot allocate memory; interfaces: None
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1005, in updateInterfaces
            interfaces = yield maybeDeferred(self.getInterfaces)
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 233, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/network.py", line 1145, in get_all_interfaces_definition
            iproute_info = get_ip_route()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/iproute.py", line 72, in get_ip_route
            ["ip", "route", "list", "scope", "global"])
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/shell.py", line 108, in call_and_check
            process = Popen(command, *args, stdout=PIPE, stderr=PIPE, **kwargs)
          File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
            restore_signals, start_new_session)
          File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child
            restore_signals, start_new_session, preexec_fn)
        builtins.OSError: [Errno 12] Cannot allocate memory

my wild guess, is that something with syslog is causing the increased memory consumption becuase there's no files written. Maybe they are all storage in memory which is causing the issue.

Changed in maas:
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.